[Tokyo Gubernatorial Candidate #TakahiroAnno #AIAnno] Category Generation and Classification using GPT-4o + LangChain

Hello, this is Shimizu from the AI Anno technical team. We are permitted to write blog posts as part of our election campaign until 11:59 PM on Saturday, July 6th. Please stay with us until the end!
In this article, I will talk about category generation and classification using GPT-4o and LangChain. This is one of the Human-in-the-loop processes used to make AI Anno smarter.

For more details about AI Anno, please check out the following article.

For an overview of the Human-in-the-loop process to make AI Anno smarter, please check out the following article.

I want to classify questions that could not be answered

AI Anno answers questions from everyone on YouTube Live, but there are some questions it cannot answer. It is useful to classify what kind of questions could not be answered in these cases. This is because we can check the questions categorized by human eyes and then enlist the help of experts to improve our FAQ for each category.
By doing this, we can increase the number of questions we can answer and help everyone better understand Takahiro Anno's policies.

The difficult part here is thatwe cannot predict or grasp in advance what kind of questions will be sent by users. Because of this, simply assigning categories to all questions is not enough.
Experts who took a quick look at the spreadsheet suggested the following categories.

However, since it was clear that this did not cover everything, there was motivation to create a mechanism to extract categories effectively after reading all the questions.

Therefore, the final design ended up as follows.

1. Have GPT-4o generate categories,
2. Narrow them down to a number convenient for humans to judge,
3. Assign categories to the questions again
This is a pipeline consisting of these three steps.

Please refer to the following article for the process of collecting questions that AI Anno could not answer.

Category generation by GPT-4o

Questions that AI Anno could not answer are converted into data in the following format.

We extract information based on this data. We referred to the TTTC (Talk to the City) project from the same team for this overall flow and code.
Please check out this article as well.

Prompt

When using LangChain to ask an LLM (Large Language Model) to perform a task, we prepare a prompt that describes the details of the task. The LLM receives that prompt and data, executes the task, and returns the results.
In this task, we provide a prompt asking it to categorize the questions sent to YouTube Live. To make it easier for humans to work with, we ask it to generate categories such that there are 7 main categories, each with 3 subcategories.
Also, because we want to process the results mechanically, we specify that we want the results in JSON format.
As a result, it became a long prompt like the one below.

# system prompt
system= """
you are a category labeling assistant. 
we are a team of volunteers who are working on a political campaign.
the campaign is for the mayor of tokyo in 2024, japan.
the candidate is a 33 year old, male, former AI engineer, 
has ran 2 successful startups,
and has written an award winning book of Science Fiction.
we have a 93 page policy document, and most of the answers come from there.
however, we would like to update/enrich the policy document 
with what the public is asking.
we have a youtube live, and users have been asking questions.
where there is sufficient information in the policy document, 
the answer is provided.
when there is not, we give an answer as 
'その質問には答えられません。私はまだ学習中であるため、答えられないこともあります。
申し訳ありません。'.

we would like to know the broad category of the questions provided, 
so we can tune 
our answers to the incoming questions.
your task is to come up with around 7 categories and 3 subcategories 
for each category.

each answer should be in a json format with keys 
'category', 'subcategory', 'subcategories'.
subcategory should be 1 of the 3 subcategories for the category.
this should be a plain one line json, with no newlines.

there will be {num_questions} questions to categorize.
the final format should be a list of jsons, one for each question.

the result should be in JAPANESE.

the question are categorized as follows:
"""

The code to execute this prompt is as follows.

def get_categories(batch, retries=3):
    num_questions = len(batch)
    human = get_human_template(num_questions)

    prompt = ChatPromptTemplate.from_messages([
        ("system", system),
        ("human", human),
    ])
    chain = prompt | llm

    questions = format_batch_questions(batch)
    result = chain.invoke(
        {
            # explode a dict like {"question_1": "foo", "answer_1": "bar"} 
            "num_questions": num_questions,
            **questions,
        }
    )
    try:
        res = json.loads(result.content)
        if len(res) != num_questions:
            raise ValueError("Expected the same number of results as questions")
        return res
    except (json.decoder.JSONDecodeError, ValueError) as e:
        if retries > 0:
            logging.info(f"error: {e} {retries=} ...")
            return get_categories(batch, retries=retries-1)
        else:
            logging.error(f"Could not parse response: {result.content}")
            default = {
                "category": "unknown",
                "subcategory": "unknown",
                "subcategories": ["unknown", "unknown", "unknown"],
            }
            return [default] * num_questions

As a refinement, I have written a process to retry if the result does not return in JSON format.
In practice, categories are assigned to questions in the following format.

question: 都道の渋滞対策はどのようなものを者大考えですか？
そうだよりもハード施策を教えてください

category: 都市計画とインフラ	
subcategory: 公共交通
subcategories: [{'subcategory': '公共交通', 'subcategory_count': 4},
 {'subcategory': 'インフラ', 'subcategory_count': 4}, 
{'subcategory': '住宅政策', 'subcategory_count': 4}]

Speeding up with multi-processing

Sending the above request to the LLM took 2-3 seconds per response. Therefore, I attempted to speed it up by sending requests in parallel and implemented the following. By using 10 parallel processes, I was able to achieve a 10x speed increase.

def parallel_process(batches: list, max_workers=10) -> list[list[dict]]:
    with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
        futures = [executor.submit(get_categories, batch) for batch in batches]
        concurrent.futures.wait(futures)
    return [future.result() for future in futures]


def flatten(list_of_lists):
    """Flatten a list of lists into a single list."""
    return [item for sublist in list_of_lists for item in sublist]

Speeding up with batch processing

In the initial code, I was making one API request per question. Since each response took 2-3 seconds, even with the multi-processing mentioned earlier, the processing time was around 20 minutes. I then considered that the LLM might actually have the capability to take 10 pieces of data and return 10 responses, so I implemented batch processing. The code is as follows.

def format_batch_questions(batch) -> dict:
    # make the batch in formatted questions with keys like
    # question_1: "foo", "answer_1": "bar"
    # question_2: "foo", "answer_2": "bar"
    # ...
    res = {}
    for i, d in enumerate(batch):
        res[f"question_{i}"] = d["question"]
        res[f"answer_{i}"] = d["answer"]
    return res


def get_human_template(num_questions: int):
    """makes a template for questions like

    question_0: {question_0}, answer_0: {answer_0}
    question_1: {question_1}, answer_1: {answer_1}
    """
    human_template = "question_{i}: {question_{i}}, answer_{i}: {answer_{i}}"
    all = []
    for i in range(num_questions):
        all.append(human_template.replace("{i}", str(i)))
    human_template_filled = "\n".join(all)
    return human_template_filled


def get_batch(df, batch_size, start=0):
    return df.iloc[start:start+batch_size].to_dict(orient="records")

Interestingly, I found that I could batch process up to 15 items, but beyond that, it would not return the correct number of items or valid JSON. After several trials, I set the batch size to 15, which increased the execution speed by 15 times and allowed me to finish the processing in about 2 minutes.

Final extraction code

Combining these, the execution code became as follows, optimized with the number of workers and batch size.

def extract(df, batch_size=15, max_workers=10):
    all = []
    for i in tqdm(range(0, len(df), max_workers * batch_size)):
        # get max_workers number of batches
        batches = [get_batch(df, batch_size, start=i+j*batch_size) for j in range(max_workers)]
        batches_res = parallel_process(batches, max_workers=max_workers)
        # concat the results to original batches and save
        df_res = pd.concat(
            [
                pd.DataFrame(flatten(batches)),
                pd.DataFrame(flatten(batches_res)),
            ],
            axis=1,
        )
        all.append(df_res)
        all_df = pd.concat(all, ignore_index=True)
        all_df.to_csv("data/df_res_回答できなかった質問.tsv", sep="\t", index=False)
    return pd.concat(all, ignore_index=True)

I designed it to save frequently during execution because I wanted to verify that it was running correctly along the way.

Category refinement using GPT-4o

200 major categories

Initially, I thought I could perform category classification directly using the category generation mentioned above. To make it usable for direct classification, I requested in the prompt to classify them into '7 major categories and 3 subcategories within each.' However, looking at the actual results, while it returned a major category and 3 subcategories for each individual item, when I collected them all, 200 major categories had been generated as shown below.

I understood that it is inevitable that creating consistent categories across the board is difficult because the LLM only sees 15 pieces of data at a time, which is the batch size, during each category generation.

Category refinement

For this reason, I then provided the LLM with the 200 major categories and numerous subcategories, and asked it to consolidate them into the most meaningful 7 major categories with 3 subcategories each. The prompt is as follows.

system = """
you are a category labelling assistant. you need to make 7 categories and 21 subcategories.
these come from users's questions to the candidate of the mayor of tokyo for 2024. 

the candidate is a 33 year old, male, former AI engineer, has ran 2 successful startups,
and has written an award winning book of Science Fiction.
we have a 93 page policy document, and most of the answers come from there.
however, we would like to update/enrich the policy document with what the public is asking.
we have a youtube live, and users have been asking questions.

from these labels, make 7 categories and 21 subcategories, that sufficently map most of the interest.
the categories should be broad, and the subcategories should be specific.
1 category should have 3 subcategories.
the potential categories and subcategories data have counts associated with them.
the current candidates for categories are 
{current_categories}

they should be taken into account, but you don't need to follow them exactly.
however, 他候補者との関係 must be a subcategory>
as you can see, the top level cateogory should be a long word or sentence that 
takes into account many aspects of the subcategories.

the result should be a json file with the categories and subcategories.
just the json file, no headers or anything else. 
add the counts to the json file as well.
so the keys should be category, category_count, subcategory, subcategory_count.

the potential categories are {categories} and the subcategories are {subcategories}.
"""

As a result, I was able to obtain the following categories.

- **経済と財政政策** [経済政策, 財政政策, 税制改革]
- **テクノロジーとイノベーション** [AI技術, ブロックチェーン, 自動運転技術]
- **教育と子育て支援** [高等教育, 奨学金, 子育て支援]
- **社会福祉と福祉政策** [障害者支援, 高齢者支援, 生活保護]
- **環境とエネルギー政策** [再生可能エネルギー, エネルギー政策, 気候変動対策]
- **都市計画とインフラ** [公共交通, 交通インフラ, 住宅政策]
- **選挙と政治活動** [選挙戦略, 他候補者との関係, 選挙活動]

Since there appeared to be some overlaps from a human perspective, I made a few changes and set the following as the final categories.

- **経済と財政政策** [経済政策, 財政政策, 税制改革]
- **テクノロジーとイノベーション** [AI技術, ブロックチェーン, 自動運転技術]
- **教育と子育て支援** [高等教育, 奨学金, 子育て支援]
- **社会福祉と福祉政策** [障害者支援, 高齢者支援, 生活保護]
- **環境とエネルギー政策** [再生可能エネルギー, 原発, 気候変動対策]
- **都市計画とインフラ** [公共交通, インフラ, 住宅政策]
- **選挙と政治活動** [選挙戦略, 他候補者との関係, 選挙活動]

Category classification using GPT-4o

Finally, using the above categories as the ground truth, I requested that the corresponding category and subcategory be assigned to each question again. The prompt became as follows.

system = """
you are a category labelling assistant. you need to make 7 categories and 21 subcategories.
these come from users's questions to the candidate of the mayor of tokyo for 2024. 

the candidate is a 33 year old, male, former AI engineer, has ran 2 successful startups,
and has written an award winning book of Science Fiction.
we have a 93 page policy document, and most of the answers come from there.
however, we would like to update/enrich the policy document with what the public is asking.
we have a youtube live, and users have been asking questions.

each question should be labelled with a category and a subcategory.
the categories and subcategories are: {categories}
read all the categories and subcategories very carefully, and use your imagination to 
find the one that matches the best.
if there is really nothing that matches, you can use the category and subcategory as "other".
before labelling as "other", you must carefully read all the categories and subcategories.
and find a one that will with some context be a match.

each answer should be in a json format with keys 'category', 'subcategory', 'subcategories'.
subcategory should be 1 of the 3 subcategories for the category.
this should be a plain one line json, with no newlines.

there will be {batch_size} questions to categorize.
the final format should be a list of jsons, one for each question.

the result should be in JAPANESE.

this question is labeled as:
"""

The rest of the code was repurposed from the category generation one. Interestingly, since errors started returning when the batch size was set to 15, I performed the category classification with a batch size of 5. Perhaps category mapping is a more difficult task for the LLM than category generation. I handed these results over to the experts in each field who were creating the FAQs. They thanked me, saying it made their work much more efficient than when there were no categories, and I felt it was worth doing.

Summary

In this project, I performed category classification for unanswerable questions by dividing the process into three steps: generation, refinement, and classification. Since this was my first time properly using LangChain, I designed it this way while checking the results sequentially. Experienced users could likely accomplish this in a single step by performing category generation and classification recursively. If you manage to achieve that, I would be very happy if you could let me know.

The long election campaign is reaching its final stage, and tomorrow is election day! Please cast your vote for Takahiro Anno!
I will be live streaming until 11:50 PM on Saturday, July 6th, so I would be happy if you could join me!

#TakahiroAnno for Governor of Tokyo

For the latest information, please follow the official X (Twitter) accounts of the candidate and his office!
Takahiro Anno Office (@annotakahiro24)
Takahiro Anno (@takahiroanno)