By Categorizing Question Logs, I Found a Path to Creating an FAQ

Last time I wrote that "dividing tasks" is the first step toward improvement. This time, I'd like to share my experience applying that concept to Slack question logs. Instead of aiming for a bot from the start, I first divided and organized them. Just by doing that, the next step (creating an FAQ → future semi-automation) became clear.

How I divided the Slack logs

I divided them along three axes.
To ensure this doesn't end as mere "classification," I also decided why those axes were chosen.

I thought that just by dividing them into these three categories, it would become easier to see "whose questions they are, what kind of questions they are, and how to answer them." (This is just my personal experience, of course.)

What became clear by dividing

It became clearly visualized that the same questions were being repeated.
Because of this, the path toward eventually turning them into an automated response bot became realistic.

Why dividing is effective for AI

Have you ever experienced AI giving terrible answers depending on how you ask?
Since it continues to evolve daily, that might be decreasing, but while it seems like a magical entity that can understand anything, I also feel that isn't quite the case.

What I do know is that in order to use it effectively, it is important for humans to divide and organize things by meaning beforehand, and by doing so, it provides high-precision responses.

In other words, rather than passing a massive amount of unorganized Slack logs as they are, it is better to "extract only the questions" and "label which service they belong to" before passing them on, as this makes the AI's answers less likely to waver.

Just like with task decomposition, I believe that pre-processing determines the accuracy of everything.
At least for now, anyway.

The mechanism called Embedding

There is a mechanism called Embedding for AI search.
It's like turning text into a map of numbers and searching for "nearby locations."

For example

• "How do I send the contract?"

• "How should I send the contract?"

Even if the words are different, they are in nearby locations on the map.
That is why using Embedding allows you to "find the same question even if the phrasing is different."

However, since it has the weakness of picking up too much, it is useful to classify it in the preceding stage.

Shortening search time with AI

For example, suppose a question like 'I want to know the delivery date for this service' is asked on Slack.
Conventionally, this would take time by searching through past interactions or asking someone.

But if you separate and organize them, AI can present candidates like 'Here are similar questions from the past.' Moreover, even if the phrasing is different, such as 'How many days will it take?' or 'How much time is needed?', it can pick up similar questions using Embedding search.

People only need to check the candidates (or it can be fully automated). I think this alone can significantly shorten response time.

Tools used to separate Slack logs

I won't go into implementation details, but these are the three tools I used.

Roughly speaking, the procedure is as follows.

1. Retrieve direct messages from Slack

2. Store in a spreadsheet using GAS

3. Have OpenAI provide candidate labels for whether it is a question, the type, and whether it is internal or external

4. After human checking and confirmation, accumulate as FAQ material

What I am doing now → Future prospects

I decided to take it step by step instead of building a bot right away.

• Step 1: Create an FAQ
  Consolidate classified question logs and link them to existing documents

• Step 2: Use the FAQ on Slack
  When a question comes in, a person searches for candidates from the FAQ and posts them (semi-automated)

• Step 3: Future transition to a bot
  Once the FAQ has grown, move to a flow where AI presents candidates and a person makes the final decision

I want to make it a fully automated bot quickly to make it hands-off, but since there is little data at first, such as few response candidates, I think I will first grow the FAQ while thinking about what kind of structure to use.

Summary

• Slack logs become much easier to navigate when categorized by 'whether it is a question / service type / internal or external'

• Using embeddings (a map of meaning) can handle paraphrasing, but it is precisely because of the 'human preprocessing' of classification that it becomes effective in practice

• The realistic approach is to first organize the FAQ and gradually move toward automation. Aiming for a bot in the future!