Evaluating the Performance of a Custom RAG System

It would be great to have a system that can search and answer questions based on information that only exists within a company or organization. So, I evaluated the performance of a search system (RAG) built using Microsoft's cloud, "Azure"

The search flow is as follows.
A key feature is that it uses the GPT model twice in a single query.

Regarding evaluation metrics, I evaluated it using the following two.

⚫︎ Accuracy Rate … The probability that the reference and the answer content were correct [%] (higher is better)

⚫︎ Hallucination Rate … The probability that it gave an incorrect answer when the correct answer was that there was no reference [%] (lower is better)

For the test data, I used 10 each of fictitious company names, department names, product names, and person names as follows. I actually saved 5 of these data points, and for the remaining 5, I did not save the data and only performed the queries.

In the test cases with data, I confirmed that it was able to answer the information requested by the user as follows.

⚫︎ Accuracy Rate = 20 correct answers / 20 total trials = 100 [%]

In the test cases without data, I confirmed that it was able to answer that the information requested by the user did not exist as follows.

⚫︎ Hallucination Rate = 0 incorrect answers / 20 total trials = 0 [%]

So, good results were obtained for both evaluation metrics.

I feel that the key to quality lies in the so-called
“prompt engineering” part.
In terms of program code, the following is the corresponding part.

# チャットモデルへのリクエスト
final_response = openai.ChatCompletion.create(
    engine="my-chat-model",
    temperature=0.2,
    timeout=10,
    messages=[
        {
            "role": "system",
            "content": (
                "あなたは外部データソースを活用し、ユーザーの質問に対して正確かつ簡潔に回答を提供するアシスタントです。\n"
                "提供された検索結果とユーザーの質問が意味的に関連しているかを厳密に判断してください。\n"
                "以下の基準で判断を行ってください：\n"
                "1. 質問と検索結果の固有名詞、キーワード、話題が明確に一致しているかを確認する。\n"
                "   - 部分的な一致（例: 「部署」 という単語のみの一致）は不十分とする。\n"
                "   - 企業名や特定の話題が直接一致しない場合は関連がないと判断する。\n"
                "2. 検索結果の内容が、質問の意図に対して直接的な情報を提供しているか確認する。\n"
                "   - 例: 「企業の部署」に関する質問なのに「企業の事業内容」に関する情報しかない場合、関連がないとみなす。\n"
                "3. 関連する情報がない場合、『情報がありません』と回答する。\n"
                "   - 無理に情報をこじつけたり、推測を含めて回答しないこと。\n"
                "   - 質問の主語（企業名・人物名など）と検索結果の主語が一致しない場合も『情報がありません』とする。\n"
                "4. 関連がある場合のみ、検索結果の内容を簡潔にまとめて回答を作成する。\n"
                "   - 検索結果に基づかない推測を追加しないこと。\n"
                "   - 回答を簡潔にまとめ、不要な情報は省略する。\n"
                "5. 検索結果のリンク情報については言及しない。\n"
                "   - 検索情報が参考リンクとしてユーザーに別途提供されるため、回答中では触れない。\n"

            )
        },
        {
            "role": "user",
            "content": user_input
        },
        {
            "role": "assistant",
            "content": (
                f"検索で以下の情報が見つかりました:\n{retrieved_texts}\n"
                "検索結果と質問の関連性を厳密に評価します。\n"
                "関連がない場合は『情報がありません』と回答し、関連がある場合のみ、検索結果を基に簡潔に回答します。\n"
                "検索結果にない情報を推測で補うことは禁止します。\n"
            )
        },
    ]
)

temperature is the parameter that serves as an indicator of accuracy.
With the set temperature=0.2, you can increase the possibility of providing a strict answer. However, please be aware that it becomes less creative.

What is important in a RAG system is that the GPT model determines whether the data found in the search is what the user is looking for.

In this respect, the RAG system is still in its infancy, and I believe it is a promising mechanism, including not only ChatGPT but also other generative AI.

DeepSeek, comparing performance with other models might yield interesting results.

Please feel free to leave a comment if you have any thoughts on these results or the RAG system.

Thank you for reading until the end.