[Digital MATSUMOTO's Observations (2024/9/6)] Is this a correct understanding of AI? 05 "I set up internal documents in RAG as they were... but that's why it's not working well"

"Many companies seem to have examples where they just set internal documents into a RAG database as they are and built a system to search for documents with an LLM, but it doesn't work at all." Why do you think that is?

I believe the observation regarding this problem stems from a misunderstanding of the characteristics of RAG (Retrieval-Augmented Generation) and how to apply it. RAG is a technology for acquiring external information and extending the in-context learning of generative AI, but several elements are important for utilizing it effectively.

First, for RAG to function effectively, thequality of information registered in the databaseis important. Various companies take the approach of "setting internal documents into the RAG database as they are," but this method is not necessarily optimal. Ifdocuments are redundantorinformation is outdated or inaccurate, inaccurate information may be retrieved through RAG. This is due to the characteristic thatwhile RAG improves accuracy by providing factual information, it generates incorrect answers if given incorrect information.

Furthermore,how RAG is provided and written within the prompt also has a significant impact on system performance. If the prompt is not designed appropriately, it may not be able to answer questions correctly due to "writing styles that make it focus too much on RAG" or "insufficient trial and error in prompt writing." Therefore, when building a system using RAG, it is necessary to optimize boththe quality of the databaseandprompt design.

Also, when utilizing RAG,structuring documentsis important.Instead of simply registering documents, it is required to appropriately structure information and make it in a format that is easy to search. This enables faster and more accurate information retrieval, which can improve the accuracy of answers by the LLM.

Ultimately, to utilize RAG effectively, it is essential to understand the characteristics of the technology and perform appropriate data management and prompt design. If this is achieved, RAG has the potential to provide significant value to companies.

① If internal terminology is not standardized, there is a possibility that it will be interpreted with different meanings depending on the file
② Depending on the file format, headers/footers may be mixed in, and character codes such as line breaks and indents can become noise
③ If each document is not dispersed (has almost the same expressions), you cannot reach the file you really want to access
④ If it contains illustrations or diagrams, it is better to describe the illustrations or diagrams in words if possible and add them to the data
⑤ If specific goals are not set, you cannot correctly measure the effect of 'whether it went well'

①【✕】
The analysis text does not specifically mention the standardization of internal terminology. While the quality of the documents is mentioned, the difference in the interpretation of terms is not explained.

②【✕】→【〇】
The analysis text does not mention file formats, headers/footers, or noise. While there is a description regarding the quality of information, the specific problem of file formats is not mentioned.
(✅ Real MATSUMOTO's comment: I will add it, but since it touches on data quality, I will mark it as OK)

③【✕】
The analysis text does not mention the problem caused by documents not being dispersed. The difficulty of accessing documents with similar expressions is not mentioned.

④【✕】
The analysis text does not specifically mention adding descriptions for illustrations or diagrams. There is no description regarding additions to the data.

⑤【✕】
The analysis text does not mention specific goal setting. There is no description regarding goal setting for effect measurement.

・How RAG is used within the prompt affects the accuracy of the answer
・Improving the quality of database information is important, and redundant, inaccurate, or old information leads to incorrect answers

I believe the analysis of this problem stems from a misunderstanding of the characteristics of RAG (Retrieval-Augmented Generation) and how to apply it. RAG is a technology for retrieving external information and extending the in-context learning of generative AI, but several elements are important for utilizing it effectively.

First, for RAG to function effectively,the quality of the information registered in the databaseis important. Many companies take the approach of 'setting internal documents as they are into the RAG database,' but this method is not necessarily optimal.

🤖 Added by Digital MATSUMOTO) Especially when using internal documents for RAG, if internal terminology is not standardized, the same term may be interpreted with different meanings depending on the file.Also, ifthe documents are redundantorthe information is old or inaccurate, inaccurate information may be retrieved through RAG.🤖 Added by Digital MATSUMOTO) Furthermore, depending on the file format, headers and footers may be mixed in, and character codes such as line breaks and indents can become noise. For documents containing illustrations or diagrams, it is desirable to describe them in words if possible and add them to the data. By supplementing visual information as text information, it is possible to improve search accuracy.

How RAG is provided and written within the prompt also has a significant impact on system performanceIf the prompt is not designed appropriately, it may not be possible to answer questions correctly due to 'writing that makes RAG focus too much' or 'insufficient trial and error in prompt writing.' Therefore, when building a system using RAG, it is necessary to optimize boththe improvement of database qualityandprompt design.

Also, when utilizing RAG,structuring documentsis also important. Rather than simplyregistering documents, if you appropriately structure the information and make it in a format that is easy to search, it becomes possible to retrieve information more quickly and accurately, and the accuracy of answers by the LLM can be improved.🤖 Added by Digital MATSUMOTO) If many internal documents are not dispersed because they have almost the same expressions, even if you perform RAG, the user may not be able to reach the information they really need. Increasing the uniqueness of documents and organizing/classifying information is an approach to solving this problem.

🤖 Added by Digital MATSUMOTO) And if specific goals are not set, you cannot correctly measure the effect of 'whether it went well.' It is important to clarify the success goals of the project and evaluate the effect using quantitative indicators.

Ultimately, to utilize RAG effectively, it is essential to understand the characteristics of the technology and perform appropriate data management and prompt design. If this is realized, RAG has the potential to provide significant value to companies.