Applying OpenAI's RAG Strategies to LangChain
I found the following article interesting, so I have summarized it briefly.
1. Introduction
“OpenAI” reported on a series of RAG experiments at their Demo Day. While evaluation metrics vary by application, it is interesting to see what worked and what did not. Below, I will explain each method and show how to implement each one yourself. The ability to understand these methods in your application is crucial. Because different problems require different retrieval methods, there is no "one-size-fits-all" solution.
2. How it fits into the RAG stack
First, we categorize each method into several "RAG categories." Below is a diagram showing each RAG experiment within its category and its placement in the RAG stack.
3. Baseline
Distance-based vector database search embeds (represents) queries in a high-dimensional space and finds similar embedded documents based on "distance." The baseline retrieval method used in OpenAI's research mentioned cosine similarity. LangChain has over 60 vector store integrations, many of which allow for the distance functions used in similarity search. Useful blog posts on various distance metrics can be found at Weaviate and Pinecone.
4. Query Transformations
“Query Transformations” is a set of approaches focused on transforming user input to improve retrieval. Please refer to here for a recent blog on this topic.
OpenAI reported the following two methods.
・Query expansion : LangChain's “Multi-query retriever” uses an LLM to achieve query expansion, generating multiple queries from various perspectives for a specific user input query. For each query, it retrieves a set of relevant documents and obtains a unique union across all queries.
・HyDE : LangChain's “HyDE” (Hypothetical Document Embeddings) retriever generates a hypothetical document for an incoming query, embeds it, and uses it for retrieval (see paper). The idea is that these simulated documents may have higher similarity to the target source documents than the questions themselves.
Other ideas to consider are as follows.
・Step back prompting : This paper demonstrates that for reasoning tasks, step-back questions can be used to synthesize answers based on higher-level concepts or principles. For example, a question about physics can be abstracted into a question and answer about the physical principles behind the user's query. The final answer can be derived from the input question and the step-back answer. For more details, please refer to this blog post or the LangChain implementation.
・Rewrite-Retrieve-Read : In this paper, the user's question is rewritten to improve retrieval. For more details, please refer to the LangChain implementation.
5. Routing
When executing queries across multiple data stores, it becomes important to route the question to the appropriate source. In the "OpenAI" presentation, it was reported that it was necessary to route questions between two vector stores and one SQL database. LangChain supports routing, which uses an LLM to gate user input to a set of defined sub-chains (which, as in this case, could be different vector stores).
6. Query Construction
Since one of the data sources mentioned in the "OpenAI" study is a relational (SQL) database, it was necessary to generate valid SQL from user input to extract the required information. LangChain supports text-to-sql. This is reviewed in detail in a recent blog focused on query construction.
Other ideas to consider include the following:
・Text-to-metadata filter (vector store)
・Text-to-Cypher (graph database)
・Text-to-SQL+semantic (semi-structured data in Postgres with Pgvector)
7. Index Construction
"OpenAI" reported that simply experimenting with chunk size during document embedding led to significant performance improvements. Since this is a central step in index construction, there is an open-source Streamlit app where you can test chunk sizes.
While significant performance gains from fine-tuning embeddings have not been reported, good results have been. OpenAI notes that this is likely not recommended as a "low-hanging fruit," but they have shared a guide for fine-tuning and there are several excellent HuggingFace tutorials that explain this in detail.
8. Post-processing
Processing documents after retrieval and before LLM ingestion is a critical strategy for many applications. Post-processing can be used to enforce diversity or recency in retrieved documents. This becomes especially important when pooling documents from multiple sources.
OpenAI reported the following two methods.
・Re-rank : The integration of LangChain with Cohere ReRank is one approach, which can be used for document compression (reducing redundancy) when retrieving a large number of documents. Related to this, "RAG-fusion" uses reciprocal rank fusion (see blog and implementation) to re-rank documents returned from a retriever, similar to multi-query (explained above).
・Classification : OpenAI classified each retrieved document based on its content and selected different prompts depending on the classification. This connects two ideas. LangChain supports tagging of text for classification (e.g., using function calling to enforce an output schema). As mentioned earlier, you can also use logical routing to route based on tags (or include the semantic tagging process within the logical routing chain itself).
Other ideas to consider include:
・MMR : To balance relevance and diversity, many vector stores offer max-marginal-relevance search (see blog post).
・Clustering : Some approaches use clustering of embedded documents via sampling. This can be useful when consolidating documents across a wide range of sources.
