How LlamaIndex's Multi-Modal RAG Works

I found the following article interesting, so I have summarized it briefly.

1. Multi-Modal RAG

One of the most exciting announcements at "OpenAI Dev Day" was the release of the "GPT-4V API". "GPT-4V" is a multi-modal model that can ingest both text/images and output text responses. This expands LLMs to a new stage.

Over the past year, the LLM application stack has emerged centered around text input and output. One of the most notable examples is "RAG" (Retrieval-Augmented Generation). It combines LLMs with external text, allowing the model to reason about data it was not trained on.

One of the biggest impacts "RAG" has had on end-users is the reduction in time required to analyze unstructured text data. By processing arbitrary documents (PDFs, web pages), loading them into storage, and feeding them into an LLM's context window, you can extract the necessary insights from them.

With the introduction of the "GPT-4V API", we can extend the concept of "RAG" to a hybrid of text/images and extract value from even larger data corpora (including images).

2. Multi-Modal RAG Pipeline

The flow of a basic multi-modal RAG pipeline is as follows:

3. Multi-Modal RAG Abstraction

Multi-modal RAG abstractions have been introduced in "LlamaIndex", making the following possible:

3-1. Multi-modal LLM

The "OpenAIMultiModal" class supports "GPT-4V", and the "ReplicateMultiModal" class supports open-source multi-modal models (as this is currently in Beta, the names may change).

While "SimpleDirectoryReader" has long been able to ingest audio, images, and video, it is now possible to pass them directly to "GPT-4V" and ask questions, as shown below.

from llama_index.multi_modal_llms import OpenAIMultiModal
from llama_index import SimpleDirectoryReader

image_documents = SimpleDirectoryReader(local_directory).load_data()

openai_mm_llm = OpenAIMultiModal(
    model="gpt-4-vision-preview", 
    api_key=OPENAI_API_TOKEN, 
    max_new_tokens=300
)
response = openai_mm_llm.complete(
    prompt="what is in the image?", 
    image_documents=image_documents
) 

Unlike the standard "LLM" class, "MultiModalLLM" can take both images and text as input.

The resources are as follows.

Future plans are as follows.

3-2. Multi-modal Embedding

We have created a "MultiModalEmbedding" that can embed both text and images. This includes methods from existing embedding models, but also includes get_image_embedding. The main implementation is "ClipEmbedding" using the CLIP model.

Future plans are as follows.

3-3. Multi-modal Index

We have created a "MultiModalVectorIndex". Unlike the existing (most popular) "VectorStoreIndex", it can store both text and images.
Indexing images requires a separate process as follows:

Both text and images are returned as results, and these results can be synthesized.

Future plans are as follows.

4. Walkthrough

4-1. Creating a Multi-modal Index

First, load the document as a combination of text and images.

documents = SimpleDirectoryReader("./mixed_wiki/").load_data()

Next, define two separate vector database collections (text and images).

Next, define the "MultiModalVectorStoreIndex".

# ローカル Qdrant ベクトルストア作成
client = qdrant_client.QdrantClient(path="qdrant_mm_db")

text_store = QdrantVectorStore(
    client=client, collection_name="text_collection"
)
image_store = QdrantVectorStore(
    client=client, collection_name="image_collection"
)
storage_context = StorageContext.from_defaults(vector_store=text_store)

# マルチモーダルインデックスの作成
index = MultiModalVectorStoreIndex.from_documents(
    documents, storage_context=storage_context, image_vector_store=image_store
)

After that, you can ask questions about the multi-modal corpus.

4-2. Captioning

Copy/paste the initial image caption as input to retrieve the retrieval-augmented output.

retriever_engine = index.as_retriever(
    similarity_top_k=3, image_similarity_top_k=3
)

# GPT-4Vのレスポンスから詳細情報を取得
retrieval_results = retriever_engine.retrieve(query_str)

The retrieved results include both images and text.

You can feed this into "GPT-4V" to ask follow-up questions or synthesize a coherent response.

4-3. Query

Ask a question here and get a response from the multimodal RAG pipeline. The "SimpleMultiModalQueryEngine" first retrieves the relevant set of text/images and feeds the input to the Vision model to synthesize the response.

from llama_index.query_engine import SimpleMultiModalQueryEngine

query_engine = index.as_query_engine(
    multi_modal_llm=openai_mm_llm,
    text_qa_template=qa_tmpl
)

query_str = "Tell me more about the Porsche"
response = query_engine.query(query_str)

The results are as follows.

The resources are as follows.