How to Build a RAG Pipeline: 12 Steps, 90 Min [2026]

August 4, 2026

23 min read

Large language models know a lot in general and nothing about your specifics. They haven’t read your internal wiki, last week’s changelog, or the PDF a customer just emailed in. Retraining a model every time a document changes is slow, expensive, and overkill for most teams. A RAG pipeline sidesteps that problem entirely: instead of baking knowledge into the model’s weights, it fetches the relevant text at query time and hands it to the model as context, so answers stay grounded in your actual data instead of whatever the model happened to memorize during training.

This tutorial builds a complete, working RAG pipeline from scratch. You’ll load documents, split them into chunks, generate embeddings, store them in a vector database, retrieve the right chunks for a given question, and generate a grounded answer with citations. Then you’ll wrap the whole thing in a FastAPI endpoint and add an evaluation layer so you can tell whether it’s actually working, not just whether it runs without crashing.

Budget about 90 minutes for the 12 steps below. Every tool used here is either open source or has a usable free tier: LangChain for orchestration, Chroma for local vector storage, and the OpenAI API for embeddings and generation, though the generation step swaps out for Claude, Gemini, or a self-hosted model with a one-line change.

Google · Preferred Sources

Don't miss new tech stories on Google

Add Tech Insider once in the Google app and our stories appear in your news suggestions.

Add Now

What Is a RAG Pipeline, and Why It Matters in 2026

Retrieval and Generation Are Two Separate Jobs

Retrieval-augmented generation pairs two systems that are each good at a different job. A retriever searches an external knowledge base and pulls back the passages most relevant to a question. A generator, almost always an LLM, reads those passages alongside the question and writes an answer. Neither half does much alone: a retriever without a generator just returns a list of documents, and a generator without a retriever can only answer from what it learned during training. Put together, the model answers using text it’s seeing for the first time in the prompt, which is the whole point.

The technique traces back to a 2020 paper from Facebook AI Research, “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” by Patrick Lewis and coauthors, which combined a dense passage retriever with a sequence-to-sequence generator and showed it beat purely parametric models on open-domain question answering. The architecture in that paper looks almost nothing like a production RAG pipeline today, but the core idea, retrieve first and generate second, hasn’t changed.

It’s a fair question whether retrieval-augmented generation still matters now that some models accept context windows large enough to swallow a small book. It does, for three reasons that have nothing to do with raw token capacity.

Why RAG Still Matters With Bigger Context Windows

Cost is the first reason: sending an entire knowledge base as context on every single query burns tokens that aren’t needed, since most questions only touch a handful of relevant passages. Freshness is the second: a system that re-indexes nightly reflects yesterday’s changes, while a fine-tuned model reflects whenever it was last trained, which is a much slower and costlier cycle to repeat. Traceability is the third, and often the most important in practice. A well-built retrieval step can cite exactly which document and passage an answer came from, which matters enormously the moment someone asks “where did that number come from” in a compliance or customer-facing context. None of that requires a small context window to be true. It just means stuffing everything into the prompt is rarely the efficient or verifiable choice, even when it’s technically possible.

Prerequisites: Tools, Versions, and Accounts You’ll Need

Get the environment sorted before writing any pipeline code. Most RAG bugs later in this tutorial trace back to a mismatched package version or a missing API key, not to the retrieval logic itself.

You’ll need Python 3.10 or newer (3.12 is a safe default for mid-2026), an OpenAI API key for embeddings and generation, and roughly 500MB of free disk space for a small test corpus plus its vector index. None of this requires a GPU. Embedding calls and chat completions both run through an API, so a laptop is plenty for everything in this guide.

Component	Version used in this guide	Notes
Python	3.10 or newer	3.12 recommended
langchain	1.3.x	langchain-core sits on the 1.0.x line as of mid-2026
langchain-openai	Latest release	Provides OpenAIEmbeddings and chat model wrappers
langchain-chroma	Latest release	Chroma integration package
chromadb	Latest release	Local vector store used in this tutorial
openai	Latest release	Python SDK for embeddings and chat completions
ragas	Latest release	Retrieval and answer-quality evaluation
fastapi / uvicorn	Latest release	Serves the pipeline as an HTTP API

If you’d rather run the generation step against a locally hosted model instead of an API, our guide on how to run an LLM locally covers getting a model downloaded and serving requests, which then drops into Step 8 below in place of the OpenAI call.

Step 1: Set Up Your Python Environment

Isolate this project in its own virtual environment so package versions don’t collide with anything else on your machine.

python3 -m venv rag-env
source rag-env/bin/activate
pip install --upgrade pip

python3 -c "import sys; print(sys.version)"

Confirm you’re on Python 3.10 or newer before continuing. On Windows, activate the environment with rag-env\Scripts\activate instead of the source command above.

Step 2: Install Core RAG Dependencies

Install the packages listed in the prerequisites table in one pass. This pulls in LangChain’s orchestration layer, the Chroma vector store, the OpenAI SDK for embeddings and chat, RAGAS for evaluation later, and FastAPI for the API wrapper at the end.

pip install langchain langchain-openai langchain-chroma langchain-community \
    langchain-text-splitters chromadb openai tiktoken python-dotenv \
    fastapi uvicorn ragas datasets

Create a .env file in your project root to hold your API key rather than hardcoding it anywhere in the pipeline code:

OPENAI_API_KEY=sk-your-key-here
CHAT_MODEL=gpt-4o-mini
CHROMA_PERSIST_DIR=./chroma_db

Keep .env out of version control. A stray committed API key is one of the most common ways a small side project turns into an unexpected bill.

Step 3: Load and Prepare Your Document Corpus

Every retrieval-augmented system starts with a corpus, the set of documents you want the model to be able to answer questions about. For this tutorial, drop a handful of Markdown or text files into a data/docs folder: product documentation, support macros, internal policy pages, whatever you’d actually want answered from. LangChain’s document loaders handle most common formats, including PDF, HTML, and CSV, with a different loader class for each.

from pathlib import Path
from langchain_community.document_loaders import DirectoryLoader, TextLoader

DATA_DIR = Path("./data/docs")

loader = DirectoryLoader(
    str(DATA_DIR),
    glob="**/*.md",
    loader_cls=TextLoader,
    show_progress=True,
)
raw_docs = loader.load()
print(f"Loaded {len(raw_docs)} source documents from {DATA_DIR}")

Each item in raw_docs is a LangChain Document object with two parts: page_content, the raw text, and metadata, a dictionary that automatically includes the source file path. Hold onto that metadata. It’s how the pipeline traces an answer back to a specific document later, and stripping it out early is one of the most common mistakes teams make when they first wire one up.

Step 4: Chunk Documents for Retrieval

You can’t embed and search a 20-page document as a single unit and expect precise results. Chunking breaks each document into smaller, overlapping pieces so the retriever can pull back just the passage that actually answers a question, instead of an entire file’s worth of mostly irrelevant text.

A chunk size of 300 to 500 tokens with roughly 10 to 20 percent overlap between consecutive chunks is a reasonable starting point for most text-heavy corpora, and it’s the range most production RAG pipelines converge on after tuning. Smaller chunks give the retriever finer precision but lose surrounding context. Larger chunks preserve context but drag in irrelevant sentences that dilute the embedding and confuse the generator. The overlap matters because it stops an answer from getting split across a hard chunk boundary, where the sentence containing the actual answer gets cut in half between two chunks and neither one scores well against the query.

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=75,
    separators=["\n\n", "\n", ". ", " ", ""],
    length_function=len,
)

chunks = splitter.split_documents(raw_docs)
avg_len = sum(len(c.page_content) for c in chunks) // len(chunks)
print(f"Split {len(raw_docs)} documents into {len(chunks)} chunks")
print(f"Average chunk length: {avg_len} characters")

RecursiveCharacterTextSplitter tries to split on paragraph breaks first, then sentence breaks, then words, only falling back to a hard character cut as a last resort. That ordering keeps chunks from breaking mid-sentence whenever the source text has any structure to work with.

Step 5: Generate Embeddings for Your Chunks

An embedding model turns each text chunk into a vector, a list of numbers that captures its meaning in a form you can compare mathematically. Two chunks about the same topic land close together in that vector space even if they don’t share any of the same words, which is what makes semantic search possible in the first place.

import os
from dotenv import load_dotenv
from langchain_openai import OpenAIEmbeddings

load_dotenv()

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

sample_vector = embeddings.embed_query(chunks[0].page_content)
print(f"Embedding dimensions: {len(sample_vector)}")

text-embedding-3-small is a reasonable default: it costs $0.02 per million input tokens and produces 1,536-dimensional vectors, which is cheap enough that embedding a few hundred pages of documentation costs a fraction of a cent. If retrieval accuracy matters more than cost for your use case, text-embedding-3-large steps up to 3,072 dimensions and scores around 64.6 on the MTEB benchmark in third-party comparisons, at $0.13 per million input tokens. Both come with batch-API pricing at roughly half the standard rate if your ingestion job can tolerate asynchronous processing.

Embedding model	Provider	Dimensions	Price per 1M input tokens	Best fit
text-embedding-3-small	OpenAI	1,536	$0.02	Default choice for most retrieval pipelines
text-embedding-3-large	OpenAI	3,072	$0.13	Higher-accuracy retrieval, MTEB ~64.6
BGE-large-en-v1.5	BAAI (open source)	1,024	Free, self-hosted compute only	Fully local pipelines, no per-token cost
Nomic Embed Text v1.5	Nomic AI (open source)	768	Free, self-hosted compute only	Long-context local embedding
Cohere Embed	Cohere	Varies by model	Check Cohere’s current pricing page	Multilingual retrieval

Whichever model you pick, use the exact same one for every embedding call across the life of the collection. Vectors from two different embedding models aren’t comparable to each other, so mixing them silently corrupts similarity search instead of throwing an obvious error, which makes it a particularly nasty bug to track down after the fact.

Step 6: Set Up a Vector Database With Chroma

A vector database stores those embeddings alongside the original text and metadata, then answers nearest-neighbor queries fast even across millions of vectors. Chroma is the pragmatic pick for this tutorial because it runs embedded in your Python process with zero infrastructure to stand up, while still using the same interface you’d swap in for a hosted option later.

from langchain_chroma import Chroma

vector_store = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory=os.getenv("CHROMA_PERSIST_DIR", "./chroma_db"),
    collection_name="rag_tutorial",
)

print(f"Indexed {vector_store._collection.count()} chunks into Chroma")

That single call embeds every chunk and writes the resulting vectors to disk under chroma_db. Re-run your ingestion script later with updated documents, and Chroma persists the new collection state to the same directory, so you don’t need to re-embed content that hasn’t changed if you add incremental-update logic on top of this basic setup.

Chroma isn’t the only option, and which vector database fits depends more on your deployment constraints than on raw search quality, since the major options have converged on similar retrieval performance for most workloads.

Database	Type	Self-hosted	Managed cloud option	Best fit
Chroma	Open source	Yes	Chroma Cloud	Local dev and small-to-mid projects, used in this tutorial
Qdrant	Open source	Yes	Qdrant Cloud	High-performance filtering, Rust core
Weaviate	Open source	Yes	Weaviate Cloud	Built-in hybrid search modules
pgvector	Open source (Postgres extension)	Yes	Any managed Postgres (RDS, Supabase, Neon)	Teams already running Postgres
Pinecone	Proprietary	No	Fully managed, serverless	Teams that want zero infrastructure to operate
Milvus	Open source	Yes	Zilliz Cloud	Billion-scale vector search

If your data already lives in Postgres, pgvector is worth a serious look before adding a whole new database to your stack purely for vector search. For everything else, the swap from Chroma to any of the others in LangChain is mostly a matter of changing which integration class you import.

Step 7: Build the Retrieval Function

With chunks indexed, retrieval is a matter of embedding the incoming question with the same model you used for the documents, then asking the vector store for the nearest matches.

def get_retriever(store, k=4):
    return store.as_retriever(
        search_type="mmr",
        search_kwargs={"k": k, "fetch_k": 20, "lambda_mult": 0.5},
    )

retriever = get_retriever(vector_store)
results = retriever.invoke("What is the refund window for a defective unit?")

for i, doc in enumerate(results, start=1):
    source = doc.metadata.get("source", "unknown")
    print(f"[{i}] ({source}) {doc.page_content[:120]}...")

search_type="mmr" uses Maximal Marginal Relevance instead of plain top-k similarity. Plain similarity search can return four chunks that all say nearly the same thing if your corpus has redundant documentation, which wastes context window on repetition. MMR pulls back results that are still relevant but more different from each other, which tends to give the generator broader coverage of the answer instead of four near-duplicates of the same paragraph.

Example output against a small internal-docs corpus:

[1] (data/docs/returns-policy.md) Defective units may be returned within
30 days of delivery for a full refund, provided the original packaging...
[2] (data/docs/returns-policy.md) Non-defective returns follow a separate
14-day window and are subject to a restocking fee of...
[3] (data/docs/shipping-faq.md) Replacement units ship within 2 business
days of a confirmed defect report, using the same shipping method...

Step 8: Connect Retrieval to an LLM for Generation

This is where retrieval and generation actually meet. The retrieved chunks get formatted into a prompt alongside the original question, and the LLM writes an answer constrained to that context instead of pulling from its own training data.

from langchain.chat_models import init_chat_model
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough

PROMPT = ChatPromptTemplate.from_template(
    """Answer the question using only the context below.
If the context doesn't contain the answer, say you don't know
instead of guessing. Cite sources using the [n] markers shown.

Context:
{context}

Question: {question}"""
)

llm = init_chat_model(
    os.getenv("CHAT_MODEL", "gpt-4o-mini"),
    model_provider="openai",
    temperature=0,
)

def format_docs(docs):
    return "\n\n".join(f"[{i}] {d.page_content}" for i, d in enumerate(docs, start=1))

rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | PROMPT
    | llm
    | StrOutputParser()
)

Two details in that prompt do most of the work. Telling the model to say “I don’t know” instead of guessing cuts down on confident-sounding hallucinated answers when the retriever comes up empty. Numbering the sources with [n] markers gives the model a simple citation format to echo back, which is what turns a plain chat answer into something a user can actually verify.

init_chat_model is intentionally provider-agnostic here. The CHAT_MODEL environment variable and model_provider argument are the only two lines you’d touch to point this pipeline at Claude, Gemini, or a self-hosted model instead of OpenAI. If you’re building against Anthropic’s API specifically, our Claude API tutorial walks through authentication and request structure in more depth than fits here.

Whether to call a hosted API or serve generation from a self-hosted model is really a cost-versus-control decision, not a technical one, since both paths plug into the same chain shown above. A hosted API means no infrastructure to manage and access to whichever frontier model your provider offers, at a per-token cost that scales with usage. Self-hosting trades that convenience for a fixed compute cost and full control over data residency, which matters if your document corpus includes anything you can’t send to a third-party API in the first place.

Step 9: Assemble the End-to-End RAG Pipeline

With every piece built and tested individually, the last step is combining ingestion and querying into one reusable script instead of scattered notebook cells. This is the file the rest of the project, including the API wrapper in Step 11, imports from.

"""rag_pipeline.py - end-to-end RAG pipeline for local documents."""
import os

from dotenv import load_dotenv
from langchain_community.document_loaders import DirectoryLoader, TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_chroma import Chroma
from langchain.chat_models import init_chat_model
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough

load_dotenv()


def build_vector_store(data_dir="./data/docs", persist_dir="./chroma_db"):
    loader = DirectoryLoader(data_dir, glob="**/*.md", loader_cls=TextLoader)
    raw_docs = loader.load()

    splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=75)
    chunks = splitter.split_documents(raw_docs)

    embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
    return Chroma.from_documents(
        documents=chunks,
        embedding=embeddings,
        persist_directory=persist_dir,
        collection_name="rag_tutorial",
    )


def build_rag_chain(vector_store):
    retriever = vector_store.as_retriever(
        search_type="mmr",
        search_kwargs={"k": 4, "fetch_k": 20, "lambda_mult": 0.5},
    )

    prompt = ChatPromptTemplate.from_template(
        """Answer using only the context below. Say "I don't know" if the
context doesn't contain the answer.

Context:
{context}

Question: {question}"""
    )

    llm = init_chat_model(
        os.getenv("CHAT_MODEL", "gpt-4o-mini"),
        model_provider="openai",
        temperature=0,
    )

    def format_docs(docs):
        return "\n\n".join(d.page_content for d in docs)

    return (
        {"context": retriever | format_docs, "question": RunnablePassthrough()}
        | prompt
        | llm
        | StrOutputParser()
    )


if __name__ == "__main__":
    store = build_vector_store()
    chain = build_rag_chain(store)

    question = "What is the refund window for a defective unit?"
    answer = chain.invoke(question)
    print(f"Q: {question}\nA: {answer}")

Running that script directly prints a grounded answer built entirely from your own documents:

Q: What is the refund window for a defective unit?
A: Defective units can be returned within 30 days of delivery for a
full refund, as long as the original packaging is included [1].
Replacement units ship within 2 business days of a confirmed
defect report [3].

Step 10: Evaluate Retrieval and Answer Quality

A pipeline that runs without errors isn’t the same as one that answers correctly. Retrieval can quietly fail, pulling back plausible-sounding but wrong chunks, and generation can quietly fail too, ignoring good context and answering from the model’s own training data instead. RAGAS is built specifically to separate those two failure modes instead of just scoring the final answer as right or wrong.

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall
from datasets import Dataset

eval_set = Dataset.from_dict({
    "question": questions,
    "answer": generated_answers,
    "contexts": retrieved_contexts,
    "ground_truth": reference_answers,
})

results = evaluate(
    eval_set,
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
)
print(results)

Each metric isolates a different failure point. Faithfulness checks whether the generated answer is actually supported by the retrieved context, catching hallucination even when the retrieved chunks were correct. Context precision and context recall score the retriever on its own, independent of anything the LLM does afterward, telling you whether the right chunks made it into the context window at all. Answer relevancy flags cases where the response is faithful to the context but doesn’t actually address what was asked.

Example scores from a 20-question evaluation set built against the returns-policy corpus used earlier in this tutorial:

{'faithfulness': 0.91, 'answer_relevancy': 0.88,
 'context_precision': 0.85, 'context_recall': 0.93}

A low context_recall score with a high faithfulness score points at a retrieval problem: tune chunk size, overlap, or k before touching the prompt. The reverse pattern, good retrieval with low faithfulness, points at a generation problem: tighten the prompt’s instructions or lower the temperature before assuming the retriever needs work.

Step 11: Wrap the Pipeline in a Simple API

A script that runs from the command line is fine for testing, but most pipelines like this one end up serving requests from a web app, a Slack bot, or another internal service. Wrapping the pipeline in a small FastAPI app makes it callable over HTTP without changing anything about the underlying retrieval or generation logic.

from fastapi import FastAPI
from pydantic import BaseModel

from rag_pipeline import build_vector_store, build_rag_chain

app = FastAPI(title="RAG Pipeline API")
store = build_vector_store()
chain = build_rag_chain(store)


class Query(BaseModel):
    question: str


@app.post("/ask")
def ask(query: Query):
    return {"question": query.question, "answer": chain.invoke(query.question)}

Building the vector store and chain once at startup, rather than inside the request handler, matters more than it looks. Rebuilding either one per request means re-embedding and re-indexing your entire corpus on every single API call, which turns a sub-second endpoint into one that times out.

uvicorn api:app --reload --port 8000

Test it with a single curl request:

curl -X POST http://localhost:8000/ask \
  -H "Content-Type: application/json" \
  -d '{"question": "What is the refund window for a defective unit?"}'

{
  "question": "What is the refund window for a defective unit?",
  "answer": "Defective units can be returned within 30 days of delivery
  for a full refund, as long as the original packaging is included [1]."
}

Step 12: Test the Complete Project End to End

Before calling the pipeline done, run through the full path once with fresh eyes: a clean environment, a real corpus, and questions you haven’t already tested against. Delete the chroma_db directory, re-run ingestion from scratch, then fire a batch of questions at the API rather than just the one example used throughout this guide.

Three checks catch most issues before they reach production. First, ask a question with an answer nowhere in your corpus, and confirm the model says it doesn’t know instead of inventing a plausible-sounding answer. Second, ask a question with an ambiguous answer split across two documents, and check that MMR retrieval pulls back both instead of two near-duplicate chunks from the same source. Third, time a handful of requests, since a production system that takes 8 seconds per query needs streaming or caching before it’s usable in a real chat interface, a topic covered in the advanced tips section below.

At this point you have a full working project: document ingestion, chunking, embedding, vector storage, retrieval, grounded generation, evaluation, and an HTTP API, all in a handful of files that fit together as rag_pipeline.py plus a thin api.py on top.

RAG Framework Comparison: LangChain vs LlamaIndex vs Haystack

This tutorial uses LangChain because its ecosystem of integrations makes it a safe general-purpose default, but it isn’t the only framework worth knowing, and picking the right one depends on what kind of RAG pipeline you’re actually building.

Framework	Current version (mid-2026)	Typical framework overhead	Best fit
LangChain	1.3.x (langchain-core 1.0.1)	~10 ms	General orchestration, largest integration ecosystem
LangGraph	1.2.2	~14 ms	Multi-step agentic RAG workflows
LlamaIndex	0.14.x (v0.14.23)	~6 ms	Document-heavy indexing and retrieval
Haystack	2.29.0	~5.9 ms	Production pipelines with built-in eval tooling

Those overhead figures come from one 2026 framework benchmark comparison and shouldn’t be read as universal, since actual latency depends far more on your embedding model, vector database, and network round trips than on framework overhead itself. The same comparison found LlamaIndex answering benchmark queries in around 0.8 seconds against roughly 1.2 seconds for LangChain in one tested scenario, with retrieval accuracy around 92 percent versus 85 percent, though both numbers are scenario-specific rather than a general verdict on either framework.

LlamaIndex leans toward being the sharper tool specifically for indexing and retrieving large document sets, which is why plenty of teams reach for it when the project is “search my documents” and not much else. LangChain, especially paired with LangGraph, is the better fit once the project grows into a multi-step agent that needs to retrieve, call tools, and reason across several turns rather than answer a single question once. LlamaCloud, LlamaIndex’s managed parsing and indexing service, offers a free tier of 10,000 credits per month with paid tiers starting at $50 per month, which is worth knowing if document parsing quality (especially for messy PDFs) turns out to be your pipeline’s actual bottleneck rather than the retrieval or generation logic.

Haystack is worth a look if evaluation and observability matter as much to your team as the pipeline itself, since it ships with pipeline-level tracing and eval tooling built in rather than bolted on. None of these choices are permanent. The retrieval and generation concepts in this tutorial carry over almost unchanged if you swap LangChain for LlamaIndex later, since the underlying vector database and LLM calls stay the same either way.

A reasonable rule of thumb: default to LangChain unless something specific pulls you elsewhere. Pick LlamaIndex if your project is primarily about parsing and indexing a large or messy document set, especially PDFs with tables and scanned pages. Pick Haystack if your organization already leans on structured pipeline configs and wants evaluation wired in from day one rather than added later. Reach for LangGraph, on top of either, the moment “retrieve then answer” stops describing what you’re building and “decide, retrieve, maybe retrieve again, then answer” does.

5 Common RAG Pipeline Pitfalls (and How to Avoid Them)

Chunking without testing chunk size against real questions. A chunk size that works well for FAQ-style documents can fail badly on dense technical specs with long tables. Test retrieval against a sample of real questions before locking in a chunk size, rather than picking one number and assuming it generalizes.
Skipping metadata during ingestion. Dropping source file, page number, or section headers at chunk time makes citations impossible to reconstruct later. Attach that metadata once, during Step 3, since retrofitting it after thousands of chunks are already indexed means re-ingesting everything from scratch.
Mixing embedding models within one collection. Re-embedding half your corpus with a new model while leaving the rest on the old one silently corrupts similarity search instead of raising an error. Re-index the entire collection any time you change embedding models.
Evaluating only the final answer, never the retrieval step. A wrong answer can come from bad retrieval, a good retrieval the LLM ignored, or both. Score context precision and recall separately from faithfulness, as covered in Step 10, or you’ll waste time tuning the wrong half of the pipeline.
No deduplication of near-identical source documents. Indexing five slightly different versions of the same policy document floods the retriever with redundant chunks, crowding out genuinely different content from the top-k results. Deduplicate or version your corpus before ingestion, not after debugging a retrieval quality complaint.
Treating retrieval-augmented generation as a one-time setup instead of a maintained system. Source documents change, and a system that never re-indexes drifts out of date just as surely as a model that’s never fine-tuned again. Schedule ingestion to run on a cadence that matches how often your underlying documents actually change.

Troubleshooting: 10 Common RAG Errors and Fixes

Most RAG pipeline failures fall into a small set of recurring patterns. Here’s what to check first for each one.

Symptom	Likely cause	Fix
Empty or irrelevant retrieval results	Different embedding model used at query time than at index time	Confirm the exact same model name loads for both ingestion and querying
“Rate limit exceeded” during ingestion	Too many embedding calls fired concurrently	Batch requests and add exponential backoff with retries
Chroma “collection already exists” error	Re-running ingestion without clearing the old collection	Delete the persist directory or use get_or_create_collection
LLM answers ignore retrieved context	Prompt doesn’t clearly restrict the model to the provided context	Tighten the prompt wording, lower temperature to 0
High latency per query	Large top-k, no caching, synchronous embedding calls	Reduce k, add a caching layer, switch to async calls
Answer cites the wrong source	No metadata attached to chunks during ingestion	Attach source and page metadata at chunk creation time
Out-of-memory error while embedding a large corpus	Entire corpus embedded in one batch	Stream documents and embed and insert in smaller batches
ragas.evaluate() throws a schema error	Eval dataset missing a required column, like contexts or ground_truth	Match the exact column names each selected metric requires
Vector search returns near-duplicate chunks	Chunk overlap too aggressive, or duplicate source documents	Reduce overlap and deduplicate the source corpus before indexing
API endpoint times out on first request	Vector store or embedding model loaded lazily on first call	Warm up the vector store and model at app startup, not per request

Advanced Tips for Production RAG Pipelines

The 12 steps above cover a working pipeline. A few extensions are worth knowing about once you’re moving past a prototype toward something real users depend on.

Hybrid search. Combine dense vector search with a sparse keyword method like BM25, then merge the two ranked lists. Pure semantic search sometimes misses exact matches on product codes, error strings, or acronyms that a keyword search catches immediately, and hybrid search covers both failure modes at once.
Re-ranking. Retrieve a wider initial set, say the top 20 chunks, then run a cross-encoder re-ranker over just that set to reorder them by relevance before passing the top few to the LLM. This catches cases where the fast approximate vector search ranked a genuinely better match a few spots too low.
Query rewriting. Short, ambiguous, or poorly phrased user questions often retrieve worse than a cleaned-up version of the same question. Running the raw query through the LLM once to expand or clarify it before embedding, sometimes called HyDE when the rewrite generates a hypothetical answer first, measurably improves retrieval on conversational or vague inputs.
Caching. Cache embeddings for documents that don’t change and cache full answers for frequently repeated questions. This is often the single biggest lever on both latency and API cost in a pipeline that sees the same handful of questions repeatedly, which is common in customer support use cases.
Streaming responses. Swap chain.invoke() for chain.stream() in the API layer so tokens reach the client as they’re generated instead of all at once at the end. For a chat-style interface, this cuts perceived latency dramatically even when the total generation time is unchanged.
Agentic RAG with LangGraph. Once a pipeline needs to decide whether to retrieve at all, retrieve from multiple sources, or call other tools before answering, a fixed retrieve-then-generate chain stops being flexible enough. LangGraph models that as a graph of nodes and conditional edges instead of a single linear chain.

RAG and fine-tuning aren’t competing techniques, and production systems increasingly use both. Our guide on fine-tuning an LLM with LoRA covers the complementary case: a model fine-tuned to follow your exact output format or tone, pulling its facts from documents retrieved through the RAG pipeline built here.

Frequently Asked Questions

What’s the difference between RAG and fine-tuning?
RAG retrieves relevant documents at query time and feeds them into the prompt, which suits knowledge that changes often. Fine-tuning changes the model’s weights directly, which suits shifting its behavior, tone, or output format rather than its factual knowledge. Many production systems combine both.

How much does it cost to run a RAG pipeline in production?
Costs split across three places: embedding the corpus once (and again on updates), a chat completion per query, and vector database hosting if you’re not self-hosting. At OpenAI’s $0.02 per million tokens for text-embedding-3-small, indexing a few hundred pages of documentation typically costs well under a dollar. Per-query generation cost depends entirely on which chat model you connect in Step 8.

What chunk size should I use?
Start around 300 to 500 tokens with 10 to 20 percent overlap, then adjust based on evaluation results from Step 10, not intuition. Dense technical content with tables often needs larger chunks, while short FAQ-style content usually retrieves better with smaller ones.

Do I need a vector database, or can I use plain cosine similarity in Python?
For a corpus of a few hundred chunks, brute-force cosine similarity in NumPy works fine and avoids adding a dependency. Past a few thousand chunks, a proper vector database’s indexing (HNSW or similar) becomes necessary for retrieval to stay fast.

Which embedding model should I pick?
text-embedding-3-small is a reasonable default for most projects. Step up to text-embedding-3-large if evaluation scores show retrieval accuracy is the bottleneck, or to an open source model like BGE if you need everything running locally with no per-token cost.

How do I know if my RAG pipeline is actually working?
Run the RAGAS evaluation from Step 10 against a set of real questions with known correct answers, not just a handful of examples you tested by hand during development. Track faithfulness and context recall specifically, since those two metrics catch the most common failure modes.

Can I build RAG without LangChain or LlamaIndex?
Yes. A RAG pipeline is fundamentally an embedding call, a vector similarity search, and a chat completion call, all of which you can wire up directly against provider SDKs. Frameworks add convenience and standard patterns like MMR retrieval, not capabilities you can’t otherwise reach.

Is RAG still relevant now that context windows are so much larger?
Yes, for cost, freshness, and traceability reasons covered earlier in this guide. Stuffing an entire knowledge base into every prompt is both expensive and imprecise compared with retrieving just the passages a specific question actually needs.

How do I add new documents without re-indexing everything?
Run the ingestion step from Step 3 through Step 6 against only the new or changed files, then add the resulting chunks to the existing Chroma collection instead of rebuilding it. For anything beyond a handful of files, track which source documents have already been indexed (a simple hash-of-file-contents check works) so a scheduled ingestion job skips files that haven’t changed.

What’s a good first real project to build with this pattern?
Internal documentation search is the easiest starting point, since the corpus is small, the audience is forgiving of rough edges, and mistakes don’t reach customers. A support-ticket assistant that answers from your help center content is a natural second step once the basic pipeline is proven out.

Related Coverage

Nadia Dubois

AI & Innovation Editor

Nadia Dubois is the AI & Innovation Editor at Tech Insider, where she tracks the rapid evolution of artificial intelligence, from foundation models to real-world enterprise deployment. She previously covered AI and startups for La Tribune and contributed to MIT Technology Review's European coverage. Nadia specializes in generative AI, AI regulation, and the intersection of technology and European industrial policy. She holds a dual degree in Computational Linguistics and Journalism from Sciences Po Paris.

View all articles