RAG: Retrieval-Augmented Generation Complete Guide

AI & LLMs AI & LLMs 9 min read 1751 words Intermediate ExcellentWiki Editorial Team

Retrieval-augmented generation (RAG) is the most impactful architectural pattern to emerge in applied LLM development. It solves the two fundamental limitations of LLMs: their knowledge cutoff (models only know what they trained on) and their tendency to hallucinate (models generate plausible-sounding but false information). RAG grounds every response in external, verifiable knowledge by retrieving relevant documents and feeding them to the model as context.

The impact is measurable. Companies implementing RAG report 40-60% reductions in hallucination rates, 30-50% improvements in answer accuracy, and the ability to deploy LLMs on proprietary data without fine-tuning. RAG has become the default architecture for knowledge-intensive applications: customer support, legal research, medical Q&A, internal documentation, and technical support.

Why RAG Over Fine-Tuning?

RAG and fine-tuning serve different purposes. RAG is for knowledge — providing the model with relevant, up-to-date information at query time. Fine-tuning is for behavior — teaching the model patterns, styles, or structures that are hard to specify through prompting.

RAG wins for knowledge tasks because it’s modular and maintainable. To add new knowledge, you add documents to the database — no retraining, no data preparation, no risk of catastrophic forgetting. To correct outdated information, you update the source documents. Fine-tuning would require a full retraining cycle for knowledge changes.

RAG also provides transparency and verifiability. Every answer can cite its source documents — the user can verify the model’s claims against the original material. This is essential in regulated industries (legal, healthcare, finance) where decisions must be auditable and attributable to specific evidence.

Core Architecture: The RAG Pipeline

A RAG system has four stages: ingestion, retrieval, augmentation, and generation. The ingestion stage processes source documents — parsing formats (PDF, HTML, Markdown, plain text), cleaning text, splitting into chunks, generating embeddings, and storing in a vector database with metadata. This is typically a batch process run when documents are added or updated.

The retrieval stage takes the user’s query, generates a query embedding (using the same embedding model used during ingestion), and searches the vector database for the most similar chunks. The search returns top-k results with similarity scores. Additional metadata filtering (by date, category, source) narrows results.

The augmentation stage inserts retrieved chunks into the prompt as context. The standard approach is “stuff” — insert all retrieved documents into a single prompt with clear separation between context and query. For larger result sets, use a context window management strategy: prioritize the most relevant chunks, truncate the least relevant if needed.

The generation stage passes the augmented prompt to the LLM, which produces a response grounded in the provided context. The system prompt should instruct the model to answer based only on the provided context and to cite specific sources when making claims.

Chunking Strategies

Document chunking is the most influential parameter in RAG quality. The chunk size, overlap, and boundary strategy determine what content is available for retrieval. Too-large chunks dilute specific information with irrelevant context. Too-small chunks lose semantic context — a chunk about “the defendant’s alibi” without the surrounding case details is meaningless.

Recursive character splitting is the default strategy. Define a hierarchy of separators (paragraphs, sentences, words, characters). Split at the highest-level separator first; if the chunk is still too large, recursively split at the next level. This produces chunks that align with natural text boundaries. LangChain’s RecursiveCharacterTextSplitter with chunk_size=1000 and chunk_overlap=200 is the most common configuration.

Semantic chunking groups text by semantic similarity rather than fixed token counts. Embed the text and split at points where adjacent sentences show low embedding similarity. This produces chunks that correspond to distinct topics or ideas. Semantic chunking improves retrieval relevance by 10-20% compared to recursive splitting but is 2-3x slower to process.

Document-aware chunking preserves document structure. Split on section headers (Markdown headers, HTML headings, LaTeX sections). This ensures each chunk has clear context (the section topic) and enables section-level filtering during retrieval. Use MarkdownHeaderTextSplitter for Markdown documents — it tracks the header hierarchy as metadata on each chunk.

Best practices: chunk at 256-512 tokens for most knowledge bases. Use 10-20% overlap between chunks to prevent information loss at boundaries. Include metadata (source document, section heading, position, date) with each chunk. For tables and code, preserve the full structure — splitting a table across chunks destroys its information value.

Embedding Models for RAG

The embedding model determines how “meaning” is captured during retrieval. Different embedding models capture different aspects of semantic similarity. The choice of embedding model is the second most important decision after chunking strategy.

OpenAI text-embedding-3-small (1536 dimensions) is the default recommendation for most RAG applications. It offers the best quality-to-cost ratio, supports dimensionality reduction, and integrates natively with the OpenAI API. For retrieval-sensitive applications, use text-embedding-3-large (3072 dimensions) and reduce to 1024 or 512 dimensions via the dimensions parameter — this often outperforms the small model’s native 1536.

Sentence Transformers provide self-hosted alternatives. BAAI/bge-large-en-v1.5 (1024 dimensions) matches OpenAI’s embedding quality on English retrieval benchmarks. intfloat/multilingual-e5-large (1024 dimensions) is the best choice for multilingual applications, supporting 100+ languages in a shared embedding space.

Cohere Embed v3 offers specialized retrieval models with built-in “search query” and “search document” prefixes that improve retrieval accuracy. Cohere’s rerank endpoint (a cross-encoder model) re-scores retrieved results by contextual relevance, providing a 10-20% improvement in ranking quality.

Retrieval Pipeline

The retrieval pipeline goes beyond simple vector similarity. A production retrieval pipeline has multiple stages: query transformation, hybrid search, filtering, and re-ranking.

Query transformation improves retrieval quality by rewriting the user’s query for better embedding similarity. Techniques include: query expansion (generate multiple paraphrases and search for all), HyDE (Hypothetical Document Embeddings — generate a hypothetical ideal document, embed it, and search using that embedding), and query decomposition (break complex queries into sub-questions and retrieve for each).

Hybrid search combines vector similarity (semantic) with keyword matching (BM25). This catches exact matches that semantic search might miss — product codes, technical terms, proper names. The results from both methods are merged using Reciprocal Rank Fusion (RRF): each result’s score from each method is 1/(rank + k), summed across methods. RRF is robust, parameter-free (k=60 is standard), and doesn’t require training.

Metadata filtering narrows results by document attributes — date range, category, author, source type. Apply filters before vector search (pre-filtering) to reduce the search space, or after (post-filtering) to guarantee filter coverage. Pre-filtering is faster but may miss relevant results outside the filter scope. Post-filtering is more accurate but slower.

Re-ranking applies a cross-encoder model to the top-50 results from the initial search. The cross-encoder computes pairwise relevance scores between the query and each result, providing much more accurate rankings than the initial embedding similarity. Re-ranking adds 50-200ms latency but improves NDCG@10 by 15-25%. Use Cohere’s Rerank API or a local cross-encoder model (BAAI/bge-reranker-v2-m3).

Advanced RAG Patterns

Multi-hop RAG breaks complex queries into sub-questions and retrieves information for each. For “What was the revenue of the company that acquired Acme Corp in 2023?”, the system first retrieves information about Acme Corp’s acquisition, extracts the acquiring company name, then retrieves that company’s revenue. Each hop uses the output of the previous hop as context.

Agentic RAG lets the LLM decide when and what to retrieve. Instead of always retrieving, the model receives a “search” tool and decides whether to call it, what to search for, and whether follow-up searches are needed. This is more flexible but requires careful guardrails — the agent can get stuck in search loops or retrieve irrelevant information.

Graph RAG combines knowledge graphs with vector retrieval. Entities and relationships are extracted from documents and stored as a graph. Retrieval traverses the graph to find related entities beyond the top-k vector matches. Microsoft’s GraphRAG (2024) showed 20-40% improvements on multi-part questions requiring information synthesis across documents.

Evaluation with RAGAS

RAGAS (Raj et al., 2024) provides a standardized evaluation framework for RAG systems. Key metrics include: context precision (are retrieved documents relevant to the query?), context recall (are all relevant documents retrieved?), faithfulness (is the generated answer supported by the retrieved documents?), and answer relevance (does the answer address the question?).

Implement RAGAS evaluation as part of your CI/CD pipeline. Create a test set of 50-200 query-answer pairs with ground truth context and expected answers. Run your RAG pipeline against each query and compute RAGAS metrics. Track these metrics over time — regressions indicate retrieval or generation quality degradation.

Production monitoring should track: retrieval latency (p50, p95), number of retrieved chunks, embedding similarity scores of retrieved results, and user feedback on answer quality. Set alerts for: average retrieval score dropping below threshold (indicates embedding drift or document quality degradation), faithfulness score dropping (indicates model or prompt issues), and context recall dropping (indicates indexing problems).

FAQ

How many documents should I retrieve? 3-5 for simple Q&A where the answer is likely in a single document. 5-10 for complex questions requiring synthesis across documents. More than 10 adds context noise and degrades generation quality — the model has more tokens to process but may miss the most relevant information. Start with top-5 and adjust based on evaluation metrics.

How do I handle dynamic or frequently updated content? Use a streaming ingestion pipeline. When content changes, re-embed only the changed documents and update the vector index. For time-sensitive applications, attach timestamps to chunks and filter by recency during retrieval. For news or live data, implement incremental embedding — process small batches frequently rather than full re-indexes.

What if my RAG system returns irrelevant results? Diagnose by testing each component independently. Test the embedding model on a small set of known query-document pairs. Test the chunking strategy — does the relevant information exist intact in the top-5 chunks? Test the re-ranking — does a cross-encoder identify better results than the initial vector search? The bottleneck is typically chunking (information is split across chunks) or embedding (the model doesn’t capture your domain’s semantic relationships).

RAG vs. fine-tuning for customer support? Use RAG as the primary architecture — it provides grounded answers that cite specific documentation. Use fine-tuning as a supplement to teach response style, tone, and standard phrases. The combination of RAG (knowledge) plus fine-tuning (behavior) is the most effective approach.

How much does RAG cost to operate? Ingestion costs are one-time: embedding tokens + vector database storage. Per-query costs: query embedding (small), vector search (near zero), LLM generation (dominant cost). A typical query with 5 retrieved chunks costs $0.002-0.02 in total. At 10,000 queries/day, this is $20-200/day. Most of the cost is the LLM generation, not the retrieval.