The Prototype Works, The Production System Does Not
Every RAG prototype starts the same way: load some documents, split them into chunks, embed them, dump everything into a vector database, retrieve the top-k results, stuff them into a prompt, and watch the model answer questions. It works surprisingly well for demos. The production version requires solving problems that the prototype papered over.
The first problem is document parsing. Real documents are messy: PDFs with complex layouts, tables that span pages, images with embedded text, headers and footers that contaminate content, scanned documents that need OCR. A simple text extraction from a PDF will often destroy the structure that makes information retrievable. Production RAG pipelines invest heavily in document preprocessing: identifying and extracting tables, handling multi-column layouts, separating content from navigation elements, and running OCR on image-based content.
Chunking Strategy Matters More Than You Think
How you split documents has an outsized effect on retrieval quality. Too large: you include irrelevant context that dilutes signal and may exceed context limits. Too small: you lose the surrounding context needed to understand the chunk. The right strategy depends on your document types and query patterns.
For long-form content with coherent sections, hierarchical chunking - splitting at section boundaries first, then by paragraph - preserves semantic units better than fixed-size chunking. For documents with important entity relationships, entity-aware chunking can keep related information together. For tabular data, table-level chunks often work better than row-level splits, though that requires custom parsing.
Adding overlap between chunks reduces the risk of cutting relevant information at chunk boundaries. A 20% overlap between adjacent chunks is a reasonable starting point. The tradeoff is more chunks and higher embedding and retrieval costs.
Embedding and Retrieval
Embedding model selection is the highest-leverage decision in the retrieval layer. General-purpose embeddings like those from OpenAI or Cohere work well across diverse document types but may underperform on domain-specific content. For applications in legal, medical, or technical domains, fine-tuned domain-specific embeddings often significantly outperform general ones. Test your specific embeddings on your specific data rather than relying on benchmark performance.
Hybrid search - combining dense vector similarity with sparse keyword matching - consistently outperforms either approach alone. BM25 keyword search captures exact matches and domain-specific terminology that dense embeddings may miss; vector search captures semantic similarity. Fusing both retrieval streams with a reciprocal rank or learned reranking step is the current standard for high-quality retrieval.
Reranking and Generation
A retrieval reranker - a cross-encoder model that scores the relevance of each retrieved document to the query - adds significant quality for modest latency cost. First-stage vector search is optimized for speed, not accuracy. A reranker can evaluate retrieved candidates more thoroughly and reorder them before passing to the generation model. Models like Cohere Rerank and open-source alternatives have made this accessible.
The generation prompt matters as much as retrieval quality. Explicit instructions to cite sources, distinguish between information from retrieved documents and implicit knowledge, and flag when retrieved information is insufficient for a complete answer all reduce hallucination risk. The model should see the retrieved documents clearly and be told what to do when they do not contain the answer.