The Concept
Why One Retrieval Method Is Not Enough
When you build a RAG system, your first instinct is usually to use vector embeddings — encode every document chunk and every query into dense vectors, then find the closest vectors by cosine similarity. This works well for semantic matching: if a user asks "how do I handle rate limits?" and your document says "implement throttling to avoid 429 errors," the embedding model recognizes that "rate limits" and "throttling" are conceptually related, even though no words overlap. That is the strength of dense retrieval — it bridges vocabulary gaps.
But dense retrieval has a blind spot. It encodes meaning, not exact strings. If a user searches for "error code AUTH-4031" and your document contains the literal string "AUTH-4031," the embedding model may not produce a similar vector for both, because "AUTH-4031" is an arbitrary identifier with no semantic content. The same problem applies to product names, policy numbers, API endpoints, legal citations, and any other exact token that carries no distributable meaning. Dense retrieval will rank a document about authentication errors above the document that contains the exact code, because the former is semantically closer.
This is where lexical retrieval (typically BM25) comes in. BM25 is a term-matching algorithm — it scores documents by how many query terms they contain, weighted by term frequency and inverse document frequency. It does not understand semantics at all, but it is excellent at exact string matching. If the query contains "AUTH-4031" and a document contains "AUTH-4031," BM25 will find it.
Hybrid retrieval combines both signals. You run dense and lexical searches in parallel, normalize their scores to a comparable scale, and fuse them with a weighted average. The weight (often called alpha) controls how much you trust semantic versus lexical matching. An alpha of 0.7 means 70% dense, 30% lexical — a common starting point for general-purpose RAG. For domains with lots of exact identifiers (error codes, product SKUs, legal citations), you might lower alpha to give lexical search more influence.
The Retrieval Funnel
Retrieval is not a single step — it is a funnel with multiple stages, each narrowing the candidate set:
Each stage serves a specific purpose:
- Query normalization — lowercase, strip punctuation, expand abbreviations. Small but important: "AUTH-4031" and "auth-4031" should match.
- Dense retrieval — find semantically similar chunks via cosine similarity. Casts a wide net (typically top 20-50 candidates).
- Lexical retrieval — find exact term matches via BM25. Also returns top 20-50 candidates.
- Candidate merge — combine results from both retrievers, deduplicating by chunk ID.
- Reranking — a more expensive model (cross-encoder) scores each (query, document) pair jointly, reordering the top candidates by true relevance. This is where precision improves dramatically.
- Context assembly — select the final top-k chunks (typically 3-7) and format them into the prompt context.
Why Reranking Is a Separate Stage
The retriever (bi-encoder) encodes the query and each document separately — it never sees them together. This is fast (you can pre-compute document embeddings) but imprecise, because the model cannot reason about how the query relates to a specific document.
A reranker (cross-encoder) encodes the query and document together in a single forward pass. It sees the full interaction between query terms and document terms, which makes it much more accurate at judging relevance. The tradeoff is cost: a cross-encoder must run a full model inference for every (query, document) pair, so you can only afford to rerank a small number of candidates (typically 20-50), not the entire corpus.
This is why production RAG uses a two-stage architecture: retrieve 20 candidates fast (bi-encoder), then rerank to top 5 precisely (cross-encoder). You get the speed of approximate retrieval with the precision of a joint model.
Your RAG system uses only dense vector retrieval. Users searching for "error code 429" get irrelevant results because the embedding model encodes semantic meaning, not exact codes. What should you add?
Dense retrieval excels at semantic similarity ("how do I handle rate limits?" → "throttling") but misses exact keyword matches like error codes, product names, or IDs. BM25 (lexical) retrieval catches exact term matches. Hybrid retrieval combines both scores, capturing both semantic and lexical relevance — this is the production standard.
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Production Case Study, Evaluation Rubric, Common Pitfalls, Evaluation, Exercises, Interview Answer, Key Terms, Interview Framing — plus a hands-on lab, quiz, and project artifact.
Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.