Phase 6: RAG Systems & Knowledge Grounding · 75 min · sentence-transformers · BM25 · Rank-BM25

Advanced Retrieval & Query Rewriting

The question the user asks is rarely the best query for your index. Rewrite it.

Hiring signal: Advanced retrieval architecture and query understanding

The Problem

Users ask vague, underspecified, or context-dependent questions. A naive vector search embeds the raw question and returns whatever is closest. Advanced retrieval rewrites the query, expands it, and combines retrieval strategies so the right information surfaces.

The Concept

Why the User's Query Is Not the Best Search Query

Users don't ask perfectly formulated search queries. They ask questions the way humans talk — with context assumed, shorthand used, and ambiguity that a human conversational partner would resolve from prior turns. A naive RAG system takes the user's raw question, embeds it, and searches the vector index. This often retrieves the wrong documents because the query doesn't match the language of the source documents.

Consider: a user in a chat says "What did we decide about pricing?" Without the prior conversation, "pricing" could refer to anything. The vector search will return documents that mention "pricing" generally, not the specific pricing decision from the conversation. The fix is query rewriting — using an LLM to reformulate the user's question into a self-contained, specific search query before sending it to the retrieval system.

Query Rewriting Techniques

User query →  Query rewriting →  Multiple retrieval passes →  Fusion →  LLM
              (hyde, expansion,   (vector + keyword +   (RRF, rerank)
               sub-questions)      metadata filters)

Hypothetical document embeddings (HyDE): The user asks a question. Before searching, you ask the LLM to generate a fake answer — a hypothetical document that would contain the answer. You then embed this fake answer and search for real documents that are similar to it. This works because the fake answer is likely to use the same vocabulary and structure as the real source documents, even if the user's question doesn't. For example, if the user asks "How do I fix NaN loss?", HyDE generates a paragraph about debugging NaN loss that mentions "gradient explosion," "learning rate reduction," and "gradient clipping" — terms that likely appear in the actual documentation.

Query expansion: Add synonyms or related terms to the query before searching. "How do I set a learning rate" becomes "How do I set a learning rate, step size, alpha, optimization hyperparameter." This helps when the source documents use different terminology than the user.

Sub-question decomposition: Break a complex question into simpler sub-questions, each of which can be answered by a single retrieval pass. "How does RAG compare to fine-tuning for a customer support chatbot?" becomes three sub-questions: "What is RAG?", "What is fine-tuning for chatbots?", and "RAG vs fine-tuning comparison." Each sub-question is retrieved separately, and results are merged.

Conversational context rewriting: When the user asks a follow-up question in a chat, rewrite it using prior chat history to make it self-contained. "What did we decide about pricing?" becomes "What did we decide about the pricing tier structure for the SaaS product in the Q3 planning meeting?"

The tradeoff with query rewriting is latency. Each rewriting step adds an LLM call before retrieval even starts. HyDE adds one LLM call. Sub-question decomposition adds one LLM call plus multiple retrieval passes. In production, you must decide whether the quality improvement justifies the added latency — typically 200-500ms per rewriting step.

A user asks "What did we decide about the pricing?" in the middle of a long chat. What is the main retrieval challenge?

The query "the pricing" is ambiguous without knowing what was discussed earlier. Conversational query rewriting must include prior context to make the query self-contained.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Common Pitfalls, Production Checklist, Interview Framing, Evaluation, Exercises, Key Terms — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.

Key terms in this lesson

Reranking