Phase 6: RAG Systems & Knowledge Grounding · 55 min · LlamaIndex · LangChain · Haystack
The Problem
A student builds an AI course assistant by uploading lecture notes into a vector database. The demo looks impressive: it answers “What is backpropagation?” and “Summarize week six.” Then the first real user asks, “Can I submit the capstone without a demo video if my GitHub repo is complete?” The assistant answers confidently, cites a portfolio checklist, and gets the policy wrong.
The failure did not happen because the model is useless. The failure happened because the builder never made a product decision: what knowledge is authoritative, what sources override other sources, when the system should abstain, and whether retrieval is even the right architecture for the problem.
RAG is useful when the product needs knowledge that is private, frequently changing, too large for the prompt, or legally/user-trust dependent on citations. RAG is not automatically useful for every AI feature. If the task is to rewrite text, classify a small input, transform JSON, generate a draft from user-provided context, or follow a deterministic workflow, RAG may be unnecessary complexity.
The Concept
What RAG Actually Does
Retrieval-Augmented Generation (RAG) solves a specific problem: LLMs are trained on public data up to a training cutoff date, but your application often needs private, current, or domain-specific knowledge that the model was never trained on. A customer support chatbot needs your company's actual policies. A legal assistant needs the current version of statutes. A course assistant needs the actual syllabus and grading rules — not what the model vaguely remembers about how courses generally work.
Without RAG, you have two options: (1) put all the knowledge in the system prompt (impossible when it's 10,000 pages), or (2) fine-tune the model on your data (expensive, slow to update, and the model can't cite sources). RAG is the third option: store your knowledge in a searchable index, retrieve only the relevant pieces for each query, and feed those pieces into the prompt as context. The model then generates an answer grounded in that specific, retrieved evidence.
The critical insight is that RAG is not just "search + LLM." A search engine returns links. RAG returns grounded answers with citations — or at least it should. The difference between a good RAG system and a bad one is not the embedding model or the vector database. It is the engineering decisions: what knowledge is authoritative, what sources override other sources, when the system should abstain (say "I don't know" instead of hallucinating), and how citations are verified.
When RAG Helps vs. When It Hurts
RAG helps when:
- The knowledge is private (your company's policies, customer data, internal docs)
- The knowledge changes frequently (product catalogs, pricing, regulations)
- The knowledge is too large for the context window (10,000 support articles)
- The answer needs citations for trust, legal, or compliance reasons
- Users need to verify where the answer came from
RAG is unnecessary complexity when:
- The task is text transformation (rewrite, summarize, translate — the input IS the context)
- The task is classification of a small input (spam detection, sentiment)
- The task follows a deterministic workflow (no external knowledge needed)
- The knowledge fits in the system prompt (a few pages of rules)
- The model's parametric knowledge is sufficient (general knowledge questions)
Adding RAG to a problem that doesn't need it means building an ingestion pipeline, vector database, embedding workflow, and retrieval logic — all for zero quality improvement. It adds latency, cost, and failure modes (what if retrieval returns the wrong chunk?) for no benefit.
The Two Halves of RAG
Retrieval-augmented generation has two halves:
- Retrieval finds relevant external evidence from your knowledge base.
- Generation uses that evidence to produce a grounded answer.
The simple mental model is:
The production mental model is more demanding:
This distinction matters. Many student projects stop at "vector database plus chatbot." Professional systems separate ingestion, retrieval, generation, evaluation, observability, permissions, and product UX. The ingestion pipeline (left side) runs independently — it parses documents, chunks them, attaches metadata, and builds the index. The query pipeline (right side) runs on every user request — it classifies intent, rewrites the query if needed, retrieves candidates, merges and reranks them, assembles context, generates an answer, validates citations, and logs everything.
You're building an AI feature that rewrites customer emails to be more professional. The input is the email text (always <500 words). Do you need RAG?
RAG retrieves external knowledge to ground the answer. Here, all needed input is already in the prompt (the email to rewrite). Adding RAG means building an ingestion pipeline, vector DB, and retrieval logic — all unnecessary. This is a pure text transformation task; a well-prompted LLM handles it directly.