Phase 6: RAG Systems & Knowledge Grounding · 75 min · LlamaIndex · LangChain loaders · Qdrant
The Problem
A company knowledge bot retrieves the wrong answer even though the correct PDF is in the system. The team blames the model. Then someone inspects the vector store and finds the real issue: the PDF was split every 1,000 characters, procedure headings were separated from procedure steps, page numbers were lost, and repeated footers were embedded hundreds of times.
This is the quiet failure mode of RAG. The assistant looks intelligent, but the retrieval layer is searching damaged knowledge. The model cannot cite a page number that ingestion discarded. It cannot answer from a table that parsing flattened into nonsense. It cannot respect permissions that were not stored as metadata.
The Concept
Why Chunking Determines Retrieval Quality
The embedding model converts text into a vector that captures its semantic meaning. But embedding models have a sweet spot — they work best on coherent, self-contained passages of a few hundred tokens. Embed an entire 50-page document as one vector and you get a blurry average that matches many queries poorly. Embed individual sentences and you get precise but context-free fragments that miss the bigger picture. Chunking is the art of finding the right granularity.
The fundamental tradeoff: smaller chunks give more precise retrieval (the embedding closely matches the specific passage) but may lack the context needed to answer the question. Larger chunks carry more context but the embedding becomes a diluted average that matches queries less precisely. The optimal chunk size depends on the document type, the query types, and the embedding model.
Chunking Strategies
Fixed-size chunking splits text every N tokens (e.g., 512 tokens). Simple and fast, but blindly cuts through sentences, paragraphs, and sections. A clause that spans a chunk boundary gets split — neither chunk has the complete information. This is the most common beginner approach and the most common source of retrieval failures.
Overlap-based chunking adds a sliding window (e.g., 512 tokens with 50-token overlap). The overlap ensures that content near boundaries appears in at least one complete chunk. This fixes the "split clause" problem but doesn't respect document structure — you can still cut through the middle of a paragraph or section.
Semantic chunking splits at natural boundaries — paragraph breaks, section headings, sentence boundaries. This preserves the document's logical structure: a chunk contains a complete idea, not half of one. For structured documents like contracts, manuals, or policies, this is dramatically better than fixed-size chunking.
Recursive chunking (used by LangChain) tries to split on the largest structural boundary first (sections), then smaller ones (paragraphs), then sentences, then characters — only falling back to smaller units when a chunk is still too large. This adapts to document structure automatically.
Why Metadata Is Not Optional
Chunks without metadata are untraceable text blobs. When the LLM generates an answer, the user asks "where did you get that?" — and you have nothing to show them. Metadata transforms a chunk from an anonymous passage into a traceable, filterable, auditable unit:
- Source attribution: document title, page number, section heading, URL — so citations can point back to the original.
- Access control: department, classification level, allowed roles — so retrieval can filter by user permissions (covered in lesson 09).
- Temporal freshness: last-updated date, version number — so the system can prefer recent documents over stale ones.
- Content type: policy, procedure, FAQ, spec — so the system can filter by document type when the query implies a specific type.
Good chunks are:
- Meaningful: they preserve enough context to answer a question.
- Traceable: they point back to source, page, URL, section, and update time.
- Filterable: metadata supports permissions, tenants, course modules, and document types.
- Auditable: humans can inspect representative chunks before trusting the system.
You're chunking a 50-page legal contract for RAG. You use fixed 512-token chunks with no overlap. Users complain that answers miss key clauses that span page boundaries. What's the fix?
Fixed chunks without overlap can split a clause, condition, or definition across two chunks — neither chunk has the complete information. Overlap ensures boundary-spanning content appears fully in at least one chunk. Semantic chunking (splitting at paragraph/section boundaries) is even better for structured documents like contracts.
Good chunks are:
- Meaningful: they preserve enough context to answer a question.
- Traceable: they point back to source, page, URL, section, and update time.
- Filterable: metadata supports permissions, tenants, course modules, and document types.
- Auditable: humans can inspect representative chunks before trusting the system.
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Production Checklist, Common Pitfalls, Evaluation, Exercises, Key Terms, Interview Framing — plus a hands-on lab, quiz, and project artifact.
Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.