Phase 6: RAG Systems & Knowledge Grounding · 75 min · RAGAS · Hugging Face · sentence-transformers

RAG Evaluation & Metrics

You can't improve RAG without measuring it. Build a repeatable evaluation harness before you tune.

Hiring signal: RAG evaluation and metric-driven iteration

The Problem

Teams spend weeks tuning chunk size, embeddings, and prompts without a clear metric. The result is guesswork. RAG evaluation separates retrieval quality from generation quality and tells you which part of the pipeline is actually broken.

The Concept

Why RAG Evaluation Is Split Into Stages

A RAG system has two major stages: retrieval (finding the right documents) and generation (producing an answer from those documents). If you only measure the final answer quality, you can't tell whether a bad answer is caused by bad retrieval (the right document wasn't found) or bad generation (the right document was found but the model ignored it or hallucinated). This is why RAG evaluation is split into retrieval metrics and generation metrics — so you can isolate the problem.

Retrieval     →   Generation     →   End-to-end
────────────────────────────────────────────────
Recall@K        Faithfulness      Answer correctness
MRR             Relevance         Context precision
NDCG            Hallucination     Answer similarity

Retrieval Metrics

Retrieval metrics tell you whether your search is finding the right documents:

Recall@K: Of all the relevant documents that exist, what fraction did you retrieve in the top-K results? If recall is low, your retrieval is missing relevant content — fix chunking, embeddings, or add hybrid search.
MRR (Mean Reciprocal Rank): How high is the first relevant document ranked? If MRR is low, the right document exists in your results but it's buried at position 15 instead of position 1 — fix reranking or adjust your scoring weights.
NDCG@K: A weighted ranking quality score that gives more credit when relevant documents appear higher in the results. More nuanced than recall because position matters.

Generation Metrics

Generation metrics tell you whether the model is using the retrieved context correctly:

Faithfulness: Does the answer only contain claims that are supported by the retrieved context? If faithfulness is low, the model is hallucinating — adding information from its parametric memory that isn't in the retrieved documents. This is the most important generation metric for RAG.
Answer Relevance: Does the answer actually address the user's question? A high faithfulness score with low answer relevance means the model faithfully reported what was in the context but didn't answer the specific question asked.
Hallucination Rate: The percentage of claims in the answer that are NOT supported by retrieved context. The inverse of faithfulness.

The Golden Test Set

A golden test set is a small, curated collection of questions with ground-truth answers and the exact document chunks that should support each answer. It is the foundation of all RAG evaluation. Without it, you're optimizing by vibes — trying different settings and eyeballing whether the answers "look better." With it, you can run automated evaluations and track metrics over time as you change chunk sizes, embedding models, or prompts.

Building a golden set is manual work but pays off enormously:

Collect 50-100 real user questions (not made-up ones — actual queries from logs or user testing)
For each question, manually identify the correct answer and the exact chunks in your corpus that contain the supporting evidence
Store these as {question, answer, ground_truth_chunks} tuples
Run your RAG pipeline against this set and compute retrieval + generation metrics
Every time you change your pipeline, re-run against the same golden set to see if metrics improved or regressed

Your RAG system gives correct-sounding answers, but they often cite the wrong source. Which metric should you improve first?

Faithfulness measures whether the generated answer is grounded in the retrieved context. If the answer sounds correct but cites wrong sources, faithfulness is the first metric to fix.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Common Pitfalls, Production Checklist, Interview Framing, Evaluation, Exercises, Key Terms — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.

Key terms in this lesson

RAG (Retrieval-Augmented Generation)