Phase 4: RAG Pipelines in Workflows · 50 min · Langfuse · Ragas · Dify

RAG Evaluation and Optimization

You can't improve what you don't measure.

Hiring signal: RAG evaluation is the skill that separates practitioners from experts. Being able to define evaluation metrics (faithfulness, answer relevance, context relevance), design test sets, and run systematic optimization experiments demonstrates the rigor that senior automation and AI engineering roles require.

What you will learn

Define RAG evaluation metrics: faithfulness, answer relevance, context relevance
Build a test set of questions with expected answers for systematic evaluation
Run optimization experiments: vary chunk size, top-k, retrieval method, and measure impact
Use Langfuse and Ragas for automated RAG evaluation

The Problem

Your RAG system is in production. It auto-processes 200 queries per day. Some answers are great. Some are wrong. You don't know which is which, or why. When the product manager asks "How accurate is our RAG system?", you say "It seems to work well." That's not an answer. You need metrics, test sets, and systematic experiments to measure and improve retrieval and generation quality.

RAG evaluation is what separates practitioners from experts. It's the difference between "I built a RAG system" and "Our RAG system has 87% faithfulness, 92% answer relevance, and we improved recall@5 from 72% to 89% by switching to hybrid search with reranking."

RAG evaluation requires three metrics, not one
A good answer requires: (1) the right context was retrieved (context relevance), (2) the answer is grounded in that context (faithfulness), and (3) the answer actually addresses the question (answer relevance). Measuring all three tells you whether to fix retrieval, generation, or prompting.

The Concept

The Three RAG Metrics

┌──────────────────────────────────────────────────────────────┐
│  RAG EVALUATION METRICS                                       │
│                                                              │
│  1. CONTEXT RELEVANCE                                        │
│  "Did we retrieve the right information?"                    │
│  Measure: Is the retrieved context relevant to the query?   │
│  Fix if low: Improve chunking, retrieval, hybrid search     │
│                                                              │
│  2. FAITHFULNESS (Groundedness)                              │
│  "Is the answer grounded in the retrieved context?"          │
│  Measure: Does every claim in the answer come from context? │
│  Fix if low: Improve prompt, lower temperature, use schema  │
│                                                              │
│  3. ANSWER RELEVANCE                                         │
│  "Does the answer actually address the question?"            │
│  Measure: Is the answer relevant to the query?              │
│  Fix if low: Improve prompt, add query rewriting             │
└──────────────────────────────────────────────────────────────┘

Metric	What It Measures	Score Range	Good Score	Fix
Context relevance	Retrieval quality	0–1	> 0.8	Better retrieval
Faithfulness	No hallucination	0–1	> 0.9	Better prompting
Answer relevance	Addresses question	0–1	> 0.85	Better prompting

Your RAG system has high context relevance (0.88) but low faithfulness (0.65). What does this mean and how do you fix it?

High context relevance means the right chunks are being retrieved. Low faithfulness means the LLM is making claims not supported by the retrieved context — it's hallucinating. The fix is on the generation side: improve the system prompt to say "Answer ONLY using the provided context. If the context doesn't contain the answer, say 'I don't know.'", lower the temperature to 0, and consider using structured outputs to constrain the answer format.

Building a Test Set

test_set = [
    {
        "question": "What is the return policy for opened items?",
        "expected_answer": "14 days with 20% restocking fee",
        "expected_context_keywords": ["opened", "14 days", "restocking"],
        "category": "policy"
    },
    {
        "question": "How do I reset my password?",
        "expected_answer": "Click 'Forgot Password' on the login page",
        "expected_context_keywords": ["password", "reset", "login"],
        "category": "support"
    },
    {
        "question": "What are your shipping rates for international orders?",
        "expected_answer": "Varies by destination, starting at $25",
        "expected_context_keywords": ["shipping", "international", "rates"],
        "category": "pricing"
    },
    # ... 20+ questions covering different categories
]

Evaluation with Ragas

┌──────────────────────────────────────────────────────────────┐
│  RAGAS EVALUATION PIPELINE                                    │
│                                                              │
│  For each test question:                                     │
│  1. Run RAG: retrieve chunks + generate answer              │
│  2. Record: question, retrieved chunks, answer, ground truth│
│  3. Run Ragas metrics:                                       │
│     ├── context_precision: relevant chunks in top-k?        │
│     ├── context_recall: all needed info retrieved?          │
│     ├── faithfulness: answer grounded in context?           │
│     └── answer_relevancy: answer addresses question?        │
│  4. Aggregate scores across all test questions              │
└──────────────────────────────────────────────────────────────┘

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Exercises, Key Terms, Common Pitfalls — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.

Key terms in this lesson

RAG (Retrieval-Augmented Generation)