Phase 8: Evaluation, Safety & Observability · 60 min · RAGAS · LangChain · Python
The Concept
Why RAG Needs Its Own Evaluation Framework
Evaluating a RAG system with generic LLM metrics ("is this a good response?") doesn't work because RAG has two distinct components that can each fail independently:
- Retrieval might find the wrong chunks, miss relevant chunks, or rank them poorly — and the LLM can't fix what it never receives.
- Generation might ignore the retrieved context, hallucinate beyond it, or answer a different question than what was asked — even when retrieval was perfect.
A good RAG evaluation framework measures both components separately so you can diagnose where failures occur. RAGAS (RAG Assessment) is the most widely used framework for this, providing four metrics that map directly to the two components.
The Four Core RAGAS Metrics
Your RAG system has high context recall (0.92) but low faithfulness (0.55). What does this tell you?
Context recall = 0.92 means retrieval is finding the needed information. Faithfulness = 0.55 means the answer is NOT grounded in that retrieved context — the LLM is generating claims not supported by the evidence. The fix is in the generation step: strengthen the prompt to require grounding, add citation requirements, or use a faithfulness-focused reranker.
┌─────────────────────────────────────────────────────────────┐
│ RAGAS METRICS │
│ │
│ RETRIEVAL QUALITY │
│ ├── Context Precision: Did we retrieve the RIGHT chunks? │
│ └── Context Recall: Did we retrieve ALL needed chunks? │
│ │
│ GENERATION QUALITY │
│ ├── Faithfulness: Is the answer grounded in context? │
│ └── Answer Relevancy: Does the answer address the query? │
└─────────────────────────────────────────────────────────────┘
| Metric | What It Measures | Score Range | Good Score |
|---|
| Faithfulness | Answer claims are supported by retrieved context | 0-1 | >0.9 |
| Answer Relevancy | Answer addresses the actual question | 0-1 | >0.8 |
| Context Precision | Relevant chunks ranked highly in retrieval | 0-1 | >0.7 |
| Context Recall | All needed information was retrieved | 0-1 | >0.8 |
How RAGAS Computes Each Metric
Faithfulness: Breaks the answer into individual claims, then checks each claim against the retrieved context. If a claim isn't supported, it's a hallucination.
Answer: "The company was founded in 2015 by John Smith."
→ Claim 1: "Founded in 2015" → In context? YES ✓
→ Claim 2: "Founded by John Smith" → In context? NO ✗
→ Faithfulness = 1/2 = 0.5
Answer Relevancy: Generates potential questions from the answer, then computes similarity between those generated questions and the original question. If the answer is off-topic, the generated questions won't match.
Context Precision: Checks whether relevant chunks appear at the top of the retrieval results. Uses an LLM to judge each chunk's relevance and computes a precision-weighted rank.
Context Recall: Checks whether the retrieved context contains all the information needed to answer the ground-truth answer.
Building a RAGAS Evaluation Suite
from ragas import evaluate
from ragas.metrics import (
faithfulness,
answer_relevancy,
context_precision,
context_recall,
)
from datasets import Dataset
# Your golden dataset: questions with ground-truth answers
eval_data = {
"question": [
"What is the refund policy?",
"How do I reset my password?",
"What are the pricing tiers?",
],
"answer": [
# Your RAG system's answers
"Customers can request refunds within 30 days of purchase.",
"Click 'Forgot Password' on the login page and follow the email link.",
"Free, Pro at $29/mo, and Enterprise with custom pricing.",
],
"contexts": [
# Retrieved chunks for each question
["Refund Policy: All purchases can be refunded within 30 days."],
["Password Reset: Use the 'Forgot Password' link on the login page."],
["Pricing: Free tier, Pro at $29/month, Enterprise requires contact."],
],
"ground_truth": [
# Human-written reference answers
"Customers can request a full refund within 30 days of purchase by contacting support.",
"To reset your password, click 'Forgot Password' on the login page and follow the link sent to your email.",
"There are three tiers: Free, Pro at $29/month, and Enterprise with custom pricing.",
]
}
dataset = Dataset.from_dict(eval_data)
results = evaluate(
dataset,
metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
)
print(results)
# {'faithfulness': 0.92, 'answer_relevancy': 0.88, 'context_precision': 0.85, 'context_recall': 0.90}
Creating a Golden Dataset
The hardest part of RAG evaluation is building the golden set. Three approaches:
1. Manual Curation (Gold Standard)
- Write 50-100 Q&A pairs by hand
- Include edge cases: ambiguous questions, multi-hop reasoning, out-of-scope queries
- This is your regression suite — never change it without versioning
2. Synthetic Generation
from ragas.testset import TestsetGenerator
# Generate questions from your documents
generator = TestsetGenerator()
testset = generator.generate(docs=your_documents, test_size=50)
# Produces questions with varying difficulty and ground-truth answers
3. Production Sampling
- Sample real user queries from logs
- Manually label answers as correct/incorrect
- Build a growing dataset from real distribution
The golden dataset is your contract
Your golden dataset is the single source of truth for RAG quality. Version it. Never modify existing entries (that's cheating). Only add new ones. Run RAGAS on every change to chunk size, embedding model, prompt, or retrieval strategy. If scores drop, you have a regression.
Interpreting Results
Scenario: Faithfulness drops from 0.92 to 0.71 after a prompt change
→ The new prompt is encouraging the model to "be creative"
→ It's generating claims not supported by retrieved context
→ Fix: tighten the prompt to say "Answer ONLY using the provided context"
Scenario: Context Recall drops from 0.88 to 0.62 after changing embedding model
→ The new embeddings are missing relevant chunks
→ Fix: check chunk overlap, try different chunk sizes, or revert embeddings
Scenario: Answer Relevancy is low (0.55) but Faithfulness is high (0.95)
→ The answer is grounded but doesn't address the actual question
→ Fix: improve the query rewriting or prompt to focus on the user's intent
CI/CD Integration
# eval_rag.py — run in CI on every PR
import json
from ragas import evaluate
THRESHOLDS = {
"faithfulness": 0.85,
"answer_relevancy": 0.80,
"context_precision": 0.70,
"context_recall": 0.80,
}
results = evaluate(dataset, metrics=list(THRESHOLDS.keys()))
failed = [m for m, threshold in THRESHOLDS.items() if results[m] < threshold]
if failed:
print(f"❌ RAG evaluation FAILED on: {', '.join(failed)}")
for m in failed:
print(f" {m}: {results[m]:.3f} (threshold: {THRESHOLDS[m]})")
exit(1)
else:
print("✅ All RAG metrics above threshold")
Here's a simplified, pure-Python version of the faithfulness idea shown above: split an answer into claims, then check each claim against the retrieved context using simple keyword overlap instead of an LLM judge. It's a rough approximation, but it shows the mechanics.
context = "Refund Policy: All purchases can be refunded within 30 days. Refunds go back to the original payment method."
claims = [
"Purchases can be refunded within 30 days.",
"Refunds are issued as store credit only.",
"The refund goes to the original payment method.",
]
def is_supported(claim, context, min_overlap=0.5):
claim_words = set(w.lower().strip(".,") for w in claim.split())
context_words = set(w.lower().strip(".,") for w in context.split())
overlap = claim_words & context_words
return len(overlap) / len(claim_words) >= min_overlap
supported = [is_supported(c, context) for c in claims]
faithfulness_score = sum(supported) / len(claims)
for claim, ok in zip(claims, supported):
print(f"{'SUPPORTED' if ok else 'NOT SUPPORTED'}: {claim}")
print(f"\nFaithfulness score: {faithfulness_score:.2f}")
# Try it: edit `context` or `claims` and see how the score changes.
# Real RAGAS uses an LLM to judge support instead of word overlap,
# but the claim-by-claim structure is the same idea.
Write a function that takes a dictionary of RAGAS metric scores and a dictionary of thresholds, and returns a list of metrics that failed (score below threshold).
~~~
def check_rag_thresholds(scores: dict[str, float], thresholds: dict[str, float]) -> list[str]:
# Return names of metrics where score < threshold
pass
~~~
scores = {"faithfulness": 0.92, "answer_relevancy": 0.71, "context_precision": 0.85}
thresholds = {"faithfulness": 0.85, "answer_relevancy": 0.80, "context_precision": 0.70}
assert check_rag_thresholds(scores, thresholds) == ["answer_relevancy"]
assert check_rag_thresholds({"a": 0.9, "b": 0.9}, {"a": 0.8, "b": 0.8}) == []
assert check_rag_thresholds({"a": 0.5}, {"a": 0.6}) == ["a"]
print("check_rag_thresholds OK")