Phase 9: Agent Evaluation & Observability · 60 min · Python · Anthropic SDK · Pydantic
The Problem
You have 500 agent outputs to evaluate after a prompt change. Human evaluation would take a week and cost thousands of dollars. You can't reduce them to exact-match metrics — the outputs are paragraphs of research synthesis, and you're evaluating dimensions like relevance, accuracy, and completeness. You need something that scales.
LLM-as-judge is the industry solution. You send each (task, output) pair to a more capable model and ask it to score the output against a rubric. At 500 evaluations, this takes minutes and costs a few dollars. At 50,000 evaluations — checking production quality every day — it's still viable. Human evaluation at that scale is not.
The catch is that LLM judges have systematic biases. They favor the first option in a comparison. They favor longer responses. They may favor outputs from their own model family. An uncalibrated judge produces confident-sounding scores that diverge significantly from human judgment. Engineering a reliable judge means designing the rubric carefully, testing for biases, and validating the judge against a held-out set of human annotations before you trust its aggregate scores.
The field is grounded in published research. G-Eval (Liu et al., 2023) showed that GPT-4-based judges can correlate well with human judgment on generation tasks such as summarization. MT-Bench (Zheng et al., 2023) uses a strong LLM as the judge; its companion, Chatbot Arena, ranks models from crowdsourced human pairwise votes — which is what let that work measure how often the LLM judge agreed with people. Wang et al. (2023), "Large Language Models are not Fair Evaluators", documented the position bias covered below. Understanding the architecture and failure modes of LLM judges is now an expected competency for senior AI engineers.
When to Use LLM-as-Judge
LLM-as-judge works best for evaluating complex, open-ended outputs where exact-match is impossible — summaries, research reports, multi-step reasoning. It is not suitable for factual verification against a known database (use exact-match or retrieval), nor for evaluating safety-critical outputs without human review. Always calibrate before trusting aggregate scores.
Judge architecture
The judge receives a structured prompt containing: the original task, the agent's output, and an explicit evaluation rubric. It returns a structured JSON object with per-criterion scores, reasoning, and a confidence estimate.
class CriterionScore(BaseModel):
criterion: str
score: int = Field(ge=1, le=5)
reasoning: str
class JudgementOutput(BaseModel):
overall_score: float = Field(ge=1.0, le=5.0)
criteria_scores: list[CriterionScore]
reasoning: str
confidence: float = Field(ge=0.0, le=1.0)
Three design decisions matter here. First, use a more capable model as the judge than the one being evaluated. A judge that is less capable than the evaluated model will miss nuanced errors. If you're evaluating Haiku outputs, use Sonnet or Opus as the judge. Second, provide explicit evaluation criteria — don't ask the judge to "rate quality." A rubric like "Accuracy: are all factual claims correct and verifiable? 5 = all facts correct, 1 = significant errors" produces far more consistent scores than open-ended quality ratings. Third, the reasoning field is not decorative. Read the reasoning on a sample of evaluations: if the reasoning doesn't match the score, the judge is confused. The reasoning is your debug window.
A five-criterion rubric for research agent outputs: Relevance, Accuracy, Completeness, Conciseness, Citation quality. Each scored 1–5. Add this instruction explicitly: "Do NOT favor responses for being longer. A concise, accurate answer is better than a verbose one." Without this, length bias will inflate scores for padded responses.
Judge bias and calibration
Three biases affect LLM judges. Position bias: in pairwise comparisons (A vs. B), the judge tends to favor whichever response appears first, even when the two are identical in quality. Length bias: judges favor longer responses unless explicitly told not to. Self-bias: a Claude judge may favor Claude-generated outputs; a GPT-4 judge may favor GPT-4 outputs. For high-stakes evaluations, use a judge from a different provider than the model being evaluated.
Position bias in LLM-as-judge comparisons means:
Position bias is one of the most well-documented biases in LLM judges. The mitigation: when comparing output A vs. B, run two evaluations — one with A first, one with B first. If A wins both, it is genuinely better. If each wins when it appears first, the scores are biased — average the two results or switch to absolute scoring instead of pairwise comparison.
Calibration is the validation step you cannot skip. Take 30–100 examples, collect human ratings for each, run the judge on the same set, then compare.
The tempting metric is "agrees with humans within 1 point" — and it is nearly meaningless on its own. On a 1–5 scale, two raters scoring at random agree within 1 point 13 times in 25 (52%), and a judge that always answers the most common human score often does better still. So compare against that do-nothing baseline, and use measures that correct for chance:
- Weighted kappa (quadratic): agreement on the 1–5 scale, corrected for chance and penalising a 1-vs-5 miss more than a 3-vs-4 one. 0 is chance; above ~0.6 is commonly read as substantial.
- Spearman correlation: does the judge rank outputs in the same order as humans, even if its scores run high or low? (Pearson measures linear agreement of the raw scores, not ranking.)
calibrator = JudgeCalibrator(judge)
report = calibrator.calibrate(annotated_examples)
# From the lesson's script, 30 annotated examples:
# within_one_rate: 0.90 <- looks great...
# baseline_within_one: 0.833 <- ...but a judge that always says the same score gets 83%
# weighted_kappa: 0.24
# spearman_correlation: 0.23
# verdict: "unreliable"
A 90% "within 1 point" judge that barely beats a constant one, with kappa and rank correlation near 0.2, does not track human judgment — it just lands near the middle of the scale. Revise the rubric, give the judge anchored examples for each score, or switch judge models, and calibrate again.
Judge calibration involves:
Calibration is the empirical validation step. You cannot assume an LLM judge is reliable — you must verify it against human ratings. Judge it with weighted kappa and Spearman correlation, and compare it with a judge that always gives the same score: "within 1 point" alone is passed by chance more than half the time on a 1–5 scale. If the judge does not clearly beat that baseline, revise the criteria, try a different judge model, or narrow the evaluation scope.
Batch evaluation at scale
Evaluating 500 outputs requires an async batch pattern plus statistical rigor. Don't just report a mean score — report a mean with a confidence interval so you know how much to trust the number.
batch_evaluator = BatchEvaluator(judge)
results, report = batch_evaluator.evaluate_batch(examples, batch_size=50)
# AggregateReport:
# mean_score: 3.82
# stdev_score: 0.61
# ci_95_lower: 3.74
# ci_95_upper: 3.90
# criteria_means: {"Relevance": 4.1, "Accuracy": 3.7, "Completeness": 3.8, ...}
The 95% confidence interval is computed via bootstrap resampling: draw 500 random samples of the same size as your evaluation set (with replacement), compute the mean for each, take the 2.5th and 97.5th percentiles. The result: "Our agent scores 3.82/5 on research quality (±0.08, 95% CI)." That is a reportable, defensible quality metric, not an impression.
Per-criterion breakdown reveals which dimension is weakest. If Citation quality is consistently 2.1/5 while everything else is 3.8+, you know exactly where to focus prompt engineering effort. Aggregate-only scores hide these patterns.