Phase 5: Evaluation & Quality for AI Products · 45 min · Rubric design · Inter-rater reliability · DeepEval
The Problem
A team building an AI-powered meeting-notes generator sets up a human eval: five reviewers rate each summary "1 to 5, how good is this?" The results come back and the average score is 3.4. Leadership asks whether that's good. Nobody can answer, because a 3.4 on an undefined scale means nothing — one reviewer's 3 is another reviewer's 4, and neither of them can say precisely what separates a 3 from a 4 without re-reading the output and reconstructing a private, unwritten standard on the spot. Worse, the same reviewer might score two nearly identical outputs differently on two different days. This isn't a measurement — it's a mood ring with a number on it.
The fix a mature eval program uses is a rubric: a scoring instrument with explicit criteria, each with written anchors describing exactly what a given score means in concrete, checkable terms. A rubric turns "how good is this" — an unanswerable question — into a small set of answerable ones: does it accurately represent the source, does it omit anything decision-relevant, is it free of fabricated claims, is the tone appropriate. Score each criterion, then combine.
Rubric-based scoring vs. holistic scoring
Holistic scoring asks a reviewer to hold every dimension of quality in their head at once and collapse it to one number. It's fast to build and almost useless to act on — a low holistic score doesn't tell you what to fix, and it doesn't reproduce across reviewers.
Rubric-based scoring decomposes quality into named criteria that map to something the product team can actually act on. For a meeting-notes generator, that might be:
| Criterion | What it measures | Weight |
|---|
| Factual accuracy | Does every claim in the summary trace back to something actually said in the meeting? | 40% |
| Completeness | Are all action items and decisions captured, with no material omission? | 30% |
| Attribution accuracy | Are action items assigned to the right person? | 20% |
| Tone/formatting | Is it scannable and appropriately professional? | 10% |
Notice the weights aren't arbitrary — factual accuracy is weighted highest because a fabricated claim in a meeting summary is a trust-destroying failure, while a formatting nit is a minor annoyance. Setting these weights is PM work: it encodes what the business actually cares about into the measurement itself.
Writing scoring anchors
A criterion without a written anchor collapses right back into holistic scoring under a different name. "Factual accuracy: 1-5" still lets two reviewers land on different numbers for the same output. Anchors fix this by describing, in concrete terms, what each score level actually looks like:
- 5 — Fully accurate. Every claim traces to something explicitly stated in the meeting. No inference presented as fact.
- 3 — Mostly accurate with a minor issue. One claim is a reasonable inference not explicitly stated, but not misleading.
- 1 — Contains a fabrication. At least one claim states something as fact that did not occur in the meeting (e.g., inventing a decision, a commitment, or a number).
A good anchor set is specific enough that two reviewers reading the same output independently land on the same score most of the time. If your anchors require the reviewer to already share your judgment to apply them consistently, they're not anchors yet — they're a restatement of the criterion name.
A pass/fail threshold matters more than a fine-grained scale
Teams often over-invest in a precise 1-10 scale and under-invest in defining what "acceptable" means. In practice, most eval programs only need two decisions per example: does this pass the bar, and if not, which criterion failed. A well-anchored 1-5 scale with an explicit rule ("any criterion scoring 1 fails regardless of the weighted average") gives you both a nuanced score for tracking trends AND a binary ship-relevant signal, without requiring reviewers to agree on the difference between a 6 and a 7 on a scale nobody defined.
Inter-rater reliability, at a PM level
Once a rubric exists, the next question is whether it actually works: do two independent reviewers scoring the same outputs land on similar scores? This is inter-rater reliability (IRR) — the degree to which raters agree, beyond what you'd expect from chance.
You don't need to compute this by hand. Cohen's kappa is the standard metric data science will report: it ranges from -1 to 1, where 0 means agreement is no better than random guessing and 1 means perfect agreement. As a rough reading guide a PM can use without doing the math: below 0.4 is weak agreement (the rubric is broken — anchors need rewriting), 0.4–0.6 is moderate (usable but noisy), 0.6–0.8 is substantial (a solid production rubric), and above 0.8 is near-perfect (rare, and worth double-checking the task isn't trivially easy).
Low IRR is diagnostic, not just a bad score to report upward. It almost always means the anchors are ambiguous, not that the reviewers are careless — the fix is to rewrite the anchor with a more concrete example, not to lecture reviewers about being more careful.
Sampling strategy for human review
Nobody has budget for a human to read every production output forever. A defensible sampling strategy balances coverage against reviewer time:
- Stratified sampling across known categories (the same categories from your golden dataset — happy path, edge case, known failure mode) so rare-but-important cases aren't drowned out by high-volume easy cases
- Overlap sampling — a subset (commonly 10-20%) scored by two or more reviewers specifically to measure IRR, not for coverage
- Triggered sampling — automatically routing low-confidence automated scores (from Lesson 3's LLM-as-judge pipeline) to human review, concentrating scarce human attention where it's most needed
A team's eval rubric asks reviewers to score "helpfulness" on a scale of 1-5 with no written description of what each number means. Two reviewers score the same 50 outputs, and their scores agree on only 44% of examples. Most of their scores cluster on 3 and 4, so two reviewers guessing would agree almost that often by chance: Cohen's kappa, which corrects for that, comes out close to zero. What is the most likely root cause, and what's the fix?
Near-chance agreement on a scale with no written anchors is a strong signal that the instrument itself is broken, not that the reviewers are careless. Without anchors, "helpfulness: 4" means whatever each reviewer privately decides it means, and two careful, attentive reviewers will still diverge because they're applying different unwritten standards. Adding more score levels makes this worse, not better -- it gives reviewers even more room to disagree without narrowing what any given number means. The fix is to write concrete anchors (what does a 5 look like, what specifically makes something a 2) so the criterion becomes checkable rather than a vibe.
Reading the signal in an interview
When asked to design an eval rubric on the spot, the strongest candidates do four things fast: decompose the vague ask ("is this good") into 3-5 named, weighted criteria; write at least one concrete anchor per criterion so it's clear what separates scores; propose a pass/fail threshold, not just a weighted average, because a single catastrophic failure (a fabrication) shouldn't be averaged away by high scores elsewhere; and mention checking inter-rater reliability on a sample before trusting the rubric at scale. Candidates who skip straight to "I'd have humans rate it 1-5" without any of this structure are demonstrating they've consumed eval results, not built an eval program.
A rubric weights factual accuracy at 40%, completeness at 30%, attribution at 20%, and tone at 10%, then ships a summary that scores 5/5 on completeness, attribution, and tone but 1/5 on factual accuracy (it contains one fabricated claim). The weighted average comes out to roughly 3.4/5 -- a passing-looking score. What does this reveal about relying only on a weighted average, and what should the rubric also include?
A weighted average is the wrong tool for catching a single catastrophic failure, because by construction it blends a severe problem in one dimension with strong performance in others into a moderate-looking overall number -- exactly what happened here, where a fabricated claim (the most trust-destroying failure type for this product) got diluted into a 3.4 that looks passable at a glance. The fix isn't to eliminate the other criteria (completeness, attribution, and tone are still useful for tracking trends) -- it's to add an explicit pass/fail gate alongside the weighted average, so any output with a fabrication is flagged as failing regardless of what its other scores look like.