Phase 5: Evaluation & Quality for AI Products · 45 min · Rubric design · Inter-rater reliability · DeepEval

Building an Eval Rubric

Two reviewers who disagree on what 'good' means aren't measuring quality -- they're measuring each other's mood.

Hiring signal: Interviewers testing the 'how do you evaluate a model that's sometimes wrong' signal frequently ask candidates to design a scoring rubric on the spot. A candidate who reaches for vague 1-5 'quality' scales gets filtered out fast; one who proposes anchored criteria, a pass/fail threshold, and an inter-rater reliability check demonstrates they've actually run a human eval program, not just consumed one.

What you will learn

Design a rubric-based scoring system with clear, anchored criteria instead of a vague holistic scale
Write scoring anchors specific enough that two independent raters converge on the same score
Explain inter-rater reliability at a level a PM needs to interpret it, without computing it by hand
Design a sampling strategy for human review that balances coverage against reviewer time

The Problem

A team building an AI-powered meeting-notes generator sets up a human eval: five reviewers rate each summary "1 to 5, how good is this?" The results come back and the average score is 3.4. Leadership asks whether that's good. Nobody can answer, because a 3.4 on an undefined scale means nothing — one reviewer's 3 is another reviewer's 4, and neither of them can say precisely what separates a 3 from a 4 without re-reading the output and reconstructing a private, unwritten standard on the spot. Worse, the same reviewer might score two nearly identical outputs differently on two different days. This isn't a measurement — it's a mood ring with a number on it.

The fix a mature eval program uses is a rubric: a scoring instrument with explicit criteria, each with written anchors describing exactly what a given score means in concrete, checkable terms. A rubric turns "how good is this" — an unanswerable question — into a small set of answerable ones: does it accurately represent the source, does it omit anything decision-relevant, is it free of fabricated claims, is the tone appropriate. Score each criterion, then combine.

Rubric-Based Scoring vs. Holistic Scoring

Holistic scoring asks a reviewer to hold every dimension of quality in their head at once and collapse it to one number. It's fast to build and almost useless to act on — a low holistic score doesn't tell you what to fix, and it doesn't reproduce across reviewers.

Rubric-based scoring decomposes quality into named criteria that map to something the product team can actually act on. For a meeting-notes generator, that might be:

Criterion	What it measures	Weight
Factual accuracy	Does every claim in the summary trace back to something actually said in the meeting?	40%
Completeness	Are all action items and decisions captured, with no material omission?	30%
Attribution accuracy	Are action items assigned to the right person?	20%
Tone/formatting	Is it scannable and appropriately professional?	10%

Notice the weights aren't arbitrary — factual accuracy is weighted highest because a fabricated claim in a meeting summary is a trust-destroying failure, while a formatting nit is a minor annoyance. Setting these weights is PM work: it encodes what the business actually cares about into the measurement itself.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Writing Scoring Anchors, Inter-Rater Reliability, at a PM Level, Sampling Strategy for Human Review, Reading the Signal in an Interview, Build It, What to Practice — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.