Phase 5: Evaluation & Quality for AI Products · 45 min · Hallucination rate · Demographic parity · Equalized odds
The Problem
A data science lead hands a PM a slide before a launch review: "hallucination rate: 3.2%. Demographic parity difference: 0.04. Toxicity: 0.1% flagged." The PM has thirty seconds to decide whether this is good news. Nodding along because the numbers look small is how a PM ends up defending a launch decision they didn't actually understand, in the postmortem, after something the numbers didn't measure goes wrong. The job here isn't to compute these metrics -- that's data science and ML engineering's work, correctly. The job is to know what each metric measures, what it structurally cannot catch, and which follow-up question turns a slide into an actual decision.
This lesson covers the three metric families that show up most often in a launch review for a generative AI feature: hallucination, bias/fairness, and safety/toxicity.
Hallucination rate
Hallucination rate measures the fraction of outputs containing a claim not supported by the provided source material or ground truth (this is the same underlying concept as the "faithfulness" metric from Lesson 3, usually reported as an error rate rather than a pass score). A 3.2% hallucination rate means roughly 1 in 31 outputs contained an unsupported claim, on whatever evaluation set produced that number.
The follow-up questions that turn this number into a decision:
- What set was this measured on? A hallucination rate from the golden set (Lesson 1) reflects known categories; a rate from shadow testing (Lesson 4) reflects real traffic distribution, which is usually the more trustworthy number for a launch call.
- Is the error uniform, or concentrated? A 3.2% overall rate that's actually 0.5% on routine queries and 18% on a specific query type (say, anything involving dates or numbers) is a very different risk profile than a flat 3.2% -- the aggregate number can hide exactly the category that matters most.
- What's the severity distribution, not just the rate? A hallucinated typo in a casual chat feature and a hallucinated dosage in a medical-information feature are both "hallucinations" at the same rate, but not remotely the same risk. The rate alone never tells you this -- you need the category breakdown from Lesson 1's golden set structure.
Bias & fairness metrics
Fairness metrics ask a specific, narrow question: does the system's behavior differ across groups defined by a protected or sensitive characteristic (race, gender, age, and similar), in a way that matters for the product's decisions? Two metrics show up most often:
- Demographic parity (also called statistical parity) compares the rate at which each group receives a particular outcome — for example, the rate at which a resume-screening AI recommends candidates for interview, broken out by gender. A demographic parity difference of 0.04 means the recommendation rate differs by 4 percentage points between the groups compared.
- Equalized odds is a stricter check: it requires the model's error rates (both false positive and false negative) to be similar across groups, not just the overall outcome rate. Two groups can have identical demographic parity while one group experiences a much higher false-negative rate — equalized odds is what catches that.
A small fairness gap is not automatically an acceptable one
There is no universal "safe" threshold for a demographic parity or equalized odds gap -- what's acceptable depends entirely on the stakes of the decision, the applicable legal context (hiring and lending have real regulatory teeth), and company risk tolerance. A 0.04 gap in a low-stakes content recommendation feature and a 0.04 gap in a hiring or lending decision are not the same risk, even though the number is identical. The PM's job is to ask what decision this number is attached to before deciding whether it's fine -- never to accept a number as inherently safe because it looks small.
A PM interpreting a fairness metric should always ask: which characteristic was checked (fairness metrics only catch what they're measured against — an unmeasured characteristic can still hide a real disparity), what's the intersectional picture (a system can look fair on gender alone and gender-and-age together can reveal a gap neither dimension shows separately), and what decision is this number gating (a recommendation feature and a lending decision carry very different regulatory and ethical stakes for the same-sized gap).
Safety & toxicity scoring
Toxicity/safety scoring flags outputs containing harmful, offensive, or unsafe content — hate speech, harassment, explicit content, dangerous instructions. Automated toxicity classifiers (often themselves smaller trained models, sometimes an LLM-as-judge call per Lesson 3) are generally reliable at catching overt cases and weaker at catching subtle or context-dependent harm — sarcasm, coded language, or harm that only becomes apparent in a multi-turn conversation rather than a single output in isolation.
A "0.1% flagged" toxicity number tells a PM the base rate of overt violations caught by the classifier in use. It does not tell the PM: what the classifier misses (its false-negative rate, which is much harder to measure and rarely reported with confidence), whether flagged outputs were actually blocked or just logged, or whether adversarial users can reliably route around the classifier (a red-teaming question, not a metric question — covered later in the course's responsible AI phase).
A data science lead reports: "hallucination rate is 3.2% overall, well within our 5% target." A PM is about to approve launch. What is the single most important follow-up question before approving?
An aggregate rate below a target threshold can still hide a dangerous concentration in one category -- for example, a 3.2% overall rate that is actually near-zero on routine queries and very high on a specific, high-stakes query type. Approving launch purely because the headline number clears a target, without checking whether the errors are evenly distributed or concentrated somewhere that matters more, is exactly the kind of decision that looks defensible on the slide and indefensible in the postmortem. The tooling used to compute the metric is irrelevant to whether the launch decision is sound.
Interpreting metrics you didn't compute
The through-line across all three metric families: a PM's job is never to run the computation, and always to interrogate the result before it becomes a decision. That means asking what set the metric was measured on, whether the aggregate could be hiding a concentrated problem, what the metric structurally cannot detect, and what decision or stakes this number is actually gating. A PM who can ask these four questions fluently, on the spot, in a launch review, is demonstrating exactly the "ships responsibly under uncertainty" signal interviewers are trained to probe for.
A launch review slide shows: "Demographic parity difference: 0.03 (interview recommendation rate by gender). Equalized odds: not measured." A PM is asked whether this is acceptable for a resume-screening feature. What's the strongest response?
Demographic parity checks only the overall outcome rate by group, which can look acceptably close even while the model's error rates diverge sharply by group -- exactly the gap equalized odds is designed to catch. For a hiring decision specifically, where disparate impact carries real regulatory and ethical weight, approving launch on demographic parity alone, with equalized odds explicitly not measured, is approving a launch with a known, unaddressed gap in the fairness picture. The right call is to require the equalized odds measurement before approving, not to treat a small parity number as sufficient on its own.