Phase 8: Evaluation, Safety & Observability · 60 min · Fairlearn · AIF360 · W&B Weave
The Problem
A lending startup ships an AI underwriting assistant. Overall accuracy: 94%, up from 89% with the old rules engine. The PM presents it at the launch review as an unambiguous win. Three months later, a journalist finds that the model denies pre-approval to self-employed applicants at more than twice the rate of salaried applicants with comparable income and credit history. The story runs under a headline that never mentions the word "accuracy" once.
Nothing in that story is a data science failure. The model does exactly what it was trained to do: minimize aggregate error. The failure is an evaluation failure — nobody measured whether the model's errors were distributed evenly across the population it serves. Responsible AI evaluation is the discipline of asking that question before launch, not after a news story.
The Concept
Accuracy hides distribution
Overall accuracy: 94%
Salaried applicants: 96% accurate, 7% false-positive rate
Self-employed: 82% accurate, 18% false-positive rate
^ hidden by the aggregate
This is the default outcome, not an edge case, whenever a group is underrepresented in training data or has a different underlying distribution. Treating "accuracy is high" as sufficient evidence of "this is safe to ship" is the single most common responsible-AI mistake.
A model achieves 94% overall accuracy on a loan pre-approval task. The false-positive rate for self-employed applicants is 18% vs 7% for salaried. What is the correct interpretation?
Aggregate accuracy can hide systematic errors for specific groups. An 11-point false-positive gap between groups is a fairness problem — the model is not equally reliable across the population it serves. This is the default outcome when a group is underrepresented in training data, not an edge case.
Consequence scanning: the one question that changes the roadmap
Before launch, ask: if this feature is wrong, who gets hurt, how badly, and how would we know?
For every feature with a decision that affects a person (approve/deny, rank/hide, flag/ignore, price up/price down):
- Who is affected by a wrong output — not "users" in aggregate, but specific groups
- What does "wrong" cost that person — a missed restaurant recommendation and a denied loan are both "errors," but one costs an afternoon and the other can cost a mortgage
- Is the harm reversible — can the person appeal, retry, or get a human review?
- Is the error distributed evenly — this is where fairness metrics enter
Fairness metrics: what to measure and why
| Metric | What it checks | What it misses |
|---|
| Demographic parity | Are outcome rates similar across groups? | Error rates — two groups can have identical outcomes while one has much higher false negatives |
| Equalized odds | Are both false-positive AND false-negative rates similar across groups? | Intersectional gaps — a system can look fair on gender alone but fail on gender × age |
| Equal opportunity | Among people who should get a positive outcome, does the model give it at similar rates across groups? | False-positive disparities — only checks one side of the error matrix |
A small fairness gap is not automatically acceptable
There is no universal "safe" threshold. What's acceptable depends on the stakes of the decision, the legal context (hiring and lending have regulatory teeth), and company risk tolerance. A 0.04 gap in content recommendations and a 0.04 gap in hiring decisions are not the same risk.
Intersectional analysis
Single-axis fairness checks can hide gaps that only appear when you cross dimensions:
Gender alone: parity gap = 0.02 (looks fine)
Age alone: parity gap = 0.03 (looks fine)
Gender × Age: parity gap = 0.11 (hidden gap!)
A system can look fair on gender and fair on age separately, while being systematically unfair to a specific gender-age combination. Always check intersectional subgroups, not just single axes.
Ship / Mitigate / Redesign
When a fairness gap is found, the response depends on severity and reversibility:
| Severity | Reversibility | Decision |
|---|
| LOW | Easily reversible | SHIP with monitoring |
| MEDIUM | Reversible | MITIGATE — rebalance data, add constraints, remeasure |
| HIGH | Hard to reverse | REDESIGN — change the feature (e.g., mandatory human review for affected group) |
| SEVERE | Any | REDESIGN — do not ship as-is |
A fairness audit finds a 9-point false-positive parity gap (above the 5-point target) for a HIGH-severity, MODERATELY-reversible outcome. The ML lead proposes rebalancing training data. What is the correct response?
HIGH severity with only moderate reversibility crosses the line from "patch the model" to "redesign the feature." A data rebalancing effort risks shipping the same harm again if it doesn't fully close the gap. The right move is a design change: route affected applicants to mandatory human review until the KPI passes with margin.
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Evaluation, Exercises, Key Terms, Common Pitfalls, Interview Framing — plus a hands-on lab, quiz, and project artifact.
Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.