Phase 5: Evaluation & Quality for AI Products · 50 min · Eval dashboards · Ship/no-ship frameworks · Braintrust
The Problem
It's launch review day for "prompt v7" of an AI shopping assistant. The dashboard, built from everything in this phase, shows: faithfulness up 3 points, relevance flat, latency down 40% (a new smaller model), cost down 55%, hallucination rate up from 2.1% to 2.8% overall but down in the highest-severity category (payment/pricing claims), demographic parity difference unchanged, and the A/B test shows a statistically significant +2.1% increase in add-to-cart rate with a confidence interval that excludes zero. Engineering wants to ship today — the cost savings alone are material. A cautious teammate points at the hallucination rate going up and says hold. Both are reading the same dashboard correctly. The disagreement isn't about the numbers — it's about how to weigh them against each other, and that weighing is a PM decision, not a data science one.
This is the moment every earlier lesson in this phase has been building toward: not reading one metric, but reading a dashboard where several metrics point in different directions and a real decision is due.
The three real decisions
Most AI PMs default to thinking in binary: ship or don't ship. A mature eval program actually has three distinct decisions available, and picking the right one — not just "yes" or "no" — is often the strongest move:
- Ship — all key metrics clear their bar, no unresolved red flags from Lessons 1-5, and the online test result is trustworthy (adequately powered, statistically significant in the intended direction, confidence interval excludes zero).
- Hold — a metric that matters more than the metrics that improved is regressing, a fairness or safety flag from Lesson 5 is unresolved, or the online test isn't yet trustworthy (underpowered, not significant, or still running).
- Ship with guardrails — the aggregate case for shipping is real, but a specific, identified risk needs a mitigation that doesn't require blocking the whole launch: routing a specific high-risk category to human review, capping rollout to a percentage of traffic while monitoring the regressing metric, or shipping to a subset of use cases while excluding the one that regressed.
"Ship with guardrails" is frequently the correct answer and frequently the one candidates fail to reach for in an interview — the instinct is to collapse a nuanced situation into a binary because binary feels more decisive. It usually isn't; it's just less work.
Weigh metrics against what they're attached to, not against each other in the abstract
An aggregate hallucination rate going up while cost drops 55% is not resolvable by asking "which number is bigger." It's resolved by asking what each number is attached to: does the hallucination increase concentrate in a high-severity category (per Lesson 5's category-breakdown practice), and does the cost savings translate into something users or the business actually value enough to accept that specific risk. In the opening scenario, the hallucination increase is overall but the highest-severity category (payment/pricing claims) actually improved -- that reframes "quality got worse" into "aggregate quality metric moved in a way that doesn't track the category that matters most," which is a very different, more shippable situation.
A framework for weighing conflicting signals
A defensible ship/no-ship framework works through signals in a fixed order, because not all signals carry equal veto power:
- Hard blockers first. Any unresolved safety flag (Lesson 5), any fairness gap on a high-stakes decision missing its equalized-odds measurement, or any online test result that isn't statistically trustworthy yet (Lesson 4) — these hold the launch regardless of what else looks good. No metric elsewhere on the dashboard buys back a hard blocker.
- Category-level quality check. Don't evaluate an aggregate quality metric in isolation — check whether a regression concentrates in a high-severity category. A regression concentrated in a low-stakes category is a different decision than the same aggregate number concentrated in a high-stakes one.
- Net business impact, only after 1 and 2 clear. This is where cost, latency, and the A/B test's business-metric result actually get weighed — and only once quality and safety are confirmed acceptable, not as a way to buy back an unresolved blocker.
- Guardrail options, before defaulting to a binary. If step 2 or 3 surfaces a specific, bounded risk, ask whether a targeted guardrail (category exclusion, capped rollout, human-review routing) lets the team capture most of the upside while containing the specific risk, instead of forcing an all-or-nothing call.
Following the four-step framework, a dashboard shows: no safety flags, no fairness flags, hallucination rate up overall but down in the one high-severity category, and a statistically significant +2.1% add-to-cart lift with a trustworthy confidence interval. What is the most defensible decision?
Working the framework in order: no hard blockers exist (no safety or fairness flags, no untrustworthy online result), so step 1 clears. Step 2's category check is exactly where this case resolves -- the aggregate hallucination rate moved up, but the specific high-severity category (the one that would actually justify a hold) improved, which reframes the aggregate number as not disqualifying. With steps 1 and 2 clear, step 3's business impact is real and statistically trustworthy, supporting a ship decision. Holding purely on the aggregate number, without checking where it concentrates, is exactly the mistake the framework is built to prevent.
Writing the decision memo
A launch decision memo a cross-functional team will actually accept is short, states the decision first, and shows the reasoning trail explicitly rather than just the conclusion:
- The decision (ship / hold / ship with guardrails), stated in the first line, not buried at the end
- The dashboard summary — the key numbers, organized by the framework's four steps, not just a raw dump
- What would change the decision — the specific metric or threshold that would flip this call, so the next review has a concrete target instead of a vague "let's revisit"
- If guardrails: the specific mitigation, who owns monitoring it, and the condition under which the guardrail gets removed (full rollout) or the launch gets rolled back
A PM's launch decision memo reads: "We reviewed the dashboard and the team feels good about shipping. Numbers look strong overall." What is the primary weakness of this memo, using the standards from this lesson?
A decisive-sounding memo isn't the same as a defensible one. 'Feels good' and 'numbers look strong' are conclusions with the reasoning stripped out -- a cross-functional reviewer (or a future incident postmortem) can't tell whether hard blockers were checked, whether a quality regression was assessed for category concentration, or what specific business result justified the call. Length isn't the issue; showing the explicit reasoning trail through the framework's steps is what makes a decision memo something a team can actually verify, challenge, or learn from later.