Phase 5: Evaluation & Quality for AI Products · 45 min · Golden datasets · Shadow testing · A/B testing
The Problem
A team improves a product-recommendation assistant's prompt, and the golden-set eval delta from Lesson 1 looks great: faithfulness up 6 points, relevance up 4, no per-category regressions. The PM greenlights a full rollout. Two weeks later, click-through on recommendations is flat, and a support ticket surfaces that the new, more "thorough" responses are longer and users are abandoning before reading them. The golden set measured exactly what it was built to measure -- output quality against a fixed, curated set of inputs -- and it measured that correctly. It was never built to measure what real users actually do when the output changes, at the volume and diversity of real traffic, with real attention spans and real competing tasks.
This is the gap between offline evaluation and online evaluation, and an AI PM needs both, in sequence, because they answer different questions and carry different risk.
Offline evaluation: golden sets and shadow testing
Offline evaluation runs before real users are exposed to a change. It includes everything from Lessons 1-3: golden-set scoring, human rubric review, automated LLM-as-judge metrics. Its strength is speed and safety -- you can run it as many times as you want with zero user-facing risk. Its limit is exactly what the recommendation-assistant example showed: a golden set only reflects the inputs and success criteria someone thought to include, and it says nothing about behavioral response -- whether users actually read, act on, trust, or abandon the new output.
Shadow testing is the bridge between offline eval and a live experiment. In shadow mode, the new model or prompt runs on real, live production traffic in parallel with the current production system, but its outputs are never shown to users -- they're logged and scored, often by the same automated pipeline from Lesson 3, against real-world input diversity instead of a fixed golden set. Shadow testing catches a category of problem golden sets structurally can't: distribution shift, where real traffic looks meaningfully different from what the golden set anticipated (new intents, new phrasing, seasonal patterns, adversarial users). It carries zero user-facing risk because nothing reaches a real user, which makes it the natural step after offline eval passes and before a live experiment begins.
Online evaluation: A/B testing AI features
A/B testing exposes a real, randomly-assigned subset of users to the new version while a control group stays on the old version, then compares outcomes on a real product metric (click-through, task completion, retention, support ticket rate) rather than an eval score. This is the only method that answers the question offline eval structurally cannot: did this change actually make the product better for real users doing real things.
A/B testing for AI features has one wrinkle traditional A/B testing usually doesn't: the output itself is often the variable, not just a UI change, so the eval instruments from Lessons 1-3 should still run inside the A/B test, scored per-arm, so a PM can see both the eval-score delta and the real behavioral-metric delta side by side. A treatment arm that wins on the business metric but loses on faithfulness is a flag worth investigating before declaring victory, not a contradiction to explain away.
Golden sets test the model. A/B tests test the product.
A common PM mistake is treating a strong offline eval delta as sufficient evidence to skip or shorten the online test. They answer different questions: the golden set says the model's outputs got better against a fixed, curated yardstick; the A/B test says users, in aggregate, behaved differently in a way that matters to the business. A change can win decisively offline and land flat or negative online (users don't read longer answers), and a change can look flat offline but win online (a small tone shift users respond to that a rubric wasn't built to detect). Neither result makes the other one unnecessary.
Statistical significance for PMs
The most common way AI PMs get burned on an A/B test is shipping on a result that looked positive but wasn't actually distinguishable from noise. A PM doesn't need to compute a p-value by hand, but needs enough intuition to ask the right question before signing off on a launch:
- Sample size matters more than the headline number. A test showing "conversion up 8%" from 40 users per arm is close to meaningless -- with that few observations, an 8% swing is easily within normal random variation. The same 8% from 40,000 users per arm is a very different claim.
- Statistical significance (commonly p < 0.05) means: if there were truly no real difference between the two versions, a result this extreme (or more extreme) would happen by chance less than 5% of the time. It does not mean "there's a 95% chance this change is good" -- it's a statement about how surprising the data would be under a no-effect assumption, not a probability the effect is real.
- Confidence intervals are more decision-useful than a single point estimate. "Conversion improved between +0.5% and +6%" tells a PM the plausible range of the true effect, including whether that range excludes zero (statistically distinguishable from no effect) and whether the low end still clears the bar to justify shipping.
- Peeking early inflates false positives. Checking a running A/B test daily and stopping the moment it crosses significance dramatically increases the odds of shipping on a fluke. The fix is deciding the sample size (or test duration) in advance and holding to it, or using a testing methodology built for early stopping (sequential testing), not eyeballing a dashboard until the number looks good.
An A/B test shows the treatment arm improving task completion by 12% after two days, with 65 users in the treatment arm and 61 in control. The PM's manager wants to ship today because the result is "clearly positive." What should the PM say?
A 12% swing on roughly 60 users per arm, checked after only two days, has two compounding problems: the sample is small enough that this swing could easily be noise rather than a real effect, and checking early and stopping the moment the number looks good ('peeking') inflates the odds of a false positive well beyond the stated significance threshold. The right move is to recommend continuing to the pre-committed sample size or duration decided before the test started -- not to abandon the online test in favor of the offline score, which answers a different question, and not to just change the traffic split, which doesn't by itself fix the underlying small-sample and early-stopping problems.
Sequencing: offline, then shadow, then online
Putting it together, a mature AI product team sequences risk deliberately: offline eval (Lessons 1-3) catches obvious regressions cheaply and fast; shadow testing validates against real traffic distribution with zero user risk; a live A/B test, sized and pre-committed in advance, is the final gate that answers whether the change actually helped the product, not just the model. Skipping straight from offline eval to a full rollout (as in the opening example) skips both the distribution-shift check shadow testing provides and the behavioral-impact check only a live test can provide.
A team wants to skip shadow testing and go straight from a strong offline golden-set score to a full production rollout, arguing shadow testing "just adds a delay for no benefit since we already know the model performs well." What is the strongest counterargument?
A golden set, no matter how well it scores, is still a fixed, curated sample chosen in advance -- it cannot reveal how the model handles real production traffic patterns nobody anticipated when the set was built. Shadow testing runs the new version on live traffic without exposing outputs to users, which is the only offline-adjacent method that can surface distribution shift before a real user is ever affected. Skipping it doesn't just add risk equal to the 'delay saved' -- it removes a specific, zero-cost detection layer for a specific class of failure a golden set structurally cannot catch.