Phase 5: Evaluation & Quality for AI Products · 45 min · Golden datasets · Shadow testing · A/B testing
Offline vs Online Evaluation
A golden dataset tells you if the model got better in the lab. An A/B test tells you if anyone in the real world cared.
Hiring signal: PMs are frequently asked to defend a launch decision backed by a small or noisy A/B test result. Interviewers testing the 'ship responsibly under uncertainty' signal want to see a candidate who knows the difference between offline confidence (it scored well on the golden set) and online confidence (it moved a real metric, and that move wasn't noise) -- and who won't ship on an underpowered test just because the number went up.
What you will learn
- Distinguish offline evaluation (golden sets, shadow testing) from online evaluation (A/B testing) and explain what each can and can't tell you
- Explain shadow testing and why it's a lower-risk step between offline eval and a live A/B test
- Reason about sample size and statistical significance well enough to know when an A/B result is trustworthy
- Avoid the most common statistical-significance mistakes PMs make when defending a launch decision
The Problem
A team improves a product-recommendation assistant's prompt, and the golden-set eval delta from Lesson 1 looks great: faithfulness up 6 points, relevance up 4, no per-category regressions. The PM greenlights a full rollout. Two weeks later, click-through on recommendations is flat, and a support ticket surfaces that the new, more "thorough" responses are longer and users are abandoning before reading them. The golden set measured exactly what it was built to measure -- output quality against a fixed, curated set of inputs -- and it measured that correctly. It was never built to measure what real users actually do when the output changes, at the volume and diversity of real traffic, with real attention spans and real competing tasks.
This is the gap between offline evaluation and online evaluation, and an AI PM needs both, in sequence, because they answer different questions and carry different risk.
Offline Evaluation: Golden Sets and Shadow Testing
Offline evaluation runs before real users are exposed to a change. It includes everything from Lessons 1-3: golden-set scoring, human rubric review, automated LLM-as-judge metrics. Its strength is speed and safety -- you can run it as many times as you want with zero user-facing risk. Its limit is exactly what the recommendation-assistant example showed: a golden set only reflects the inputs and success criteria someone thought to include, and it says nothing about behavioral response -- whether users actually read, act on, trust, or abandon the new output.
Shadow testing is the bridge between offline eval and a live experiment. In shadow mode, the new model or prompt runs on real, live production traffic in parallel with the current production system, but its outputs are never shown to users -- they're logged and scored, often by the same automated pipeline from Lesson 3, against real-world input diversity instead of a fixed golden set. Shadow testing catches a category of problem golden sets structurally can't: distribution shift, where real traffic looks meaningfully different from what the golden set anticipated (new intents, new phrasing, seasonal patterns, adversarial users). It carries zero user-facing risk because nothing reaches a real user, which makes it the natural step after offline eval passes and before a live experiment begins.
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers Online Evaluation: A/B Testing AI Features, Statistical Significance for PMs, Sequencing: Offline, Then Shadow, Then Online, Build It, What to Practice — plus a hands-on lab, quiz, and project artifact.
Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.
Browse all courses · View pricing · DeVenture Academy