Phase 5: Evaluation & Quality for AI Products · 40 min · Golden datasets · DeepEval · Confident AI
The Problem
A PM at a mid-size SaaS company ships an AI-powered "summarize this support ticket" feature. QA clicks through a dozen tickets before launch, the summaries look reasonable, the feature ships. Three weeks later, a customer success lead forwards a summary that fabricated a refund promise the customer never received. The PM's first instinct is to ask engineering to "just fix that case." Engineering fixes it. Two weeks after that, a different fabrication surfaces, in a different shape, on a different ticket type. There was never a system — just a sequence of individually patched anecdotes, and no way to know if last week's fix broke something the week before.
This is the default failure mode for AI PMs coming from traditional software. In deterministic software, "does it work" is close to binary: you write test cases, they pass or fail, and a passing test suite is durable evidence the behavior holds until someone changes the code. AI outputs are probabilistic. The same prompt against the same model can produce a materially different answer on a re-run, and a prompt or model change that improves 80% of cases can quietly regress the other 20%. "Looks good in the demo" is not evidence of anything, because a demo is a sample size of one, and it's usually a sample the PM chose because it worked.
The fix isn't more careful manual QA — it's institutional memory in the form of a system: a fixed set of test cases with known-good answers, run automatically every time the prompt, model, or retrieval pipeline changes, producing a comparable score over time instead of a fresh anecdote every time someone gets nervous.
The eval-first mindset
An eval-first AI team treats evaluation as a piece of the product, not a QA gate bolted on before launch. Concretely, that means: before a prompt or model gets touched, there is already a dataset of representative inputs and an agreed way to score the outputs — so any change is judged against the same yardstick as the last change, not against the engineer's or PM's mood that afternoon. Confident AI's own knowledge base and Braintrust's product philosophy both converge on the same practice: every prompt change ships with an eval delta — a before/after score on the same fixed test set — never a "trust me, I tried it and it seemed better."
This matters enormously for an AI PM specifically, because you are usually the person in the room who is not going to personally read every diff or every model card. What you can do is demand and read the eval delta. "We improved response quality" is not a decision-grade sentence. "We went from 84% to 91% faithfulness on the 150-ticket golden set, with no regression on the refund-policy subset" is.
Golden datasets: A reusable product asset, not a one-off test
A golden dataset is a curated, representative set of inputs paired with either a known-correct answer, a rubric for what a correct answer looks like, or both. Critically, it is not the same thing as a QA checklist that gets thrown away after launch. A well-built golden dataset:
- Represents real distribution, not just happy paths — including edge cases, ambiguous inputs, adversarial inputs, and known historical failure modes (the fabricated-refund case above should become a permanent entry, not a one-time fix)
- Gets versioned alongside the product, so you can tell whether a regression was caused by a prompt change, a model upgrade, or a shift in what users are actually asking
- Grows over time as production traffic surfaces new edge cases — this is the same "data flywheel" idea from Phase 3, applied specifically to eval
- Is owned jointly, with the PM setting what "correct" means for the product and data science building the scoring pipeline against that definition
Treating the golden set as a product asset changes the economics of every future change. Without one, every prompt tweak or model swap is a fresh, unscoped risk that has to be manually re-verified from zero. With one, a full regression check is a five-minute automated run.
The golden dataset is a durable asset, not a launch artifact
Teams that treat eval as a pre-launch QA pass throw the dataset away once the feature ships — then rebuild it from scratch, worse, under pressure, the next time something breaks in production. Teams that treat the golden dataset as a permanent product asset keep adding real production failure cases to it forever, so the set of "things we know this feature must get right" only grows. The second team ships faster after month three, not slower, because every future change gets checked against accumulated institutional memory instead of re-litigated from anecdote.
Who owns the eval system
Eval ownership splits cleanly along a line PMs should know before they get surprised by it: the PM owns what correctness means for the product — what does a good ticket summary include, what counts as an unacceptable hallucination, which edge cases matter to the business. Data science and ML engineering own how that gets measured — building the scoring pipeline, choosing between human review and automated metrics, instrumenting the eval to run in CI. A PM who tries to write the scoring code is wasting their scarcest skill; a PM who has no opinion on what correct means is abdicating the one part of this only they can define.
Eval debt
"Eval debt" is what accumulates when a team skips this system and ships on manual spot-checks instead. Its symptoms are specific and recognizable: every launch decision becomes a fresh argument with no shared evidence; regressions surface in production instead of before merge; nobody can answer "did the last change actually help" with a number; and post-incident, the team's only fix is a narrow patch for the exact failure that was reported, leaving the class of failure it belongs to completely unaddressed. Eval debt compounds the same way technical debt does — it's cheapest to pay down on lesson one of a phase, and most expensive after the third public incident.
A team ships an AI feature after a PM and two engineers each tried five prompts in a playground and agreed the outputs "looked right." Three months later, a model provider update changes output behavior, and the team has no way to tell whether quality improved, regressed, or stayed flat. What is missing, specifically?
Multiple people eyeballing outputs in a playground is still a sample size that's small, unversioned, and non-repeatable -- there's no way to rerun the exact same test after the model update and get a comparable score. What's missing is a golden dataset: a fixed, representative, versioned set of inputs with known-correct answers or scoring criteria, so "did this change help" has a measurable answer instead of a fresh, unscoped argument every time something changes upstream. More QA headcount treats the symptom (nobody re-checks manually) without fixing the root cause (no repeatable measurement).
Reading the signal in an interview
Interviewers testing this signal often ask some version of "walk me through how you'd know if this AI feature is working." A weak answer talks about user satisfaction surveys and support ticket volume — lagging, noisy signals that arrive weeks after a regression shipped. A strong answer starts with the golden dataset: what's in it, who defined correctness, how it's scored, how often it runs, and how it grows from production data. That's the difference between a candidate who has used AI products and one who has actually built the system that keeps one honest.
In an interview, a candidate is asked "how would you know if this AI feature is working well?" and answers: "I'd track user satisfaction scores and support ticket volume related to the feature." What is the strongest follow-up critique of this answer?
Satisfaction scores and ticket volume are real signals, but they are slow and diffuse -- by the time enough users are dissatisfied to move a survey number or generate a spike in tickets, the regression has already been live and hurting users for days or weeks. A strong answer leads with a golden-dataset eval system that runs on every change and catches quality regressions before or at the moment of shipping, treating satisfaction/ticket data as a downstream confirmation signal, not the primary detection mechanism. This is exactly the gap interviewers are probing for when they ask this question.