Phase 5: Evaluation & Quality for AI Products · 45 min · DeepEval · Confident AI · Braintrust
The Problem
A team building an AI research assistant has a golden dataset of 400 examples and a human eval rubric from the last lesson. It works well, but it doesn't scale: a human review pass on 400 examples takes two reviewers the better part of a day, and the team wants to run a full eval on every prompt change, sometimes several times a day during active iteration. Waiting a day for a score turns eval-first development into eval-eventually development, and the team quietly starts skipping the review pass on "small" changes -- exactly the changes most likely to introduce a subtle regression nobody catches until production.
The fix the industry converged on is automated evaluation: scoring pipelines that estimate the same quality dimensions a human rubric would check, but computed automatically in seconds, so the full golden set can be rerun on every single change with no human in the loop for the common case. The 2026 tooling landscape here is mature -- Confident AI's DeepEval library ships 50+ pre-built metrics, and platforms like Braintrust and Langfuse layer tracing, prompt versioning, and eval hooks directly into the development workflow. But automated scores are only useful if a PM understands what they're actually measuring and where they break.
How automated metrics work
Most production automated metrics fall into two families:
Reference-based metrics compare a model's output against a known-correct reference answer using statistical or embedding-based similarity. These are fast and cheap but brittle: two answers can be semantically identical while scoring low on surface-level overlap, or superficially similar while being factually wrong in a way the metric can't detect.
Model-based metrics ("LLM-as-judge") use a separate LLM call to evaluate the output, typically against a rubric-like prompt. This is how most of DeepEval's metrics work under the hood. A few worth knowing by name, because they show up on real dashboards:
- Faithfulness — does every claim in the output logically follow from the provided source/context, with no unsupported additions? This is the automated proxy for the "factual accuracy" criterion from Lesson 2's human rubric.
- Answer relevance — does the output actually address what was asked, without padding or off-topic content?
- Contextual precision/recall (RAG-specific) — did the retrieval step surface the right supporting documents, and did the model use them?
- Bias — does the output show measurably different treatment across protected-characteristic framings of an otherwise-equivalent input?
- Toxicity — does the output contain harmful, offensive, or unsafe language?
How LLM-as-judge actually works
An LLM-as-judge metric is, mechanically, a second model call: the judge model receives the original input, the output being evaluated, and (for reference-based judge prompts) a rubric or reference answer, then returns a score and often a reasoning trace explaining it. This is powerful because it can evaluate open-ended, non-deterministic text the way a human rubric-scorer would, at machine speed and scale.
It is also a model making a probabilistic judgment, which means it inherits LLM failure modes: a judge model can be inconsistent across repeated calls on the same input, can be fooled by confident-sounding but wrong outputs (the same fabrication problem the metric is supposed to catch), and can be biased toward outputs that are stylistically similar to its own writing (a documented phenomenon sometimes called self-preference bias). None of this makes LLM-as-judge useless -- it makes it a tool that needs the same governance a human rubric needs: known failure modes, periodic calibration, and a defined scope of trust.
Automated scores need periodic human calibration, or they drift silently
A team that sets up an LLM-as-judge pipeline once and never checks it again is trusting a black box that can quietly drift out of alignment with what actually matters to users -- especially after a judge-model upgrade changes its scoring behavior with no announcement. The fix used in mature eval programs: periodically score a sample of outputs with both the automated judge and a human rubric (the same overlap-sampling idea from Lesson 2), and treat a growing gap between the two as a signal to recalibrate the judge prompt, not evidence that humans are being too strict.
The 2026 tooling landscape
A PM doesn't need to operate these tools personally, but should recognize the landscape well enough to have an informed opinion in a vendor conversation:
- DeepEval / Confident AI — an open-source Python library (DeepEval) plus a hosted platform (Confident AI) offering 50+ pre-built metrics including faithfulness, relevance, bias, and toxicity, designed to plug into CI so evals run automatically on every change.
- Braintrust — a platform centered on prompt versioning and evaluating changes against real production data, with a co-pilot ("Loop") that helps iterate on prompts against eval results directly.
- Langfuse — combines observability/tracing with prompt versioning and eval hooks, popular for teams that want a single tool covering both "what happened in production" and "how good was it."
The common thread across all three, and the practice a PM should insist on regardless of vendor: every prompt or model change ships with an eval delta, computed automatically, comparable to the last change, not a fresh anecdote.
When to trust automated scoring vs. when you need humans
Automated scoring is trustworthy for: high-volume regression checks on every code change, catching large, obvious quality drops fast, and tracking trend lines over time. It is not a full substitute for humans when: the judge metric hasn't been calibrated against human scores recently, the failure mode is subtle or highly product-specific (a domain-expert judgment call a general-purpose judge model wasn't tuned for), or the stakes of a wrong call are high enough that a false "pass" is expensive (safety-critical outputs, anything touching legal or medical claims).
A practical rule many mature eval programs use: automated scores gate the fast, frequent iteration loop; a human rubric pass (sampled, per Lesson 2's sampling strategy) gates any change before it reaches a meaningful fraction of production traffic. Neither replaces the other -- automated evals buy speed, human evals buy trust in what the automated score means.
A team's LLM-as-judge faithfulness metric has scored every prompt version at 90%+ for six months, including through a recent judge-model provider upgrade. A new PM asks when the judge was last checked against human ratings, and nobody can answer. What is the most likely risk here, and what should the PM do?
A suspiciously stable score across a provider upgrade to the judge model itself is a specific red flag, not reassurance -- judge models can change scoring behavior after an upgrade with no announcement, and a pipeline nobody has recalibrated against human judgment in six months has had no opportunity to catch that drift. The right move isn't to abandon automation (that sacrifices the speed and scale automated evals exist to provide) -- it's to run a calibration pass: score an overlap sample with both the judge and a human rubric and check whether they still agree. A growing gap is the signal to rewrite the judge prompt, not evidence that the underlying feature quality changed.
Reading the signal in an interview
Interviewers testing this signal are listening for whether a candidate treats automated scores as ground truth or as a calibrated instrument. A weak answer says "we run DeepEval and check the dashboard." A strong answer adds: which specific metrics map to which product risks, how the judge is periodically checked against human scores, and where the team has decided automated scoring alone isn't sufficient to ship. That last point -- knowing the boundary of the tool -- is usually the difference-maker.
A candidate is asked to design an automated eval pipeline for a customer-facing medical-information chatbot. They propose running DeepEval's faithfulness and toxicity metrics on every change and shipping automatically if both score above 95%. What is the strongest critique of this design?
The metrics chosen are reasonable, but the design's real flaw is treating automated scores as sufficient gatekeeping for a domain where a wrong call is genuinely dangerous. Automated LLM-as-judge scoring can be wrong in ways that are hard to detect without a human check -- a subtly incorrect medical claim can still read as fluent and 'faithful' to a judge model that isn't a domain expert. Raising the numeric bar doesn't fix this; the fix is a human review gate before high-stakes changes ship, with automated evals handling the fast iteration loop underneath it.