Phase 7: Production Observability & Incident Response · 45 min · Langfuse · Arize Phoenix · Python

Quality Monitoring at Scale

One bad score is noise. The same drop sustained for 15 minutes across the same route is a real signal — the two require completely different responses.

Hiring signal: Knowing the canonical sustained-drift threshold (a few points, held for 15-60 minutes) instead of alerting on every noisy sample is the specific statistical judgment that separates production-grade quality monitoring from a dashboard nobody trusts.

What you will learn

Explain why sampled production traces should be scored against the same rubric that gates CI
Distinguish a single noisy quality score from a sustained rolling-mean drift
Apply the canonical drift threshold: a 2-5 point drop sustained over 15-60 minutes
Explain why drift detection must be scoped per-route and per-prompt-version, not as one global average

Introduction

Quality Monitoring at Scale

An on-call engineer gets paged because a single sampled production response scored an 82 against the quality rubric, five points under the usual mid-90s baseline. They spend twenty minutes investigating, find nothing wrong, and the next ten sampled responses all score normally. This was noise — one response that happened to land on an ambiguous case the rubric scores harshly, not a real regression — and paging for every single low score trains the team to ignore pages, which is exactly how a genuine regression eventually gets missed too. A different team's dashboard shows the same route's rolling-mean quality score sitting five points below baseline for the last 40 minutes straight, across dozens of sampled responses. That's not noise. That's a real, sustained signal, and it deserves the page the single low score didn't.

Same rubric, continuous sampling

The quality rubric from c12-06-1 that gates a PR before merge doesn't stop being useful once code ships — sampling a percentage of real production traces and scoring them against that identical rubric closes the loop between "did this pass review" and "is this still behaving correctly with real traffic." A model that passed every pre-merge check can still degrade in production for reasons that have nothing to do with the code itself: a retrieved document going stale, a prompt template drifting from what was tested, real user inputs exercising edge cases nobody wrote a test for. Continuous sampling and scoring is what catches that category of drift, which no amount of pre-merge testing can, because pre-merge testing only ever sees the inputs someone thought to write tests for.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The canonical drift threshold, Why per-route, per-prompt-version, not one global average, Alerting on quality as a first-class signal, Build It — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.