Phase 8: Launch, Metrics & Scaling AI Products · 45 min · Python
Monitoring Model Drift & Post-Launch Quality
A model that was 94% accurate at launch and 89% accurate three months later didn't 'break' — it drifted, and nobody was watching the right signal.
Hiring signal: AI PM candidates who can distinguish data drift from concept drift, set threshold-based alerting that catches real regressions without false-alarming, and define retraining triggers before a model degrades in production demonstrate the post-launch operational fluency that separates someone who has shipped an AI feature from someone who has kept one healthy.
What you will learn
- Distinguish data drift, concept drift, and model decay as separate failure modes requiring different responses
- Set alerting thresholds that catch real regressions without false-alarming constantly
- Choose between scheduled retraining, threshold-triggered retraining, or both
- Read a production monitoring dashboard as a PM, not just an ML engineer
The Problem
A team ships an AI-powered customer support triage feature that classifies incoming tickets by urgency. At launch, it's 94% accurate on the eval set. The team moves on to the next feature. Three months later, a support operations manager notices that critical tickets are sitting in the queue longer than they used to. Investigation reveals the model's accuracy on critical-ticket classification has dropped from 94% to 79% — but the overall accuracy dashboard still reads 91%, because the model is still fine on the easy categories (general inquiries, password resets) that make up 80% of tickets. The model didn't break. It drifted: the types of tickets customers submit shifted (a new product launch generated unfamiliar issue types), and the model's training data didn't include them. Nobody was watching per-subtype performance, nobody set a drift alert, and the regression went unnoticed for three months because the headline metric looked fine.
This is the post-launch reality of AI features: models degrade. Not because they're buggy, but because the world changes around them. Input distributions shift, user behavior evolves, and the relationship between inputs and correct outputs can change. A traditional software feature that works at launch keeps working — the code doesn't change. An AI feature that works at launch starts degrading the moment it encounters data it wasn't trained on. The PM's job is to know this, watch for it, and have a plan before it happens.
Three Types of Drift
Understanding drift starts with distinguishing three separate failure modes that require different responses:
- Data drift (input drift): The distribution of inputs the model sees in production shifts away from what it was trained on. Example: a document summarization model trained on 2-page memos starts receiving 20-page legal contracts. The model's mapping (summarize this) may still be correct, but it's seeing inputs outside its training distribution, so quality degrades. Response: retrain with representative production data, or add input validation that flags out-of-distribution inputs.
- Concept drift: The relationship between input and correct output changes. Example: a spam classifier trained on 2023 email patterns faces new spam tactics in 2025. The inputs (emails) may look similar, but what counts as "spam" has evolved. The model's mapping is now wrong even for inputs it was trained on. Response: update labels (redefine what "correct" means), retrain with updated labels, and increase retraining cadence.
- Model decay: The model itself degrades — not because of external changes but because of internal issues. This is rare for LLM-based features (the model weights don't change) but common for custom-trained models that are updated incrementally. Response: rollback to a known-good model version, or full retrain from scratch.
Data drift and concept drift require different responses
Data drift means the model is seeing inputs it wasn't prepared for — the fix is retraining on representative data. Concept drift means the definition of "correct" has changed — the fix is redefining labels and retraining. Confusing the two leads to the wrong response: retraining on new data won't fix concept drift if the labels are still wrong.
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers Setting Alerting Thresholds, Retraining Triggers, Reading a Monitoring Dashboard as a PM, Build It, What to Practice — plus a hands-on lab, quiz, and project artifact.
Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.
Browse all courses · View pricing · DeVenture Academy