Phase 9: LLMOps & Production Deployment · 60 min · Evidently · Prometheus/Grafana · MLflow

MLOps: Monitoring, Drift & Retraining

A model's accuracy is highest the day you deploy it. Everything after is decay you have to detect.

Hiring signal: Owns models in production: monitors, detects drift, and retrains on a trigger

What you will learn

Distinguish data drift, concept drift, and training-serving skew
Choose what to monitor: system, data, and model-quality signals
Design drift detection and alerting with sensible thresholds
Define retraining triggers (scheduled vs performance-based)
Write a rollback and incident runbook for a model in production

The Problem

Software, once correct, stays correct. Models don't. The world they predict keeps changing — user behavior shifts, a competitor launches, a holiday hits, an upstream data field changes format — and the model silently gets worse while every server health check stays green. The team finds out from angry users or a revenue dip, weeks late. Monitoring, drift detection, and a retraining/rollback plan are what turn "we trained a model" into "we operate a model." This is the heart of MLOps and a guaranteed senior interview theme: "how do you know your model is still working in production?"

The Concept

Why ML Monitoring Is Different from Software Monitoring

Traditional software monitoring is straightforward: if the server is up, responding within latency targets, and not throwing errors, everything is fine. The software's behavior is deterministic — the same input always produces the same output. If the code is correct and the server is healthy, the system is working.

ML systems break this assumption. A model can be perfectly healthy from an infrastructure perspective — 99.9% uptime, low latency, zero errors — while its predictions are silently degrading. The model's behavior depends on the relationship between its inputs and the patterns it learned during training. When that relationship changes (and it always does), the model gets worse without any error, crash, or alert.

This is why ML monitoring requires three layers, not one:

The thing that makes ML monitoring different from app monitoring: the system can be perfectly healthy while the predictions rot. So you monitor three layers:

1. System health  : latency, errors, throughput, saturation  (standard SRE)
2. Data health     : input distributions, nulls, ranges, schema (is the INPUT normal?)
3. Model quality   : accuracy/metric over time, prediction drift (are OUTPUTS still good?)

Most teams have layer 1 and miss 2 and 3 — which is exactly where models fail.

Your model serving API has 99.9% uptime, p99 latency < 200ms, and zero errors. But user complaints are rising. System health looks perfect. What layer are you missing?

System health (latency, errors, uptime) being green doesn't mean predictions are correct. Data drift (input distribution changed) or concept drift (the relationship between input and output changed) can silently degrade model quality while the infrastructure looks perfectly healthy. You need to monitor input distributions (layer 2) and prediction quality metrics over time (layer 3).

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Dashboards, Alerts and Responses, Rollback Procedure, Retraining Procedure, Use It, Ship It, Evaluation, Exercises, Key Terms, Common Pitfalls, Interview Framing — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.

Key terms in this lesson

Latency