Phase 9: LLMOps & Production Deployment · 60 min · Evidently · Prometheus/Grafana · MLflow
The Problem
Software, once correct, stays correct. Models don't. The world they predict keeps changing — user behavior shifts, a competitor launches, a holiday hits, an upstream data field changes format — and the model silently gets worse while every server health check stays green. The team finds out from angry users or a revenue dip, weeks late. Monitoring, drift detection, and a retraining/rollback plan are what turn "we trained a model" into "we operate a model." This is the heart of MLOps and a guaranteed senior interview theme: "how do you know your model is still working in production?"
The Concept
Why ML Monitoring Is Different from Software Monitoring
Traditional software monitoring is straightforward: if the server is up, responding within latency targets, and not throwing errors, everything is fine. The software's behavior is deterministic — the same input always produces the same output. If the code is correct and the server is healthy, the system is working.
ML systems break this assumption. A model can be perfectly healthy from an infrastructure perspective — 99.9% uptime, low latency, zero errors — while its predictions are silently degrading. The model's behavior depends on the relationship between its inputs and the patterns it learned during training. When that relationship changes (and it always does), the model gets worse without any error, crash, or alert.
This is why ML monitoring requires three layers, not one:
The thing that makes ML monitoring different from app monitoring: the system can be perfectly healthy while the predictions rot. So you monitor three layers:
1. System health : latency, errors, throughput, saturation (standard SRE)
2. Data health : input distributions, nulls, ranges, schema (is the INPUT normal?)
3. Model quality : accuracy/metric over time, prediction drift (are OUTPUTS still good?)
Most teams have layer 1 and miss 2 and 3 — which is exactly where models fail.
Your model serving API has 99.9% uptime, p99 latency < 200ms, and zero errors. But user complaints are rising. System health looks perfect. What layer are you missing?
System health (latency, errors, uptime) being green doesn't mean predictions are correct. Data drift (input distribution changed) or concept drift (the relationship between input and output changed) can silently degrade model quality while the infrastructure looks perfectly healthy. You need to monitor input distributions (layer 2) and prediction quality metrics over time (layer 3).