The Problem
Your agent launches on a Tuesday. For three days, everything is fine. Then a model provider pushes a silent update on Friday evening. Your agent's quality score drops from 4.1 to 3.4 (on a 5-point scale). Token costs spike 3× because the updated model is using longer context windows. Error rate climbs from 2% to 12%.
You find out on Monday morning — when the support inbox fills up. Your users have had three days of degraded service. The team spends a day in incident response, tracing the root cause to the model update, rolling back to a pinned version, and re-running evaluations to confirm the fix. Total damage: 72 hours of user-visible quality regression.
With production monitoring, you find out Friday night. An automated alert fires at 11pm: "Quality score dropped 0.7 points, z-score = 3.1, severity: critical." You pin the model version and notify users of a brief maintenance window before market open Monday.
This is the difference between operational maturity and operating blind. Senior AI engineers are expected to instrument their systems for production visibility before anything ships. The tools and patterns are borrowed directly from SRE practice — the same metrics (p50/p95/p99, error rate, cost) and alerting patterns (baseline + z-score) that backend engineers use for API services.
Four Metric Categories
Operational: latency (p50, p95, p99), error rate, queue depth, worker utilization. Financial: cost per run, cost per user, daily spend, cost vs. baseline. Quality: LLM-as-judge score (online evaluation), task success rate, user feedback signals. Usage: requests per minute, task type distribution. All four are required — operational metrics alone do not tell you whether the agent's outputs are any good.
The four metric categories
Each category answers a different question about your agent's health.
Operational metrics tell you whether the system is working. Latency percentiles are more useful than averages. A p50 of 4 seconds and p99 of 45 seconds means most users are satisfied but 1% have terrible experiences. p99 often reveals specific failure modes — retry storms, large context requests, slow tools — that p50 completely masks. Alert thresholds: p95 latency > 30 seconds, error rate > 5%.
Financial metrics tell you whether the system is viable. Daily spend trending up without a corresponding increase in usage signals a bug: runaway loops, unexpected token growth, or a model update that uses more tokens for the same tasks. Alert threshold: daily cost > 120% of 30-day baseline.
Quality metrics tell you whether the agent's outputs are good. This requires online evaluation — an LLM judge running automatically on every production trace. Without it, you have no automatic way to detect quality changes from model updates or prompt drift. Alert threshold: rolling mean quality score more than 1 standard deviation below 30-day baseline.
Usage metrics tell you how the system is being used. Task type distribution shifts (users suddenly doing more "compare" tasks and fewer "summarize" tasks) can explain quality or cost changes. Requests per minute spikes can precede error rate spikes.
| Metric | Alert threshold | Alert channel |
|---|
| Error rate | > 5% | Slack (warning), PagerDuty (critical) |
| p95 latency | > 30,000ms | Slack |
| Daily cost | > 120% of baseline | Slack + email |
| Quality score | > 1 std dev below baseline | Slack |
| Quality score | > 2 std dev below baseline | PagerDuty |
Your LLM-as-judge quality score drops from 4.1 to 3.4 two days after a model provider pushes a silent model update. Without monitoring, you find out when?
This is a real scenario that has happened to production AI teams. Model providers sometimes push silent updates that change model behavior. Without online evaluation running on production traces, there is no automatic quality signal. By the time user complaints accumulate, you have had days of degraded service. Online evaluation with alert thresholds catches this within hours of the first degraded runs arriving.
Drift detection
Prompt drift is a production reality: your agent's behavior changes over time without any code changes. Sources: model provider updates, changes in retrieved content (RAG data), distribution shifts in user queries, or cumulative effects of prompt edits.
Detect drift using z-score alerting against a rolling baseline:
z = (current_value - baseline_mean) / baseline_stdev
For metrics where lower is worse (quality score, success rate): alert when z is significantly negative. For metrics where higher is worse (error rate, latency): alert when z is significantly positive. Thresholds: z > 2.0 → warning, z > 3.0 → critical.
detector = DriftDetector(collector, baseline_n=25)
# After establishing baseline on first 25 runs:
current_metrics = {
"quality_score": 3.4, # was 4.1
"error_rate": 0.12, # was 0.02
"p99_latency_ms": 58000 # was 42000
}
alerts = detector.run_checks(current_metrics, baselines)
# → fires: quality_score CRITICAL (z=-3.1), error_rate WARNING (z=2.4)
Tool-level drift detection is also valuable. Track per-tool error rates. If web_search error rate doubles in 24 hours, an upstream API changed. If read_page latency triples, a CDN or proxy changed behavior. Tool-level granularity tells you exactly which component to investigate.
p99 latency of 45 seconds (meaning 1% of requests take more than 45 seconds) is important to monitor because:
Averages hide the worst cases. p99 latency captures the tail experience — the users with the worst outcomes. A p50 of 5 seconds and p99 of 45 seconds means most users are satisfied but 1% have unacceptable wait times. More importantly, p99 often reveals underlying failure modes (retry storms, cache misses, unusually large context requests) that p50 and even p95 completely miss. Alert on p99, not just mean. Fix the causes revealed by the tail, not the average.
Incident response playbook
When an alert fires, the on-call engineer needs a structured procedure. Write the playbook into the monitoring system itself — don't rely on engineers to remember it at 2am.
The automated first-response diagnostic (implemented in run_incident_playbook()):
- Check error distribution — are failures concentrated in specific task types or time windows?
- Check latency percentiles in the last hour — is there a tail latency spike?
- Check quality scores in the last hour — has online evaluation captured the degradation?
- Generate recommended actions based on which metric triggered the alert
- Save the diagnostic report as an incident record
The recommended actions for a quality alert:
- "Compare current prompt version to 30-day baseline. Run offline eval on 20 examples."
- "Check for recent model version changes in your API logs."
- "Roll back to the last known-good prompt version and re-run evaluation to confirm."
For a cost alert:
- "Check for runaway loops — look at steps_taken distribution, flag any runs > 3× median."
- "Audit token usage per task type — is one task type suddenly using far more tokens?"
Document every incident. A post-mortem captures: what changed, how it was detected, how it was resolved, and what monitoring or testing would have caught it earlier. This drives your observability improvements.