Phase 9: Agent Evaluation & Observability · 60 min · Python · Langfuse · Grafana

Production Monitoring & Alerting

Find out from your dashboard — not from user complaints.

Hiring signal: Production monitoring is the operational maturity signal that distinguishes engineers who have shipped and maintained systems from those who only build prototypes. Any company with paying users needs this — it is a core expectation for senior AI engineers and often the first system a platform team builds.

What you will learn

Build a production metrics system capturing p50/p95/p99 latency, cost, error rate, and quality scores per run
Implement drift detection using z-score alerting with a rolling baseline for quality and operational metrics
Design and execute an incident response playbook for agent quality degradation events

The Problem

Your agent launches on a Tuesday. For three days, everything is fine. Then a model provider pushes a silent update on Friday evening. Your agent's quality score drops from 4.1 to 3.4 (on a 5-point scale). Token costs spike 3× because the updated model is using longer context windows. Error rate climbs from 2% to 12%.

You find out on Monday morning — when the support inbox fills up. Your users have had three days of degraded service. The team spends a day in incident response, tracing the root cause to the model update, rolling back to a pinned version, and re-running evaluations to confirm the fix. Total damage: 72 hours of user-visible quality regression.

With production monitoring, you find out Friday night. An automated alert fires at 11pm: "Quality score dropped 0.7 points, z-score = 3.1, severity: critical." You pin the model version and notify users of a brief maintenance window before market open Monday.

This is the difference between operational maturity and operating blind. Senior AI engineers are expected to instrument their systems for production visibility before anything ships. The tools and patterns are borrowed directly from SRE practice — the same metrics (p50/p95/p99, error rate, cost) and alerting patterns (baseline + z-score) that backend engineers use for API services.

Four Metric Categories
Operational: latency (p50, p95, p99), error rate, queue depth, worker utilization. Financial: cost per run, cost per user, daily spend, cost vs. baseline. Quality: LLM-as-judge score (online evaluation), task success rate, user feedback signals. Usage: requests per minute, task type distribution. All four are required — operational metrics alone do not tell you whether the agent's outputs are any good.

The Four Metric Categories

Each category answers a different question about your agent's health.

Operational metrics tell you whether the system is working. Latency percentiles are more useful than averages. A p50 of 4 seconds and p99 of 45 seconds means most users are satisfied but 1% have terrible experiences. p99 often reveals specific failure modes — retry storms, large context requests, slow tools — that p50 completely masks. Alert thresholds: p95 latency > 30 seconds, error rate > 5%.

Financial metrics tell you whether the system is viable. Daily spend trending up without a corresponding increase in usage signals a bug: runaway loops, unexpected token growth, or a model update that uses more tokens for the same tasks. Alert threshold: daily cost > 120% of 30-day baseline.

Quality metrics tell you whether the agent's outputs are good. This requires online evaluation — an LLM judge running automatically on every production trace. Without it, you have no automatic way to detect quality changes from model updates or prompt drift. Alert threshold: rolling mean quality score more than 1 standard deviation below 30-day baseline.

Usage metrics tell you how the system is being used. Task type distribution shifts (users suddenly doing more "compare" tasks and fewer "summarize" tasks) can explain quality or cost changes. Requests per minute spikes can precede error rate spikes.

Metric	Alert threshold	Alert channel
Error rate	> 5%	Slack (warning), PagerDuty (critical)
p95 latency	> 30,000ms	Slack
Daily cost	> 120% of baseline	Slack + email
Quality score	> 1 std dev below baseline	Slack
Quality score	> 2 std dev below baseline	PagerDuty

Your LLM-as-judge quality score drops from 4.1 to 3.4 two days after a model provider pushes a silent model update. Without monitoring, you find out when?

This is a real scenario that has happened to production AI teams. Model providers sometimes push silent updates that change model behavior. Without online evaluation running on production traces, there is no automatic quality signal. By the time user complaints accumulate, you have had days of degraded service. Online evaluation with alert thresholds catches this within hours of the first degraded runs arriving.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Drift Detection, Incident Response Playbook, Build It, What to Practice — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.