Phase 7: AI Evaluation & Observability · 50 min · Python · LangSmith · Braintrust
Production Monitoring & Drift Detection
Detect the drift before the client detects the failure.
Hiring signal: FDE interviews test production monitoring knowledge: candidates who can describe input distribution shift, output distribution shift, and concept drift — and set up alerting for each — pass. Candidates who monitor only uptime and latency fail. Online evaluation (sampling production traffic, running evaluators, alerting on quality regressions) is the 2026 production pattern that every AI company expects.
What you will learn
- Detect drift: input distribution shift (new query types), output shift (response style changes), concept drift
- Detect anomalies: unusual tool call patterns, unexpected agent trajectories, spike in errors
- Monitor costs: per-request LLM cost, token usage trends, cost anomaly alerting, budget enforcement
- Track latency: p50/p95/p99, SLO alerting, slow query identification
- Implement online evaluation: sampling production traffic, running evaluators, alerting on regressions
What You'll Learn
This lesson takes approximately 50 min. By the end, you will be able to:
- Detect drift: input distribution shift (new query types), output shift (response style changes), concept drift
- Detect anomalies: unusual tool call patterns, unexpected agent trajectories, spike in errors
- Monitor costs: per-request LLM cost, token usage trends, cost anomaly alerting, budget enforcement
- Track latency: p50/p95/p99, SLO alerting, slow query identification
- Implement online evaluation: sampling production traffic, running evaluators, alerting on regressions
The Problem
In production, things drift. Data drift: the client's documents change over time. Model drift: the LLM provider updates the model. Concept drift: the business problem evolves. This lesson covers drift detection algorithms, alerting thresholds, online evaluation, and the metrics that predict production failures before the client notices.
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers Drift Detection: Input, Output, and Concept Drift, Anomaly Detection: Agent Trajectories and Tool Calls, Cost Monitoring and Budget Enforcement, Latency SLO Monitoring, Practical Application, What Hiring Managers Look For, Resources, Key Takeaways, Next Steps — plus a hands-on lab, quiz, and project artifact.
Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.
Browse all courses · View pricing · DeVenture Academy