Phase 7: AI Evaluation & Observability · 50 min · Python · LangSmith · Braintrust

Production Monitoring & Drift Detection

Detect the drift before the client detects the failure.

What you will learn

Detect drift: input distribution shift (new query types), output shift (response style changes), concept drift
Detect anomalies: unusual tool call patterns, unexpected agent trajectories, spike in errors
Monitor costs: per-request LLM cost, token usage trends, cost anomaly alerting, budget enforcement
Track latency: p50/p95/p99, SLO alerting, slow query identification
Implement online evaluation: sampling production traffic, running evaluators, alerting on regressions

Objective

What you'll learn

This lesson takes approximately 50 min. By the end, you will be able to:

Detect drift: input distribution shift (new query types), output shift (response style changes), concept drift
Detect anomalies: unusual tool call patterns, unexpected agent trajectories, spike in errors
Monitor costs: per-request LLM cost, token usage trends, cost anomaly alerting, budget enforcement
Track latency: p50/p95/p99, SLO alerting, slow query identification
Implement online evaluation: sampling production traffic, running evaluators, alerting on regressions

The Problem

In production, things drift. Data drift: the client's documents change over time. Model drift: the LLM provider updates the model. Concept drift: the business problem evolves. This lesson covers drift detection algorithms, alerting thresholds, online evaluation, and the metrics that predict production failures before the client notices.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Concept, Build, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.