Phase 7: Production Observability & Incident Response · 45 min · Langfuse · LangSmith · Arize Phoenix

AI-Native Observability

An LLM failure doesn't throw a 500 — it returns 200 OK with a fluent, wrong answer. Standard infrastructure monitoring will never see it.

Hiring signal: Being able to reconstruct a specific bad production response from trace data alone — without re-running the request — is the concrete skill that separates teams who can debug AI incidents from teams that can only speculate.

What you will learn

Name what a production trace must capture: input, resolved prompt, model/version, token counts, latency, tool calls, output
Explain the four production health signals: quality score, p95 latency, error/refusal rate, cost per request
Explain why LLM failures are invisible to standard infrastructure monitoring (uptime, HTTP status, latency alone)
Reconstruct why a specific production response went wrong using only its stored trace record

Introduction

AI-Native Observability

A support ticket comes in: "the assistant told me my refund would take 3-5 business days, but our policy is 7-10." The on-call engineer pulls up the standard monitoring dashboard — uptime is 100%, the request returned HTTP 200, p95 latency was normal. Every signal a traditional API's observability stack tracks says nothing went wrong. Nothing did, from infrastructure's point of view: the server responded successfully, quickly, with a complete, fluent, plausible-sounding answer. It was just wrong. Without a system specifically built to trace what an LLM actually saw and said, there's no way to know whether the refund-policy document was stale, whether it was retrieved at all, or whether the model simply stated something not in its context — the incident is invisible to every tool the team already had, because those tools were built to catch a different category of failure.

Why standard monitoring misses LLM failures

A traditional API failure announces itself: an exception, a stack trace, a non-200 status code, something a log aggregator flags automatically. An LLM "failure" usually looks identical, from the infrastructure's perspective, to a success — the request completed, the response was well-formed, the latency was normal. The content being wrong isn't a category of problem HTTP status codes or uptime monitors were ever designed to catch. This is the core reason AI-native observability has to trace something standard APM tools don't: not just whether a request succeeded mechanically, but what the model actually saw and said.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers What a trace has to capture, Four signals for system health, Tracing tools handle the plumbing, not the discipline, Build It — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.