Phase 7: Production Observability & Incident Response · 45 min · Langfuse · Arize Phoenix · Python

Incident Response Playbook

There's rarely one broken line of code to fix. There's a distribution that shifted, and the postmortem's job is turning that into a permanent test, not just a closed ticket.

Hiring signal: Explaining why an LLM incident's postmortem produces a permanent golden-set entry, not just a closed ticket, is the specific practice that shows you understand LLM failures as distributional, not single-bug, problems.

What you will learn

Explain why LLM incidents are often distributional failures with multiple interacting causes, unlike a typical API bug
Walk an incident through the six-stage playbook: detect, triage, contain, eval, fix, review
Explain why the eval stage needs to target the specific failure mode discovered, not a generic test
Turn a postmortem's failure case into a permanent golden-set regression entry

Introduction

Incident Response Playbook

A traditional API incident usually has a shape everyone already knows how to handle: something broke, there's a line of code (or a config, or a bad deploy) responsible, you find it, you fix it, you ship the fix, you close the ticket. An LLM incident frequently doesn't have that shape at all. A hallucination showing up in production might trace back to a retrieved document that's slightly outdated, combined with a prompt template that's slightly more ambiguous than intended, combined with a model provider's silent update shifting how it weighs conflicting information — three partial contributors, each individually reasonable, together producing a wrong answer with no single line anyone could point to and say "that's the bug." This is a distributional failure: the system's statistical behavior shifted, not its code. Running the traditional "find the bug, fix the bug" playbook against this kind of incident wastes time looking for something that was never going to be there in the singular form the playbook expects.

Six stages, built for distributional failures

Detect — the drift/quality signals from earlier in this phase surface that something's wrong. Triage — classify the incident (c12-07-3's four classes) and assign severity. Contain — take the severity-appropriate first action (immediate for S1, or move straight to investigation for S2). Eval — build or run an evaluation specifically targeting the failure mode just discovered; because the problem is distributional, a single manual test of "does it work now" doesn't tell you anything — you need a check that exercises the actual pattern that failed, across enough cases to see whether it's actually fixed. Fix — address the root cause, which for a distributional failure is often addressing several partial contributors together (updating the stale document, tightening the ambiguous prompt), not changing one line. Review — the postmortem, which for an LLM incident has one more job than a traditional one does.

Why "eval" is its own stage, not folded into "fix"
For a traditional bug, verifying the fix often just means re-running the one test that used to fail. For a distributional failure, the "test" that would have caught it originally may not have existed — the eval stage is where you build the check that exercises the actual failure pattern (not just the one specific input that happened to surface it), so you can tell whether the fix addressed the pattern or just that one instance of it.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The postmortem becomes a golden-set entry, Why the stages must run in order, This connects back to the harness itself, Build It — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.

Key terms in this lesson

Golden Set