Phase 4: Release, Reliability & Compliance · ~60 minutes · Python (stdlib · toy chaos experiment runner)

Chaos Engineering for LLM Production

Chaos engineering for LLMs is its own discipline in 2026.

Hiring signal: Can operate chaos engineering for llm production in production

What you will learn

Name the five chaos engineering prerequisites (SLI/SLO, observability, rollback, runbooks, on-call) and explain why skipping any breaks the practice.
Diagram the four planes (control, target, safety, observability) and the feedback loop into SLO.
Enumerate five LLM-specific experiments (memory overload, network fail, provider outage, malformed prompt, KV eviction storm).
Pick a tool — Harness, LitmusChaos, Chaos Mesh — given stack.

Introduction

Prerequisites before running experiments in production: defined SLI/SLO, trace+metric+log observability, automated rollback, runbooks, on-call. Architecture has four planes: control (experiment scheduler), target (services, infra, data stores), safety (guards + abort + traffic filters), observability (metrics + traces + logs), feedback (into SLO adjustments). Guardrails are mandatory: burn-rate alerts pause experiments if daily error-budget burn > 2x expected; suppression windows + trace-ID correlation dedupe alert noise. Cadence: weekly small canary + SLO review; monthly game day + postmortem; quarterly cross-team resilience audit + dependency mapping. LLM-specific experiments: memory overload, network failures, provider outages, malformed prompts, KV cache eviction storms. Tooling: Harness Chaos Engineering (LLM-derived recommendations, blast-radius downscaling, MCP tool integration); LitmusChaos (CNCF); Chaos Mesh (CNCF Kubernetes-native).

Type: Learn Languages: Python (stdlib, toy chaos experiment runner) Prerequisites: Phase 17 · 23 (SRE for AI), Phase 17 · 13 (Observability) Time: ~60 minutes

Objective

Learning objectives

Name the five chaos engineering prerequisites (SLI/SLO, observability, rollback, runbooks, on-call) and explain why skipping any breaks the practice.
Diagram the four planes (control, target, safety, observability) and the feedback loop into SLO.
Enumerate five LLM-specific experiments (memory overload, network fail, provider outage, malformed prompt, KV eviction storm).
Pick a tool — Harness, LitmusChaos, Chaos Mesh — given stack.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.

Key terms in this lesson

Inference