Phase 4: Release, Reliability & Compliance · ~60 minutes · Python (stdlib · toy canary-progression simulator)

Shadow Traffic, Canary Rollout, and Progressive Deployment for LLMs

LLM rollouts combine the hardest parts of software deployment: no unit tests, diffuse failure modes, delayed signals.

Hiring signal: Can operate shadow traffic, canary rollout, and progressive deployment for llms in production

What you will learn

Distinguish shadow mode (zero-impact compare), canary (live traffic progressive), and A/B (stability-confirmed comparison).
Enumerate five LLM-specific canary metrics (latency, cost/request, error/refusal, output-length distribution, user feedback).
Explain why LLM non-determinism (up to 15%) changes what "stable" means in a rollout.
Design a rollback path that takes seconds (policy flip) not hours (redeploy).

Introduction

The sequence is (1) shadow mode — duplicate prod requests to candidate model, log, compare with zero user impact; catches obvious distribution issues but is not a quality guarantee; (2) canary rollout — progressive traffic shift 10% → 25% → 50% → 75% → 100% with gates at each step; track latency percentiles, cost/request, error/refusal rate, output length distribution, user-feedback rate; (3) A/B testing for distinct alternatives after stability confirmed. Non-determinism is irreducible — up to 15% accuracy variation across runs with identical inputs due to GPU FP non-associativity plus batch-size variance. Cost is a variable, not constant — a 20% better model can be 3x more expensive per call. Rollback speed is decisive: if rollback requires redeploy, you are too slow. Policy lives in config/flags; model lives in registry with pinned digests; rollback = flip policy + revert threshold + pin old model in seconds.

Type: Learn Languages: Python (stdlib, toy canary-progression simulator) Prerequisites: Phase 17 · 13 (Observability), Phase 17 · 21 (A/B Testing) Time: ~60 minutes

Objective

Learning objectives

Distinguish shadow mode (zero-impact compare), canary (live traffic progressive), and A/B (stability-confirmed comparison).
Enumerate five LLM-specific canary metrics (latency, cost/request, error/refusal, output-length distribution, user feedback).
Explain why LLM non-determinism (up to 15%) changes what "stable" means in a rollout.
Design a rollback path that takes seconds (policy flip) not hours (redeploy).

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.