Phase 9: LLMOps & Production Deployment · 60 min · FastAPI · MLflow · Prometheus

Shadow Deployments: Safe Model Rollout in Production

Test in production without affecting production.

Hiring signal: Shadow deployments are standard practice at any company running ML in production. Engineers who understand safe rollout patterns — shadow, canary, blue-green — are qualified for ML Engineer and MLOps roles.

What you will learn

Understand shadow deployments and how they differ from canary and blue-green
Implement a shadow deployment that mirrors traffic to a new model without returning its responses
Build a comparison pipeline that evaluates shadow model output against production
Define promotion criteria: when to promote from shadow to canary to full
Handle shadow deployment failure modes (latency, cost, log volume)

The Problem

You trained a new recommendation model. Offline metrics look great — 12% lift in recall@10 on your test set. But offline metrics don't capture everything:

Latency: The new model is 3x slower. Will it cause timeouts under real load?
Edge cases: The test set doesn't include the 5% of queries that are malformed, empty, or adversarial
Distribution shift: The test set is from last month. Today's traffic has different patterns
Downstream effects: The new model's recommendations change user behavior in ways that affect other systems

You need to test the new model on real production traffic — but you can't risk degrading the user experience if it's broken.

The Concept

Three Safe Rollout Patterns

Pattern	Traffic to new model	Users affected	Risk	Best for
Shadow	100% (mirrored, not served)	0%	None to users	Validating quality and latency safely
Canary	10% → 25% → 50% → 100%	Gradual	Limited, controlled	Gradual confidence building
Blue-Green	0% → 100% (instant switch)	100% after switch	High at switch	Fast rollback when something breaks

Your team wants to test a new fraud detection model in production. The stakes are high: a wrong prediction could block legitimate transactions. Which rollout pattern should you use, and why?

How Shadow Deployment Works

Mirror: Every incoming request is duplicated. The original goes to the production model. The copy goes to the shadow model.
Serve: Only the production model's response is returned to the user. The shadow model's response is discarded.
Log: The shadow model's response is logged along with the production model's response and the input.
Compare: An offline or async process compares the two responses: agreement rate, latency, error rate, output distribution.
Decide: If the comparison metrics meet promotion criteria, promote the shadow model to canary. If not, debug and retrain.

The Critical Detail: Async Shadow Execution

The shadow model must NOT block the production response. If the shadow model takes 500ms and the production model takes 50ms, the user should still get their response in 50ms.

import asyncio
import time
from dataclasses import dataclass

@dataclass
class ModelResponse:
    prediction: any
    latency_ms: float
    model_version: str
    error: str = None

class ShadowDeployment:
    def __init__(self, prod_model, shadow_model, comparison_store):
        self.prod_model = prod_model
        self.shadow_model = shadow_model
        self.comparison_store = comparison_store  # DB or log for comparisons
    
    async def predict(self, request: dict) -> ModelResponse:
        # 1. Run production model — this is the user-facing path
        prod_start = time.monotonic()
        prod_response = await self.prod_model.predict(request)
        prod_latency = (time.monotonic() - prod_start) * 1000
        
        # 2. Fire shadow prediction asynchronously — DO NOT await
        # The user gets their response immediately
        asyncio.create_task(
            self._run_shadow(request, prod_response, prod_latency)
        )
        
        return prod_response
    
    async def _run_shadow(self, request, prod_response, prod_latency):
        """Run shadow model and log comparison. Errors here must not affect users."""
        try:
            shadow_start = time.monotonic()
            shadow_response = await self.shadow_model.predict(request)
            shadow_latency = (time.monotonic() - shadow_start) * 1000
            
            # Log the comparison for offline analysis
            self.comparison_store.log({
                "request_id": request.get("id"),
                "timestamp": time.time(),
                "prod_prediction": prod_response.prediction,
                "prod_latency_ms": prod_latency,
                "shadow_prediction": shadow_response.prediction,
                "shadow_latency_ms": shadow_latency,
                "agreement": prod_response.prediction == shadow_response.prediction,
                "prod_model_version": prod_response.model_version,
                "shadow_model_version": shadow_response.model_version,
            })
        except Exception as e:
            # Shadow failures must be silent — log but never propagate
            self.comparison_store.log_error({
                "request_id": request.get("id"),
                "error": str(e),
                "timestamp": time.time(),
            })

Your shadow model takes 800ms to respond, while your production model takes 100ms. A user sends a request. How long does the user wait for their response?

Promotion Criteria: When to Move from Shadow to Canary

@dataclass
class PromotionCriteria:
    # Quality metrics
    min_agreement_rate: float = 0.95      # Shadow must agree with prod 95%+ of the time
    max_disagreement_on_positive: float = 0.02  # < 2% disagreement on positive predictions
    
    # Latency metrics
    max_shadow_latency_p99: float = 500.0  # Shadow p99 latency must be < 500ms
    max_latency_increase: float = 2.0      # Shadow can't be more than 2x slower than prod
    
    # Reliability metrics
    max_shadow_error_rate: float = 0.01    # Shadow error rate must be < 1%
    min_sample_size: int = 10000           # Need at least 10K shadow predictions
    
    # Distribution metrics
    max_prediction_distribution_shift: float = 0.05  # KL divergence < 0.05

def evaluate_promotion(comparison_logs: list[dict], criteria: PromotionCriteria) -> dict:
    """Evaluate whether the shadow model is ready for canary promotion."""
    total = len(comparison_logs)
    if total < criteria.min_sample_size:
        return {"promote": False, "reason": f"Insufficient samples: {total}/{criteria.min_sample_size}"}
    
    agreements = sum(1 for log in comparison_logs if log["agreement"])
    agreement_rate = agreements / total
    
    shadow_errors = sum(1 for log in comparison_logs if log.get("shadow_error"))
    error_rate = shadow_errors / total
    
    shadow_latencies = [log["shadow_latency_ms"] for log in comparison_logs]
    prod_latencies = [log["prod_latency_ms"] for log in comparison_logs]
    shadow_p99 = sorted(shadow_latencies)[int(len(shadow_latencies) * 0.99)]
    prod_p50 = sorted(prod_latencies)[len(prod_latencies) // 2]
    
    results = {
        "samples": total,
        "agreement_rate": agreement_rate,
        "error_rate": error_rate,
        "shadow_p99_latency_ms": shadow_p99,
        "prod_p50_latency_ms": prod_p50,
        "latency_ratio": shadow_p99 / prod_p50 if prod_p50 > 0 else float('inf'),
    }
    
    checks = {
        "agreement_rate": agreement_rate >= criteria.min_agreement_rate,
        "error_rate": error_rate <= criteria.max_shadow_error_rate,
        "latency_p99": shadow_p99 <= criteria.max_shadow_latency_p99,
        "latency_ratio": results["latency_ratio"] <= criteria.max_latency_increase,
    }
    
    results["all_checks_passed"] = all(checks.values())
    results["promote"] = results["all_checks_passed"]
    results["failed_checks"] = [k for k, v in checks.items() if not v]
    
    return results

The Full Rollout Pipeline

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Ship It, Use It, Common Pitfalls, Sources, Evaluation, Exercises, Key Terms, Interview Framing — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.

Key terms in this lesson

Latency