Phase 8: Launch, Metrics & Scaling AI Products · 40 min · LaunchDarkly · Python

Rollout Strategies for AI Features

You wouldn't ship a payments migration to 100% of users on day one. Don't do it to a probabilistic system either.

What you will learn

Distinguish shadow mode, canary release, gradual rollout, and feature flags as rollout mechanisms with different risk/signal tradeoffs
Match a rollout strategy to an AI feature's blast radius and reversibility
Design a kill-switch plan specific to a generative feature, not just a generic on/off toggle
Produce a rollout plan for a real AI feature naming the specific mechanism, stages, and rollback trigger

The Problem

A team finishes an AI-powered contract-review feature: upload a contract, the system flags risky clauses for a legal team to review. It passed every eval in the golden dataset. The PM, eager to show launch momentum, ships it to 100% of the company's enterprise customers on a Tuesday. By Thursday, the feature has flagged an unusually high number of false positives on a contract type barely represented in the eval set — international distribution agreements, which happened to make up less than 2% of the golden dataset but a much larger share of one large customer's actual contract volume. That customer's legal team, now buried in noise, stops trusting the tool entirely and requests it be disabled account-wide. A slower rollout would have surfaced this exact gap against a small slice of real traffic, with a fast, contained way to turn it off, before it reached a customer's full contract volume and burned their trust in the feature for good.

This is the core argument of this lesson: a generative feature's eval suite, however thorough, is tested against a golden dataset — a necessarily finite, imperfect sample of the space of real-world inputs. The feature's very first exposure to full-scale, unfiltered real-world input is the actual, complete test of whether the eval sample was representative enough, and that test should never happen at 100% of traffic simultaneously. Rollout strategy is the mechanism that controls how much real-world surface area a still-unproven system is exposed to before you've confirmed the eval results hold up outside the eval set.

Four mechanisms, four different signals

Shadow mode — the model runs against real production traffic and produces real outputs, but those outputs are never shown to users; they're logged and compared against the current system's behavior (or against human judgment, sampled). This is the lowest-risk rollout stage because it produces zero user-facing blast radius while still surfacing exactly how the model performs against real, unfiltered input distribution — the gap the contract-review example's eval set missed would likely have shown up here, as an unusual disagreement rate on international distribution agreements specifically.
Canary release — a small, fixed slice of real traffic (often 1-5%) gets the real, live feature, with tight monitoring on that slice before expanding. Unlike shadow mode, users in the canary group actually experience the feature, so this stage tests real user reactions and behavior, not just model output quality in isolation.
Gradual rollout — expanding the live percentage over time (5% → 25% → 50% → 100%) with defined checkpoints and go/no-go criteria at each stage, rather than a single canary-to-100% jump. This is the right mechanism when a feature's risk profile is moderate and you want multiple checkpoints to catch a problem that only appears at larger scale or after longer exposure (a quality issue that surfaces after a week of usage, not in the first hour).
Feature flags — an independent, product-level on/off switch (often per-customer, per-segment, or global) that isn't itself a rollout stage but is the mechanism that makes every other stage actually reversible fast. A canary or gradual rollout without a working feature flag is not meaningfully safer than a full launch, because "safer" depends entirely on how fast and how completely you can turn the feature off when a problem surfaces — this is the tool LaunchDarkly and similar platforms are built around.

The rollout mechanism should be chosen by blast radius and reversibility, not by team habit
A feature that only affects what a user sees in their own session (a chat response, a generated summary shown just to them) has a small blast radius and a fast, cheap reversal — mistakes are contained and correctable per-interaction. A feature that writes to a shared record, auto-triggers a downstream action (an auto-approved refund, an auto-sent email, a routing decision that affects another team's queue), or shapes a decision with real consequences has a large blast radius and a slow, expensive, or sometimes impossible reversal. The first kind of feature can often go straight to a fast canary-to-gradual rollout. The second kind of feature earns a mandatory shadow-mode stage first, specifically because the cost of learning about a systematic problem after it's already affected real actions is categorically higher than a bit of launch delay.

A PM is choosing between shadow mode and a 5% canary release as the very first rollout stage for a new AI feature that auto-sends a follow-up email to customers based on generated content. Which is the more appropriate first stage, and why?

An auto-send action is exactly the DOWNSTREAM_ACTION blast-radius category this lesson flags as needing a mandatory shadow-mode stage first: once an email is sent, it can't be unsent, so even a small 5% canary risks real customers receiving a bad auto-generated email before the team has any real-traffic signal on the model's output quality. Shadow mode lets the team observe exactly how the model would have behaved against full, unfiltered real traffic with zero chance of an actual bad email reaching anyone, which is the right way to build confidence before accepting any live exposure at all, however small.

Designing the kill switch, not just the ramp-up

A rollout plan that only specifies the ramp-up percentages and skips the exact rollback mechanism is half a plan. A real kill-switch design answers: who has authority to pull the flag (and can they do it without needing a deploy — the whole point of a feature flag system is that disabling doesn't require a code push), what specific metric or threshold triggers an automatic or strongly-recommended rollback (not "if it seems bad," but a named number), and what happens to in-flight state when the feature is disabled mid-session (does a user mid-conversation with the disabled feature see an error, a graceful fallback to the old behavior, or something else). For the contract-review example, the actual fix wasn't just "roll back slower next time" — it was defining a canary stage against a stratified sample specifically including underrepresented contract types, with an explicit rollback trigger ("false-positive rate on any single contract type exceeds 2x the eval baseline") that would have caught the international-distribution-agreement gap during the canary stage instead of after full rollout.

A team ships an AI feature with a gradual rollout (10% -> 50% -> 100% over two weeks) and a feature flag, but the flag can only be toggled by an engineer with deploy access, and there's no specific metric threshold defined for when to roll back — the plan just says "monitor closely and use judgment." A quality regression appears at the 50% stage on a Friday afternoon. What's the most likely outcome, and what was missing from the rollout plan?

A gradual rollout stage percentage and a feature flag existing are necessary but not sufficient -- the plan also needs a rollback authority who can act without needing deploy access (especially at an inconvenient time, like Friday afternoon) and a specific, named metric threshold that removes the ambiguity of 'use judgment' under time pressure. Without both, exactly the failure mode described plays out: the regression is real, but nobody with fast rollback authority is confident enough (or available enough) to pull the trigger over a weekend, and 50% of traffic keeps experiencing the regression far longer than the technical rollout mechanism would have required.

Build

Build it

The code artifact lesson_08_01_rollout_planner.py implements a rollout strategy planner:

BlastRadius enum (SESSION_ONLY, SHARED_RECORD, DOWNSTREAM_ACTION) and Reversibility enum (INSTANT, DELAYED, IRREVERSIBLE) used to score a feature's risk profile
RolloutStage — one stage in a plan: mechanism (shadow/canary/gradual/full), traffic percentage, duration, and go/no-go criteria
KillSwitchPlan — rollback authority, the specific metric threshold that triggers rollback, and in-flight session behavior on disable
recommend_rollout_stages() — given a feature's blast radius and reversibility, recommends a stage sequence (a high-blast-radius, low-reversibility feature always gets a mandatory shadow-mode stage first)
render_rollout_plan() — produces a markdown rollout plan combining the staged sequence and kill-switch design
A worked example: the contract-review feature from this lesson, redesigned with a shadow-mode stage and a specific rollback trigger tied to the international-distribution-agreement gap

"""
lesson_08_01_rollout_planner.py

A rollout strategy planner for AI features.

Scores a feature's risk profile (blast radius, reversibility) and
recommends a rollout stage sequence -- shadow mode, canary, gradual
rollout -- along with a kill-switch design specifying rollback
authority, a named metric threshold, and in-flight session behavior.
This is the artifact that turns "we'll monitor closely and use
judgment" into an actual, fast, executable rollback plan.

Reference: LaunchDarkly — https://launchdarkly.com/

Usage:
    python lesson_08_01_rollout_planner.py

No external dependencies, no network calls, no API keys required.
"""

from __future__ import annotations

from dataclasses import dataclass, field
from enum import Enum

class BlastRadius(Enum):
    SESSION_ONLY = "Session-only (affects only what one user sees)"
    SHARED_RECORD = "Shared record (writes to data other users/teams see)"
    DOWNSTREAM_ACTION = "Downstream action (triggers an auto-action: refund, email, routing)"

class Reversibility(Enum):
    INSTANT = "Instant (disabling the flag fully undoes the effect)"
    DELAYED = "Delayed (effect persists until manually corrected)"
    IRREVERSIBLE = "Irreversible (effect cannot be undone once it happens)"

class Mechanism(Enum):
    SHADOW_MODE = "Shadow mode"
    CANARY = "Canary release"
    GRADUAL_ROLLOUT = "Gradual rollout"
    FULL_LAUNCH = "Full launch"

@dataclass
class RolloutStage:
    mechanism: Mechanism
    traffic_pct: float
    duration_days: int
    go_no_go_criteria: str

@dataclass
class KillSwitchPlan:
    rollback_authority: str  # who can pull the flag, and how fast
    rollback_metric_threshold: str  # a specific, named number/condition
    in_flight_behavior: str  # what happens to users mid-session on disable

def recommend_rollout_stages(blast_radius: BlastRadius, reversibility: Reversibility) -> list[RolloutStage]:
    high_risk = (
        blast_radius in (BlastRadius.SHARED_RECORD, BlastRadius.DOWNSTREAM_ACTION)
        or reversibility in (Reversibility.DELAYED, Reversibility.IRREVERSIBLE)
    )

    stages: list[RolloutStage] = []
    if high_risk:
        stages.append(RolloutStage(
            Mechanism.SHADOW_MODE, 100.0, 7,
            "Model output logged against 100% of real traffic with zero user-facing exposure; "
            "disagreement rate with current system/human baseline reviewed across all input "
            "subtypes, not just in aggregate.",
        ))
    stages.append(RolloutStage(
        Mechanism.CANARY, 5.0, 5,
        "No metric regresses beyond the defined threshold on the canary slice; canary traffic "
        "is stratified to include underrepresented input subtypes, not just random sampling.",
    ))
    stages.append(RolloutStage(
        Mechanism.GRADUAL_ROLLOUT, 25.0, 4,
        "Metrics hold at 25% for the full duration before advancing.",
    ))
    stages.append(RolloutStage(
        Mechanism.GRADUAL_ROLLOUT, 50.0, 4,
        "Metrics hold at 50% for the full duration before advancing.",
    ))
    stages.append(RolloutStage(Mechanism.FULL_LAUNCH, 100.0, 0, "N/A — full launch."))
    return stages

def render_rollout_plan(
    feature_name: str,
    blast_radius: BlastRadius,
    reversibility: Reversibility,
    kill_switch: KillSwitchPlan,
) -> str:
    stages = recommend_rollout_stages(blast_radius, reversibility)
    lines = [
        f"# Rollout Plan — {feature_name}",
        "",
        f"**Blast radius:** {blast_radius.value}",
        f"**Reversibility:** {reversibility.value}",
        "",
        "## Rollout stages",
        "",
        "| Stage | Mechanism | Traffic % | Duration | Go/No-Go Criteria |",
        "|---|---|---|---|---|",
    ]
    for i, s in enumerate(stages, 1):
        lines.append(
            f"| {i} | {s.mechanism.value} | {s.traffic_pct:.0f}% | {s.duration_days}d | "
            f"{s.go_no_go_criteria} |"
        )

    lines += [
        "",
        "## Kill-switch plan",
        "",
        f"- **Rollback authority:** {kill_switch.rollback_authority}",
        f"- **Rollback metric threshold:** {kill_switch.rollback_metric_threshold}",
        f"- **In-flight session behavior on disable:** {kill_switch.in_flight_behavior}",
    ]
    return "\n".join(lines)

# ── Worked example ───────────────────────────────────────────────────────────
# The contract-review feature from the lesson's opening, redesigned with a
# mandatory shadow-mode stage and a specific rollback trigger tied to the
# international-distribution-agreement gap the original eval set missed.

def build_example_plan() -> tuple[str, BlastRadius, Reversibility, KillSwitchPlan]:
    feature_name = "Contract Risk-Clause Flagging Assistant"
    blast_radius = BlastRadius.SHARED_RECORD  # flags shown to legal team, shapes their review
    reversibility = Reversibility.DELAYED  # a bad flag session can already have consumed legal team trust/time
    kill_switch = KillSwitchPlan(
        rollback_authority="Any on-call PM or eng lead can disable via the feature-flag "
                            "dashboard without a deploy, 24/7.",
        rollback_metric_threshold="False-positive rate on any single contract type (tracked "
                                   "per-type, not blended) exceeds 2x the eval baseline for "
                                   "that type, OR overall false-positive rate exceeds 15%.",
        in_flight_behavior="Users mid-review see a banner: 'Automated flagging is temporarily "
                            "unavailable — your uploaded contract remains available for manual "
                            "review.' No partial or stale flags are shown.",
    )
    return feature_name, blast_radius, reversibility, kill_switch

if __name__ == "__main__":
    feature_name, blast_radius, reversibility, kill_switch = build_example_plan()
    print(render_rollout_plan(feature_name, blast_radius, reversibility, kill_switch))
    print()
    print(
        "Takeaway: because this feature writes to a shared record the legal team relies on "
        "(SHARED_RECORD blast radius) and a bad session isn't cleanly undoable (DELAYED "
        "reversibility), the recommended plan opens with a mandatory shadow-mode stage -- "
        "exactly the stage that would have surfaced the international-distribution-agreement "
        "false-positive gap against real traffic before any legal team ever saw a live flag."
    )

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Check Yourself — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.