The Problem
A team ships an AI-powered customer support triage feature that classifies incoming tickets by urgency. At launch, it's 94% accurate on the eval set. The team moves on to the next feature. Three months later, a support operations manager notices that critical tickets are sitting in the queue longer than they used to. Investigation reveals the model's accuracy on critical-ticket classification has dropped from 94% to 79% — but the overall accuracy dashboard still reads 91%, because the model is still fine on the easy categories (general inquiries, password resets) that make up 80% of tickets. The model didn't break. It drifted: the types of tickets customers submit shifted (a new product launch generated unfamiliar issue types), and the model's training data didn't include them. Nobody was watching per-subtype performance, nobody set a drift alert, and the regression went unnoticed for three months because the headline metric looked fine.
This is the post-launch reality of AI features: models degrade. Not because they're buggy, but because the world changes around them. Input distributions shift, user behavior evolves, and the relationship between inputs and correct outputs can change. A traditional software feature that works at launch keeps working — the code doesn't change. An AI feature that works at launch starts degrading the moment it encounters data it wasn't trained on. The PM's job is to know this, watch for it, and have a plan before it happens.
Three types of drift
Understanding drift starts with distinguishing three separate failure modes that require different responses:
- Data drift (input drift): The distribution of inputs the model sees in production shifts away from what it was trained on. Example: a document summarization model trained on 2-page memos starts receiving 20-page legal contracts. The model's mapping (summarize this) may still be correct, but it's seeing inputs outside its training distribution, so quality degrades. Response: retrain with representative production data, or add input validation that flags out-of-distribution inputs.
- Concept drift: The relationship between input and correct output changes. Example: a spam classifier trained on 2023 email patterns faces new spam tactics in 2025. The inputs (emails) may look similar, but what counts as "spam" has evolved. The model's mapping is now wrong even for inputs it was trained on. Response: update labels (redefine what "correct" means), retrain with updated labels, and increase retraining cadence.
- Model decay: The model itself degrades — not because of external changes but because of internal issues. This is rare for LLM-based features (the model weights don't change) but common for custom-trained models that are updated incrementally. Response: rollback to a known-good model version, or full retrain from scratch.
Data drift and concept drift require different responses
Data drift means the model is seeing inputs it wasn't prepared for — the fix is retraining on representative data. Concept drift means the definition of "correct" has changed — the fix is redefining labels and retraining. Confusing the two leads to the wrong response: retraining on new data won't fix concept drift if the labels are still wrong.
Setting alerting thresholds
The goal of alerting is to catch meaningful regressions without crying wolf. If every minor fluctuation triggers an alert, the team develops alert fatigue and stops responding. If thresholds are too loose, real regressions go unnoticed for months — exactly the opening scenario.
Calibrate to the eval baseline:
- Establish the baseline: what's the eval accuracy, and what's the natural variance? If the eval suite runs 10 times and accuracy varies by ±1%, then a drop from 94% to 93% is within natural variance — don't alert on it.
- Set the threshold beyond natural variance: a 3-point drop (94% → 91%) is likely real. A 5-point drop is definitely real.
- Use relative thresholds for subtypes: per-subtype metrics have smaller sample sizes and higher variance, so absolute thresholds (e.g., "alert at <90%") are too noisy. Use relative thresholds: "alert when false-negative rate on any subtype exceeds 2x the eval baseline for that subtype."
- Separate quality from noise: a one-day dip may be a data pipeline issue, not model degradation. Alert on sustained regressions (3+ days below threshold) rather than single-day spikes.
A team sets an alerting threshold at "overall accuracy drops below 90%." The model's baseline is 94% with ±1% variance. Three months after launch, a new product launch causes an unfamiliar ticket type to surge. The model performs poorly on this new type (60% accuracy) but the overall accuracy only drops to 92% because the new type is 15% of traffic. What's wrong with this threshold?
The lesson's core point about per-subtype monitoring applies directly: a blended overall metric can mask severe per-subtype regressions, especially when the affected subtype is a minority of traffic. The fix is per-subtype thresholds with relative baselines (e.g., "alert when any subtype drops below 80% of its eval baseline"), not a single absolute threshold on the blended number. This is the same principle from the metrics lesson: separate the tracks, don't blend them.
Retraining triggers
When a regression is detected, retraining is the most common response — but when to retrain, and how, depends on the feature's drift risk profile:
Scheduled retraining: Retrain every N weeks (e.g., monthly) regardless of metrics. Good for features with predictable, gradual drift — the model degrades slowly and steadily, so regular refreshes keep it current. Risk: you may retrain when nothing has changed (wasted effort) or wait too long between scheduled retrains if drift is faster than expected.
Threshold-triggered retraining: Retrain only when a monitoring metric crosses a defined threshold. Good for features with unpredictable drift — the model may be stable for months, then degrade suddenly when input distribution shifts. Risk: if thresholds are too loose, you retrain too late; if too tight, you retrain too often.
Both (scheduled + threshold-triggered): Retrain on a scheduled cadence (e.g., quarterly) AND allow threshold-triggered retraining between scheduled cycles if a metric crosses a critical threshold. Best for high-stakes features where you want regular refreshes but can't afford to wait if drift accelerates. This is the recommended approach for most production AI features.
A team runs a content-moderation feature on a fast-changing social platform (new slang, memes, and evasion tactics emerge weekly) and a separate document-summarization feature for internal corporate memos (format and vocabulary barely change year to year). Both currently use the same policy: scheduled retraining every quarter, no threshold-triggered retraining. Which feature is most at risk from this policy, and why?
The retraining strategy should match the drift risk profile, not be applied uniformly. Content moderation on a fast-changing platform faces drift that can appear within days (new evasion phrasing, new meme formats) and carries high stakes (real harm from missed violations) — a quarterly-only schedule leaves it exposed for up to three months after a new tactic emerges. The document-summarization feature, with slow, low-stakes drift, is a reasonable fit for scheduled-only retraining. Applying one policy to both ignores exactly the risk-profile-based reasoning this lesson teaches.
Reading a monitoring dashboard as a PM
An ML engineer reads a monitoring dashboard for technical signals: loss curves, gradient norms, prediction confidence distributions. A PM reads the same dashboard for product signals:
- Per-subtype performance: Is one user segment or input type degrading while others are stable? This is the most common drift pattern — and the most invisible in blended metrics.
- Guardrail metrics: Are we violating latency, cost, or safety constraints? A model that's accurate but takes 8 seconds to respond may be worse than a slightly less accurate model that responds in 2 seconds.
- Trend over time: Is this a sudden drop (likely an incident — investigate immediately) or a gradual decline (likely drift — schedule retraining)? The shape of the trend determines the response.
- User-facing signals: Are support tickets increasing? Is NPS dropping? Is usage declining? These lag model metrics but confirm that a technical regression is actually affecting users.
The PM's unique contribution is connecting model metrics to user impact. An ML engineer sees "false-negative rate on critical tickets increased from 6% to 21%." A PM translates: "critical tickets are being misrouted to low-priority queues, which means urgent customer issues are sitting unresolved for hours longer than they should — this is a customer satisfaction and potential churn risk." Both perspectives are needed; neither is sufficient alone.
Build
Build it
The code artifact lesson_08_03_drift_monitor.py implements a drift monitoring plan builder:
DriftType enum (DATA_DRIFT, CONCEPT_DRIFT, MODEL_DECAY) and DriftSignal — a dataclass for one monitoring signal: name, drift type, metric being tracked, baseline value, and alerting thresholdRetrainingTrigger — a dataclass for a retraining decision: trigger type (scheduled, threshold, both), cadence, threshold condition, and rationaleMonitoringPlan — a collection of drift signals, alerting thresholds, retraining triggers, and a PM-facing dashboard specificationcheck_threshold_quality() — flags thresholds that are too loose (would miss real regressions), too tight (would false-alarm), or set on blended metrics only (would miss per-subtype regressions)render_monitoring_plan() — produces a markdown report with the full plan, threshold quality assessment, and a PM dashboard specification- A worked example: the customer support triage feature from this lesson's opening, redesigned with per-subtype monitoring, calibrated thresholds, and a combined retraining strategy
"""
lesson_08_03_drift_monitor.py
A drift monitoring plan builder for AI features.
Distinguishes data drift, concept drift, and model decay; sets
calibrated alerting thresholds; defines retraining triggers; and
produces a PM-facing dashboard specification that connects model
metrics to user impact.
Usage:
python lesson_08_03_drift_monitor.py
No external dependencies, no network calls, no API keys required.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from enum import Enum
from typing import Optional
class DriftType(Enum):
DATA_DRIFT = "data_drift"
CONCEPT_DRIFT = "concept_drift"
MODEL_DECAY = "model_decay"
class TriggerType(Enum):
SCHEDULED = "scheduled"
THRESHOLD = "threshold"
BOTH = "both"
@dataclass
class DriftSignal:
name: str
drift_type: DriftType
metric: str
baseline: float
threshold: str
subtype: Optional[str] = None
rationale: str = ""
@dataclass
class RetrainingTrigger:
trigger_type: TriggerType
cadence: str = ""
threshold_condition: str = ""
rationale: str = ""
@dataclass
class DashboardMetric:
name: str
metric: str
frequency: str
user_impact_translation: str
@dataclass
class MonitoringPlan:
feature_name: str
drift_signals: list[DriftSignal] = field(default_factory=list)
retraining_trigger: Optional[RetrainingTrigger] = None
dashboard_metrics: list[DashboardMetric] = field(default_factory=list)
def check_threshold_quality(self) -> list[str]:
warnings = []
has_subtype = any(s.subtype for s in self.drift_signals)
if not has_subtype:
warnings.append(
"No per-subtype signals defined — blended metrics can mask "
"severe per-subtype regressions."
)
for s in self.drift_signals:
if "overall" in s.metric.lower() and not s.subtype:
warnings.append(
f"Signal '{s.name}' tracks an overall metric without "
f"subtype breakdown — consider adding per-subtype signals."
)
if not self.retraining_trigger:
warnings.append("No retraining trigger defined.")
return warnings
def add_signal(self, signal: DriftSignal) -> "MonitoringPlan":
self.drift_signals.append(signal)
return self
def add_dashboard_metric(self, metric: DashboardMetric) -> "MonitoringPlan":
self.dashboard_metrics.append(metric)
return self
def render_monitoring_plan(plan: MonitoringPlan) -> str:
lines = [f"# Drift Monitoring Plan: {plan.feature_name}\n"]
lines.append("## Drift Signals\n")
lines.append("| Signal | Drift Type | Metric | Subtype | Baseline | Threshold |")
lines.append("|--------|-----------|--------|---------|----------|-----------|")
for s in plan.drift_signals:
sub = s.subtype or "—"
lines.append(
f"| {s.name} | {s.drift_type.value} | {s.metric} | {sub} | "
f"{s.baseline} | {s.threshold} |"
)
lines.append("")
if plan.retraining_trigger:
rt = plan.retraining_trigger
lines.append("## Retraining Trigger\n")
lines.append(f"- **Type:** {rt.trigger_type.value}")
if rt.cadence:
lines.append(f"- **Cadence:** {rt.cadence}")
if rt.threshold_condition:
lines.append(f"- **Threshold condition:** {rt.threshold_condition}")
lines.append(f"- **Rationale:** {rt.rationale}\n")
lines.append("## PM Dashboard (Weekly Review)\n")
lines.append("| Metric | What to Watch | Frequency | User Impact |")
lines.append("|--------|--------------|-----------|-------------|")
for d in plan.dashboard_metrics:
lines.append(
f"| {d.name} | {d.metric} | {d.frequency} | "
f"{d.user_impact_translation} |"
)
lines.append("")
warnings = plan.check_threshold_quality()
if warnings:
lines.append("## Threshold Quality Warnings\n")
for w in warnings:
lines.append(f"- ⚠️ {w}")
lines.append("")
return "\n".join(lines)
def build_example_plan() -> MonitoringPlan:
plan = MonitoringPlan(feature_name="Customer Support Triage")
plan.add_signal(DriftSignal(
name="Critical ticket accuracy",
drift_type=DriftType.DATA_DRIFT,
metric="accuracy on critical-urgency tickets",
baseline=0.94,
threshold="alert when accuracy drops below 80% of baseline (75.2%)",
subtype="critical-urgency",
rationale="Critical tickets are high-stakes; per-subtype monitoring "
"catches regressions that blended metrics mask.",
))
plan.add_signal(DriftSignal(
name="New ticket type detection",
drift_type=DriftType.CONCEPT_DRIFT,
metric="fraction of tickets classified as 'unknown' category",
baseline=0.02,
threshold="alert when unknown fraction exceeds 10% for 3+ consecutive days",
subtype="unknown-category",
rationale="A surge in 'unknown' classifications signals the input "
"distribution has shifted — new ticket types the model "
"wasn't trained on are appearing.",
))
plan.add_signal(DriftSignal(
name="Overall latency",
drift_type=DriftType.DATA_DRIFT,
metric="p95 inference latency",
baseline=1.2,
threshold="alert when p95 exceeds 3.0 seconds",
rationale="Longer inputs (e.g., 20-page contracts vs 2-page memos) "
"increase inference time — a latency guardrail.",
))
plan.retraining_trigger = RetrainingTrigger(
trigger_type=TriggerType.BOTH,
cadence="quarterly scheduled retrain",
threshold_condition="immediate retrain when any subtype accuracy "
"drops below 80% of baseline for 3+ days",
rationale="Customer support triage is high-stakes and subject to "
"unpredictable drift (new products, new issue types). "
"Scheduled quarterly retraining maintains baseline freshness; "
"threshold-triggered retraining catches unexpected drift "
"between scheduled cycles.",
)
plan.add_dashboard_metric(DashboardMetric(
name="Critical ticket routing accuracy",
metric="per-subtype accuracy on critical-urgency tickets",
frequency="weekly",
user_impact_translation="If this drops, urgent customer issues sit "
"unresolved longer — direct CSAT and churn risk.",
))
plan.add_dashboard_metric(DashboardMetric(
name="Unknown category rate",
metric="fraction of tickets classified as 'unknown'",
frequency="weekly",
user_impact_translation="Rising unknown rate means the model can't "
"categorize new issue types — tickets get "
"manual-routed, increasing response time.",
))
plan.add_dashboard_metric(DashboardMetric(
name="Support ticket backlog",
metric="number of tickets in queue > 24 hours",
frequency="weekly",
user_impact_translation="A lagging indicator — if this rises while "
"model metrics are stable, investigate "
"non-model causes. If it rises alongside "
"a model regression, the model is the cause.",
))
return plan
if __name__ == "__main__":
plan = build_example_plan()
print(render_monitoring_plan(plan))
warnings = plan.check_threshold_quality()
if not warnings:
print("✅ All threshold quality checks passed.")
else:
print(f"⚠️ {len(warnings)} warning(s) — see plan output above.")