Phase 8: Launch, Metrics & Scaling AI Products · 40 min · Python

Cross-Functional Leadership for AI Products

The PM who stops talking to ML engineers after launch is the PM whose feature quietly degrades for three months before anyone notices.

What you will learn

Partner with ML engineers and data scientists post-launch, not just during build
Run a model review meeting: what it covers, who owns what, and how often it should happen
Run an eval review meeting: turning eval deltas into ship/hold decisions
Navigate escalation and decision rights when a PM and an ML lead disagree on a launch call

The Problem

A team ships an AI-powered content recommendation feature. During the build phase, the PM and ML lead met daily — discussing model architecture, eval results, tradeoffs, and timelines. The feature launches successfully. The PM moves on to the next project. The ML lead goes back to model improvements. Three months later, the PM is surprised to learn in a quarterly review that the feature's click-through rate has dropped 22% because the model drifted (new content types emerged that the model wasn't trained on), and nobody escalated it because "the PM wasn't in the model review meetings anymore" and "we assumed someone else was watching the metrics." The feature that launched with celebration is now underperforming, and the PM — who should have been the first to know — was the last.

This is the post-launch partnership gap. During build, the PM and ML team are tightly coupled: daily standups, shared Slack channels, joint decision-making. After launch, the PM often disengages — "the ML team has it from here" — and the ML team often stops communicating upward — "the PM moved on, they don't need to know." Both assumptions are wrong. Post-launch is when the model is most vulnerable (it's encountering real-world data for the first time), and the PM is the bridge between model health and business impact. Disengaging means losing visibility into a critical aspect of product quality.

Post-launch partnership: what changes

The partnership doesn't end after launch — it changes in character:

Dimension	Build Phase	Post-Launch Phase
Frequency	Daily	Weekly or biweekly (model review)
Focus	Building the model	Monitoring, maintaining, improving the model
PM's role	Define requirements, prioritize features	Translate model metrics to user impact, make ship/hold decisions
ML's role	Build, train, evaluate	Monitor, retrain, diagnose regressions
Key meetings	Design reviews, eval reviews	Model reviews, eval reviews, incident response
Decision type	"What should we build?"	"Should we ship this model update? Should we retrain?"

The PM's post-launch job is not to stop engaging but to engage differently: less "what should we build?" and more "is what we built still working, and what should we do about it?"

The model review meeting

A model review is a recurring meeting (weekly or biweekly) where the PM and ML team review the model's production performance. It's not a status update — it's a diagnostic and decision-making meeting.

Agenda (30-45 minutes):

Performance vs. baseline (10 min): ML presents current metrics vs. eval baseline. PM asks: "What does this mean for users? Which segments are affected?" If a metric dropped, ML explains the technical cause; PM translates to user impact.

Drift indicators (5 min): ML presents drift signals (data drift, concept drift). PM asks: "Are we seeing new input types? Should we update the training data?"

Incident review (5 min, if applicable): If there was an incident since the last review, PM presents user impact (support tickets, complaints, churn signals), ML presents technical root cause and fix status.

Retraining decision (10 min): Joint decision: should we retrain? On what data? What's the expected timeline? PM owns the product impact assessment ("is the current degradation acceptable while we retrain, or do we need to roll back?"); ML owns the technical assessment ("retraining will take 2 weeks; we can ship a partial fix in 3 days").

Action items and owners (5 min): Specific tasks, owners, and deadlines. Not "we should look into that" — "ML team investigates retraining data by Friday; PM communicates timeline to stakeholders by Wednesday."

The PM's unique contribution is the user-impact translation
An ML engineer sees "false-negative rate on critical tickets increased from 6% to 21%." A PM translates: "critical tickets are being misrouted, which means urgent customer issues sit unresolved for hours longer — this is a CSAT and churn risk that costs us approximately $X per week in at-risk revenue." Both perspectives are needed. Without the ML perspective, the PM doesn't know what's happening. Without the PM perspective, the ML team doesn't know whether it matters to users.

A model review meeting's agenda covers performance vs. baseline and drift indicators, but has no dedicated retraining-decision item — retraining just happens whenever the ML lead personally decides it's warranted, without a joint discussion. Three months in, the PM discovers a retrain shipped two weeks ago that changed the model's behavior on a segment the PM was actively fielding stakeholder questions about. What structural gap does this reveal?

This lesson's agenda explicitly makes the retraining decision a joint item — not because the ML team needs permission to do their job, but because retraining changes production behavior in ways that have real product and stakeholder consequences the ML team may not have visibility into (ongoing stakeholder conversations, timing sensitivities, communication needs). A model review agenda that omits this item lets exactly this kind of surprise happen: a technically sound decision that blindsides the PM and disrupts stakeholder trust simply because there was no structural checkpoint requiring the two sides to sync before it shipped.

The eval review meeting

An eval review happens when there's a new model version ready to potentially replace the production model. It's the ship/hold decision meeting.

Process (30 minutes):

Present the delta (10 min): ML presents the new version's eval results vs. the current production version. Focus on deltas: where is the new version better, where is it worse, where is it unchanged?

Check guardrails (5 min): Did the new version regress on any guardrail metric (latency, cost, safety)? If yes, this is a potential hold — even if quality improved.

Check per-subtype performance (5 min): Did the new version regress on any critical subtype? A blended improvement that masks a per-subtype regression is a hold — same principle as the metrics lesson.

Ship/hold decision (10 min): PM makes the call based on product impact. "The new version is 3% better overall and doesn't regress on any guardrail — ship it." Or: "The new version is 2% better overall but regresses 5% on the critical-urgency subtype — hold until we fix that regression, because the critical subtype is what matters most for customer satisfaction."

The PM owns the ship/hold decision because it's a product judgment: "is this improvement worth the risk of the regression?" The ML team provides the data; the PM makes the call.

Escalation and decision rights

When the PM and ML lead disagree on a ship/hold or retrain decision, the escalation path should already exist — not be improvised in the moment.

Escalation path:

Data-driven resolution (first attempt): PM and ML lead attempt to resolve with data. PM quantifies user impact of each option. ML quantifies technical implications. Often, the data makes the decision clear.

Shared manager escalation (if unresolved): Escalate to the shared manager (e.g., Director of Product / Director of Engineering) with a one-page trade-off summary: the decision, the options, the data for each, and the disagreement. The manager makes the call based on the summary, not by re-litigating the technical details.

VP escalation (if still unresolved): For high-stakes decisions where the shared manager can't decide. Rare — most disagreements resolve at step 1 or 2.

The one-page trade-off summary template:

Decision needed: [ship/hold/retrain/rollback]
Option A: [description] — user impact: [X], technical risk: [Y]
Option B: [description] — user impact: [X], technical risk: [Y]
PM recommendation: [A/B] because [product rationale]
ML recommendation: [A/B] because [technical rationale]
Key disagreement: [the specific point of contention]

A PM wants to ship a model update that improves overall accuracy by 3% but increases p95 latency by 200ms. The ML lead says the latency increase is "unacceptable" and wants to hold. The PM says the accuracy improvement is "worth it for users." How should this be resolved?

This is a product trade-off, not a purely technical one. The resolution: check if the latency increase violates the SLA (objective test). If it doesn't, the accuracy improvement likely wins. If it does, quantify the user impact of both options and make a data-driven call. The PM owns "what matters more to users: accuracy or speed?" The ML lead owns "what are the technical implications?" Both perspectives feed into the decision. If they still disagree, use the escalation path with a one-page trade-off summary.

Build

Build it

The code artifact lesson_08_05_model_review.py implements a cross-functional playbook builder:

ReviewSection — a dataclass for one section of a model review meeting: name, owner (PM/ML/JOINT), duration, and key questions
ShipHoldCriterion — a dataclass for one ship/hold criterion: metric, condition (improve/no-regress), and threshold
EscalationStep — a dataclass for one step in the escalation path: step number, who decides, and required input
CrossFunctionalPlaybook — a collection of review sections, ship/hold criteria, escalation steps, and role boundaries
check_playbook_completeness() — flags missing sections, unclear ownership, or missing escalation steps
render_playbook() — produces a markdown document with the full playbook
A worked example: the content recommendation feature from this lesson's opening, redesigned with a proper model review cadence, eval review process, and escalation path

"""
lesson_08_05_model_review.py

A cross-functional playbook builder for AI product teams.

Designs model review meetings, eval review processes, ship/hold
criteria, and escalation paths for PM/ML disagreements post-launch.

Usage:
    python lesson_08_05_model_review.py

No external dependencies, no network calls, no API keys required.
"""

from __future__ import annotations

from dataclasses import dataclass, field
from enum import Enum
from typing import Optional

class Owner(Enum):
    PM = "PM"
    ML = "ML"
    JOINT = "Joint"

class Condition(Enum):
    IMPROVE = "must_improve"
    NO_REGRESS = "must_not_regress"

@dataclass
class ReviewSection:
    name: str
    owner: Owner
    duration_min: int
    key_questions: list[str] = field(default_factory=list)

@dataclass
class ShipHoldCriterion:
    metric: str
    condition: Condition
    threshold: str = ""
    rationale: str = ""

@dataclass
class EscalationStep:
    step: int
    decider: str
    required_input: str
    timeline: str = ""

@dataclass
class RoleBoundary:
    pm_owns: str
    ml_owns: str

@dataclass
class CrossFunctionalPlaybook:
    feature_name: str
    review_cadence: str = ""
    review_sections: list[ReviewSection] = field(default_factory=list)
    ship_hold_criteria: list[ShipHoldCriterion] = field(default_factory=list)
    escalation_steps: list[EscalationStep] = field(default_factory=list)
    role_boundary: Optional[RoleBoundary] = None
    disagreement_scenario: str = ""

    def check_completeness(self) -> list[str]:
        warnings = []
        if not self.review_sections:
            warnings.append("No model review sections defined.")
        if not self.ship_hold_criteria:
            warnings.append("No ship/hold criteria defined.")
        if not self.escalation_steps:
            warnings.append("No escalation path defined.")
        if not self.role_boundary:
            warnings.append("No role boundary defined — unclear who owns what.")
        if not self.disagreement_scenario:
            warnings.append("No disagreement scenario described — "
                          "escalation path is untested.")
        has_joint = any(s.owner == Owner.JOINT for s in self.review_sections)
        if not has_joint:
            warnings.append("No joint-ownership sections in model review — "
                          "some decisions require both PM and ML input.")
        return warnings

    def add_section(self, section: ReviewSection) -> "CrossFunctionalPlaybook":
        self.review_sections.append(section)
        return self

    def add_criterion(self, criterion: ShipHoldCriterion) -> "CrossFunctionalPlaybook":
        self.ship_hold_criteria.append(criterion)
        return self

    def add_step(self, step: EscalationStep) -> "CrossFunctionalPlaybook":
        self.escalation_steps.append(step)
        return self

def render_playbook(pb: CrossFunctionalPlaybook) -> str:
    lines = [f"# Cross-Functional Playbook: {pb.feature_name}\n"]

    lines.append(f"**Review cadence:** {pb.review_cadence}\n")

    lines.append("## Model Review Meeting\n")
    lines.append("| Section | Owner | Duration | Key Questions |")
    lines.append("|---------|-------|----------|--------------|")
    for s in pb.review_sections:
        qs = "; ".join(s.key_questions)
        lines.append(f"| {s.name} | {s.owner.value} | {s.duration_min} min | {qs} |")
    lines.append("")

    lines.append("## Ship/Hold Criteria (Eval Review)\n")
    for c in pb.ship_hold_criteria:
        cond = "must improve" if c.condition == Condition.IMPROVE else "must not regress"
        lines.append(f"- **{c.metric}**: {cond}")
        if c.threshold:
            lines.append(f"  - Threshold: {c.threshold}")
        if c.rationale:
            lines.append(f"  - Rationale: {c.rationale}")
    lines.append("")

    lines.append("## Escalation Path (PM/ML Disagreement)\n")
    lines.append("| Step | Decider | Required Input | Timeline |")
    lines.append("|------|---------|----------------|----------|")
    for s in pb.escalation_steps:
        lines.append(f"| {s.step} | {s.decider} | {s.required_input} | {s.timeline} |")
    lines.append("")

    if pb.role_boundary:
        lines.append("## Role Boundaries\n")
        lines.append(f"**PM owns:** {pb.role_boundary.pm_owns}\n")
        lines.append(f"**ML owns:** {pb.role_boundary.ml_owns}\n")

    if pb.disagreement_scenario:
        lines.append("## Disagreement Scenario (Example)\n")
        lines.append(pb.disagreement_scenario + "\n")

    warnings = pb.check_completeness()
    if warnings:
        lines.append("## Completeness Warnings\n")
        for w in warnings:
            lines.append(f"- ⚠️ {w}")
        lines.append("")

    return "\n".join(lines)

def build_example_playbook() -> CrossFunctionalPlaybook:
    pb = CrossFunctionalPlaybook(
        feature_name="Content Recommendation AI",
        review_cadence="Weekly, 30 minutes, PM + ML lead",
    )

    pb.add_section(ReviewSection(
        name="Performance vs. baseline",
        owner=Owner.ML,
        duration_min=10,
        key_questions=[
            "What's the current CTR vs. baseline?",
            "Any per-segment regressions?",
            "What does this mean for users? (PM translates)",
        ],
    ))

    pb.add_section(ReviewSection(
        name="Drift indicators",
        owner=Owner.ML,
        duration_min=5,
        key_questions=[
            "New content types emerging?",
            "Input distribution shifting?",
            "Should we update training data?",
        ],
    ))

    pb.add_section(ReviewSection(
        name="Incident review",
        owner=Owner.JOINT,
        duration_min=5,
        key_questions=[
            "Any incidents since last review?",
            "User impact (PM) + root cause (ML)?",
            "Fix status and timeline?",
        ],
    ))

    pb.add_section(ReviewSection(
        name="Retraining decision",
        owner=Owner.JOINT,
        duration_min=10,
        key_questions=[
            "Should we retrain? On what data?",
            "Is current degradation acceptable while retraining, or rollback?",
            "Timeline and stakeholder communication?",
        ],
    ))

    pb.add_criterion(ShipHoldCriterion(
        metric="Overall CTR",
        condition=Condition.IMPROVE,
        threshold="at least 1% relative improvement",
        rationale="Minor improvements aren't worth the deployment risk.",
    ))

    pb.add_criterion(ShipHoldCriterion(
        metric="Per-segment CTR (any segment)",
        condition=Condition.NO_REGRESS,
        threshold="no segment regresses more than 2% relative",
        rationale="A blended improvement that masks a segment regression "
                  "is a hold — same principle as the metrics lesson.",
    ))

    pb.add_criterion(ShipHoldCriterion(
        metric="p95 latency",
        condition=Condition.NO_REGRESS,
        threshold="p95 must stay below 500ms",
        rationale="Latency SLA — regression here affects user experience "
                  "even if accuracy improves.",
    ))

    pb.add_step(EscalationStep(
        step=1,
        decider="PM + ML lead (joint, data-driven)",
        required_input="User impact analysis (PM) + technical assessment (ML)",
        timeline="Same day",
    ))

    pb.add_step(EscalationStep(
        step=2,
        decider="Director of Product / Director of Engineering",
        required_input="One-page trade-off summary with options, data, and recommendations",
        timeline="Within 48 hours",
    ))

    pb.add_step(EscalationStep(
        step=3,
        decider="VP Product / VP Engineering",
        required_input="Escalated trade-off summary with prior decision rationale",
        timeline="Within 1 week",
    ))

    pb.role_boundary = RoleBoundary(
        pm_owns="User impact translation, stakeholder communication, "
                "ship/hold decisions, product trade-off assessment",
        ml_owns="Technical assessment, retraining execution, "
                "monitoring infrastructure, model quality diagnostics",
    )

    pb.disagreement_scenario = (
        "PM wants to ship a model update that improves CTR by 3% but increases "
        "p95 latency by 80ms (from 420ms to 500ms — right at the SLA boundary). "
        "ML lead says the latency increase is risky and wants to hold. "
        "Resolution: Step 1 — PM quantifies user impact of +3% CTR (estimated "
        "$12K/month in additional engagement value). ML quantifies risk of "
        "500ms latency (5% of users already at 450ms+, so 500ms pushes them "
        "over the SLA). Both agree the SLA boundary risk is real. Decision: "
        "hold until ML can reduce the latency increase to <50ms, then ship. "
        "Documented in trade-off summary, communicated to stakeholders."
    )

    return pb

if __name__ == "__main__":
    pb = build_example_playbook()
    print(render_playbook(pb))
    warnings = pb.check_completeness()
    if not warnings:
        print("✅ All completeness checks passed.")
    else:
        print(f"⚠️ {len(warnings)} warning(s) — see playbook output above.")

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Check Yourself — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.

Key terms in this lesson

Latency