Phase 3: Data Strategy for AI Products · 40 min · Python

Data Flywheels

A flywheel is a claim about causality, not a diagram with an arrow pointing back to the start.

What you will learn

Diagram the four stages of a data flywheel: usage, signal, retrain, redeploy
Distinguish a real, compounding data flywheel from a vanity-metrics loop that never actually improves the model
Design a bootstrapping plan for the cold-start problem before a flywheel has enough usage to spin
Evaluate whether a proposed flywheel is defensible enough to function as a product moat

The Problem

A team building an AI-powered resume-screening tool puts a slide in their board deck: a circular diagram with four boxes — "Usage," "Data," "Better Model," "More Usage" — arrows connecting each to the next, labeled "Our Data Flywheel." The board is impressed. Eighteen months later, a new PM joins, asks to see the retraining cadence, and finds out the model has been retrained exactly once, on the day it launched. The "usage" the diagram promised was compounding into model improvement was, in fact, sitting in a logging table nobody had built a pipeline to read from. The flywheel was real as a diagram and fictional as a mechanism — every usage event that should have been closing the loop was instead accumulating, unused, in storage the team was paying for.

This is the most common failure mode in how AI PMs talk about flywheels: treating the diagram as the deliverable instead of treating it as a claim that needs four separate, verifiable mechanisms behind it. A flywheel isn't "we have usage data and a model" — every product with users and a model technically has that. A flywheel is a specific, working pipeline: usage produces a signal worth capturing, that signal gets turned into labeled or otherwise usable training data, that data measurably improves the model when it's retrained, and the improved model drives more or better usage, which produces more signal. If any one of those four links is missing, you don't have a flywheel — you have a static model with a growing, unused log file.

The four stages, and where they actually break

Stage	What has to be true	Where it silently breaks
Usage	Users are interacting with the AI feature in a way that generates a signal about quality	Usage exists, but the interaction never generates a signal — a user reading a generated summary and closing the tab tells you nothing about whether the summary was good
Signal	The interaction produces something that indicates quality — an edit, a thumbs-down, an acceptance, a correction, a regeneration request	Signal is collected but never structured — "user clicked away" gets logged as an undifferentiated event instead of being distinguished from "user copied the output," which mean opposite things
Retrain	Someone actually builds a pipeline that turns accumulated signal into a retraining or fine-tuning run on a fixed cadence	The pipeline exists on a roadmap slide, not in a scheduler — this is the most common real-world break, exactly as in the resume-screening example
Redeploy	The retrained model actually gets evaluated and shipped, with a before/after comparison proving it's better	The retrain happens, but nobody runs an eval to confirm the new model actually improved, so a worse model could ship and nobody would know until users complain

Each stage has to be independently true, and each one is a specific, ownable piece of infrastructure — not a mindset. A team can be extremely rigorous about the Usage and Signal stages (great instrumentation, well-structured feedback) and still have a completely fake flywheel because the Retrain stage doesn't exist as a running pipeline, only as an intention.

The fastest way to test if a flywheel is real: ask "when was the model last retrained on this signal, and what changed?"
If nobody in the room can answer with a date and a specific, measured delta (accuracy up X%, override rate down Y%), the flywheel is a diagram, not a mechanism. A real flywheel has a retraining cadence you could put on a calendar and a running log of before/after eval deltas for every retrain. If the honest answer is "we have all the data, we just haven't gotten to retraining yet," that's not a flywheel in progress — it's Stage 3 not existing, and every claim about the product's data moat built on top of it is currently unsupported.

Cold start: bootstrapping the first turn

A flywheel's entire value proposition — that data compounds — requires a first turn before it can compound anything, and early on, there isn't enough usage to generate meaningful signal. Most successful early-stage flywheels are bootstrapped with a source of signal that isn't the flywheel itself: purchased or licensed labeled data, a small internal team hand-labeling for a few weeks, a beta cohort explicitly recruited and paid or incentivized for detailed feedback rather than passive usage signal, or synthetic data used as a scaffold with a clear plan to replace it as real usage signal accumulates. The mistake is assuming organic usage will bootstrap itself — with too little usage, the signal is too sparse and noisy to retrain anything meaningfully, so the flywheel never spins fast enough to reach the volume where it becomes self-sustaining, and the team either gives up or (worse) ships retraining runs on statistically meaningless sample sizes and calls the resulting noise "improvement."

A team's AI writing assistant has been live for 3 months with 200 daily active users. The PM wants to start a weekly retraining cadence using only in-product thumbs-up/down signal, which currently produces about 40 labeled events per week. What is the most likely outcome, and what should the PM do instead?

40 labeled events per week is too small and noisy a sample to reliably show a real quality signal versus random variation — retraining on it weekly risks shipping model changes that are essentially fitting noise, which can just as easily degrade quality as improve it, with no way to tell the difference at that sample size. The fix is either to slow the cadence (monthly or quarterly, accumulating enough signal to be meaningful) or bootstrap additional signal (a paid annotation batch, a more deliberately recruited feedback cohort) so there's enough real signal to retrain on a faster cadence responsibly. Increasing frequency without increasing volume makes the noise problem worse, not better.

Is the flywheel actually a moat?

Not every real, working flywheel is a defensible one. A flywheel is a moat when the specific data it accumulates is (a) hard for a competitor to replicate quickly, (b) genuinely causally connected to output quality in a way users can perceive, and (c) accumulating faster than the market is growing, so the leader's lead compounds rather than just tracks market size. A generic thumbs-up/down signal on generic outputs is real signal, but it's often replicable by any competitor with comparable usage — it's not much of a moat. A flywheel built on proprietary, hard-to-replicate context (a company's own historical support resolutions, a specific enterprise customer's internal terminology corrected over months of use) is much closer to a real moat, because a competitor starting today can't buy or scrape their way to that same corpus. Part of the PM's job in flywheel design is being honest about which category a given flywheel falls into — a working flywheel that isn't defensible is still worth building, but it shouldn't be pitched to a board as a moat.

Two competing AI note-taking apps both have working flywheels: usage generates thumbs-up/down signal, a retraining pipeline runs monthly, and each retrain is measurably better than the last. Company A's signal is generic thumbs-up/down on summary quality. Company B's signal is corrections users make to auto-generated meeting-attendee names and action-item owners, drawn from each customer's own organizational chart data synced via their calendar integration. Which flywheel is the stronger competitive moat, and why?

Both flywheels are real and working — that's necessary but not sufficient for a moat. The distinguishing factor is replicability: Company A's generic thumbs-up/down signal on summary quality is a pattern any competitor with similar usage volume could reproduce within a couple of quarters, so it's a working flywheel without much defensibility. Company B's signal is tied to something a competitor genuinely cannot access without that specific customer's own organizational data synced over time — that's the kind of proprietary, hard-to-replicate context that makes a flywheel function as an actual competitive moat, not just a quality-improvement mechanism.

Build

Build it

The code artifact lesson_03_03_flywheel_designer.py implements a data flywheel design and health-check tool:

FlywheelStage enum for the four stages (Usage, Signal, Retrain, Redeploy)
StageStatus — for each stage, whether it's WORKING, PARTIAL, or MISSING, plus supporting evidence text
FlywheelAssessment — scores whether a flywheel is real (all four stages working), partial, or a diagram-only fake, and flags the weakest link
MoatAssessment — scores defensibility against replicability, causal connection to quality, and compounding-vs-market-growth criteria
render_flywheel_report() — produces a markdown report combining the stage assessment and moat assessment into a single design document
A worked example: the resume-screening tool from this lesson's opening, assessed honestly (Usage and Signal working, Retrain missing) alongside a redesigned version with a concrete bootstrapping and retraining plan

"""
lesson_03_03_flywheel_designer.py

A data flywheel design and health-check tool.

Assesses whether each of the four flywheel stages (Usage, Signal, Retrain,
Redeploy) is actually working, identifies the weakest link, and separately
scores whether a working flywheel is defensible enough to function as a
product moat. This is the artifact a PM uses to replace a flywheel diagram
on a slide with an honest, verifiable design document.

Usage:
    python lesson_03_03_flywheel_designer.py

No external dependencies, no network calls, no API keys required.
"""

from __future__ import annotations

from dataclasses import dataclass, field
from enum import Enum

class FlywheelStage(Enum):
    USAGE = "Usage"
    SIGNAL = "Signal"
    RETRAIN = "Retrain"
    REDEPLOY = "Redeploy"

class StageStatus(Enum):
    WORKING = "working"
    PARTIAL = "partial"
    MISSING = "missing"

@dataclass
class StageAssessment:
    stage: FlywheelStage
    status: StageStatus
    evidence: str  # what's actually true right now, in specific terms

@dataclass
class FlywheelAssessment:
    product_name: str
    stages: list[StageAssessment]

    def weakest_stage(self) -> StageAssessment:
        order = {StageStatus.MISSING: 0, StageStatus.PARTIAL: 1, StageStatus.WORKING: 2}
        return min(self.stages, key=lambda s: order[s.status])

    def is_real_flywheel(self) -> bool:
        return all(s.status == StageStatus.WORKING for s in self.stages)

    def verdict(self) -> str:
        if self.is_real_flywheel():
            return "Real, working flywheel"
        if any(s.status == StageStatus.WORKING for s in self.stages):
            return "Partial flywheel — at least one stage is not yet a running mechanism"
        return "Diagram-only flywheel — no stage is confirmed working end-to-end"

@dataclass
class MoatFactor:
    name: str
    score: int  # 1-5, 5 = strongly defensible
    rationale: str

@dataclass
class MoatAssessment:
    factors: list[MoatFactor] = field(default_factory=list)

    def overall_score(self) -> float:
        if not self.factors:
            return 0.0
        return sum(f.score for f in self.factors) / len(self.factors)

    def label(self) -> str:
        score = self.overall_score()
        if score >= 4:
            return "Defensible moat"
        if score >= 2.5:
            return "Weak moat — real advantage, but replicable over time"
        return "Not a moat — treat as a quality mechanism, not a competitive barrier"

def render_flywheel_report(flywheel: FlywheelAssessment, moat: MoatAssessment) -> str:
    lines = [
        f"# Data Flywheel Design Report — {flywheel.product_name}",
        "",
        f"**Flywheel verdict:** {flywheel.verdict()}",
        "",
        "## Stage-by-stage assessment",
        "",
        "| Stage | Status | Evidence |",
        "|---|---|---|",
    ]
    for s in flywheel.stages:
        lines.append(f"| {s.stage.value} | {s.status.value.upper()} | {s.evidence} |")

    weakest = flywheel.weakest_stage()
    lines += [
        "",
        f"**Weakest link:** {weakest.stage.value} ({weakest.status.value}) — "
        f"this is the highest-leverage stage to fix before the flywheel claim is credible.",
        "",
        "## Moat assessment",
        "",
        f"**Overall moat score:** {moat.overall_score():.1f}/5 ({moat.label()})",
        "",
        "| Factor | Score | Rationale |",
        "|---|---|---|",
    ]
    for f in moat.factors:
        lines.append(f"| {f.name} | {f.score}/5 | {f.rationale} |")

    return "\n".join(lines)

# ── Worked example ───────────────────────────────────────────────────────────
# The resume-screening tool from the lesson's opening: usage and signal are
# real, retrain is missing (the actual, common failure mode), redeploy is
# therefore also not happening. Moat factors scored honestly.

def build_sample_assessment() -> tuple[FlywheelAssessment, MoatAssessment]:
    flywheel = FlywheelAssessment(
        product_name="Resume Screening Copilot",
        stages=[
            StageAssessment(
                FlywheelStage.USAGE,
                StageStatus.WORKING,
                "1,200 recruiters actively screening resumes weekly through the tool.",
            ),
            StageAssessment(
                FlywheelStage.SIGNAL,
                StageStatus.WORKING,
                "Every recruiter accept/reject/override of a model recommendation is logged "
                "with a reason code.",
            ),
            StageAssessment(
                FlywheelStage.RETRAIN,
                StageStatus.MISSING,
                "No retraining pipeline exists. The model has been retrained exactly once, "
                "at launch, 18 months ago. Override data has been accumulating unused.",
            ),
            StageAssessment(
                FlywheelStage.REDEPLOY,
                StageStatus.MISSING,
                "Since no retraining occurs, there is nothing to redeploy or eval against a "
                "before/after comparison.",
            ),
        ],
    )
    moat = MoatAssessment(
        factors=[
            MoatFactor(
                "Replicability",
                2,
                "Override/reason-code signal is a common pattern any competitor with similar "
                "usage volume could replicate within a couple of quarters.",
            ),
            MoatFactor(
                "Causal connection to quality",
                4,
                "Recruiter overrides with reason codes are a strong, direct quality signal — "
                "if retraining actually happened, this would meaningfully move accuracy.",
            ),
            MoatFactor(
                "Compounding vs. market growth",
                1,
                "Irrelevant while Retrain is missing — zero compounding is currently occurring "
                "regardless of usage growth.",
            ),
        ]
    )
    return flywheel, moat

if __name__ == "__main__":
    flywheel, moat = build_sample_assessment()
    print(render_flywheel_report(flywheel, moat))
    print()
    print(
        "Takeaway: Usage and Signal are both genuinely working here, which is exactly what "
        "makes this flywheel dangerous to describe as real -- two working stages out of four "
        "creates the appearance of a data moat on a slide while the actual retraining "
        "mechanism, the stage that would make any of this matter, does not exist. The fix is "
        "not more instrumentation; it's standing up the Retrain and Redeploy pipeline against "
        "signal that's already sitting there unused."
    )

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Check Yourself — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.