Phase 8: Launch, Metrics & Scaling AI Products · 40 min · Python

Defining AI Product Metrics & KPIs

A single 'AI health score' hides exactly the information you need when something breaks: which of quality, adoption, or cost actually broke.

Hiring signal: The research grounding this course names 'defining/tracking model + business metrics (lift, false positive rate, latency)' as core AI PM duty, distinct from a traditional PM's metrics work. A candidate who can build a metric tree that separates quality from adoption from cost — instead of one blended 'engagement' number — demonstrates the data literacy the interview loop's technical-fluency round is specifically designed to test.

What you will learn

Distinguish north star metrics from guardrail metrics for an AI feature
Separate quality, adoption, and cost metrics instead of blending them into one health score
Identify metrics that can be gamed by a model that's technically 'engaging' but not actually helpful
Build a metric tree connecting a feature-level metric to a business outcome

The Problem

A team ships an AI writing assistant and tracks one dashboard number: "AI Feature Health Score," a blended metric combining usage frequency, session length, and a rough sentiment classifier on user feedback text, normalized into a single 0-100 score. For the first two months, the score climbs steadily — 62, 68, 74 — and the team reports it proudly in the all-hands as proof the feature is working. In month three, a support ticket surge reveals the truth: the model quietly regressed after an unrelated infrastructure change, generating noticeably lower-quality text, but usage frequency kept climbing anyway because the feature was now embedded in a workflow users felt obligated to use even when the output was mediocre, and session length went up because users were spending more time editing bad drafts, not because they loved the tool more. The blended score kept rising through all of this, because two of its three inputs (usage, session length) went up for exactly the wrong reason while the third (sentiment) was too noisy and lagging to move the needle before the score had already told a false success story for weeks.

This is what happens when quality, adoption, and cost get collapsed into one number: each is trying to tell you something different, and averaging them together produces a metric that's technically responsive to reality but tells you nothing about which reality changed. A rising blended score can mean the feature genuinely improved, or it can mean adoption metrics are compensating for a quality regression, or cost is spiraling while everything else looks fine — and you cannot tell which from the blended number alone. The fix isn't a better blending formula; it's refusing to blend in the first place.

North Star vs. Guardrail Metrics

A north star metric is the single measure that best captures whether the feature is delivering its core value — it should move when the feature genuinely gets better at its job and shouldn't move (or shouldn't move much) for reasons unrelated to that job. Guardrail metrics are the metrics you're not trying to optimize but are watching to make sure the north star doesn't improve at their expense — cost per user, latency, error rate, complaint rate. A north star metric without guardrails invites exactly the kind of gaming that happened in the opening example: usage climbed (looked like north-star progress) while quality silently degraded (a guardrail failure nobody was watching separately). The discipline is choosing a north star that's hard to move without the underlying thing actually getting better, and a small set of guardrails that catch the most likely ways someone (or some optimization process) could move the north star for the wrong reasons.

A metric that moves for the wrong reason isn't a bad metric — it's an unguarded one
Usage frequency isn't inherently a bad metric to track; the opening example's mistake wasn't choosing usage as an input, it was blending it with other signals into one score with no separate guardrail catching the specific failure mode (quality dropping while usage rises because the feature became obligatory rather than good). Almost any single metric can be gamed or can rise for a bad reason in some scenario. The fix isn't finding a perfect, ungameable metric — it's pairing your north star with the specific guardrail metrics that would catch the most plausible way it could rise for a bad reason, and watching them separately, not folded into the same number.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Quality, Adoption, and Cost: Three Separate Tracks, Building a Metric Tree, Build It, What to Practice — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.

Key terms in this lesson

Guardrails