The Problem
A team ships an AI writing assistant and tracks one dashboard number: "AI Feature Health Score," a blended metric combining usage frequency, session length, and a rough sentiment classifier on user feedback text, normalized into a single 0-100 score. For the first two months, the score climbs steadily — 62, 68, 74 — and the team reports it proudly in the all-hands as proof the feature is working. In month three, a support ticket surge reveals the truth: the model quietly regressed after an unrelated infrastructure change, generating noticeably lower-quality text, but usage frequency kept climbing anyway because the feature was now embedded in a workflow users felt obligated to use even when the output was mediocre, and session length went up because users were spending more time editing bad drafts, not because they loved the tool more. The blended score kept rising through all of this, because two of its three inputs (usage, session length) went up for exactly the wrong reason while the third (sentiment) was too noisy and lagging to move the needle before the score had already told a false success story for weeks.
This is what happens when quality, adoption, and cost get collapsed into one number: each is trying to tell you something different, and averaging them together produces a metric that's technically responsive to reality but tells you nothing about which reality changed. A rising blended score can mean the feature genuinely improved, or it can mean adoption metrics are compensating for a quality regression, or cost is spiraling while everything else looks fine — and you cannot tell which from the blended number alone. The fix isn't a better blending formula; it's refusing to blend in the first place.
North star vs. guardrail metrics
A north star metric is the single measure that best captures whether the feature is delivering its core value — it should move when the feature genuinely gets better at its job and shouldn't move (or shouldn't move much) for reasons unrelated to that job. Guardrail metrics are the metrics you're not trying to optimize but are watching to make sure the north star doesn't improve at their expense — cost per user, latency, error rate, complaint rate. A north star metric without guardrails invites exactly the kind of gaming that happened in the opening example: usage climbed (looked like north-star progress) while quality silently degraded (a guardrail failure nobody was watching separately). The discipline is choosing a north star that's hard to move without the underlying thing actually getting better, and a small set of guardrails that catch the most likely ways someone (or some optimization process) could move the north star for the wrong reasons.
A metric that moves for the wrong reason isn't a bad metric — it's an unguarded one
Usage frequency isn't inherently a bad metric to track; the opening example's mistake wasn't choosing usage as an input, it was blending it with other signals into one score with no separate guardrail catching the specific failure mode (quality dropping while usage rises because the feature became obligatory rather than good). Almost any single metric can be gamed or can rise for a bad reason in some scenario. The fix isn't finding a perfect, ungameable metric — it's pairing your north star with the specific guardrail metrics that would catch the most plausible way it could rise for a bad reason, and watching them separately, not folded into the same number.
Quality, adoption, and cost: three separate tracks
- Quality metrics measure whether the AI output is actually good on its own terms — accuracy against a golden set, hallucination rate, guardrail violation rate, human-eval rubric scores. These should be measurable independent of whether anyone is using the feature yet (that's exactly what an eval suite does before launch, and continues to sample after launch).
- Adoption metrics measure whether people are choosing to use the feature and whether that usage reflects genuine value — activation rate, retention, feature-specific engagement, but also more diagnostic ones like "regeneration rate" (how often a user asks for a different output, a proxy for first-output quality) or "edit distance" (how much a user changes a generated draft before using it, a proxy for how close-to-final the output actually was).
- Cost metrics measure what the feature costs to run at current usage — cost per request, cost per active user, infrastructure spend trend — covered in depth in Phase 4's cost modeling and Lesson 4 of this phase, but tracked continuously post-launch, not just projected once before launch.
Tracking these three separately, even if you still report a single north star metric drawn primarily from adoption, means a quality regression shows up as a quality-track alarm regardless of what adoption is doing, and a cost spike shows up as a cost-track alarm regardless of what quality is doing — which is exactly the diagnostic power the opening example's blended score destroyed.
A team's AI code-review assistant shows rising "suggestions accepted per week" (an adoption metric) over two months. A closer look at the quality track shows the false-positive rate on suggestions (flagging correct code as buggy) has also been rising over the same period. What is the most likely explanation for both trends happening together, and what does it reveal about using "suggestions accepted" alone as a north star?
A plausible and common failure mode is that as false-positive rate rises, developers -- especially under time pressure -- start accepting suggestions with less real scrutiny, either because there are now too many to carefully evaluate each one or because trust in flagging has eroded into reflexive compliance. That would make 'suggestions accepted' rise for a reason that has nothing to do with genuine improvement, and possibly for a reason connected to the tool getting worse. This is exactly why adoption metrics need paired quality guardrails: an adoption number rising in isolation cannot tell you whether it's rising for a good reason, and this scenario is a concrete example of it rising for a bad one.
Building a metric tree
A metric tree makes the connection between a feature-level metric and a business outcome explicit and traceable, instead of leaving it as an assumed, undocumented link. The structure: at the top, the business outcome (revenue, retention, cost reduction); below it, the north star metric that's believed to drive that outcome; below that, the input metrics (quality, adoption, cost signals) that drive the north star; and explicitly noted at each level, the causal claim connecting it to the level above ("we believe regeneration rate below 20% predicts adoption because X," not just an arrow with no stated reasoning). Writing the causal claim down at each level does the same job the leading-indicator discipline from Phase 4 does for a PRD's success criteria: it forces you to be honest about whether you actually believe the connection or you're just hoping it's there, and it gives you something concrete to revisit and test as real data comes in.
A metric tree states the causal claim "we believe weekly active users predicts retention because engaged users churn less." Two months of data come in: WAU keeps rising, but retention is flat. A PM concludes the metric tree is fine and no action is needed because WAU (the north star) is still going up. What's the flaw in that reasoning?
The entire point of writing the causal claim down explicitly is so it can be tested against real data, not treated as a settled assumption. Two months of rising WAU with flat retention is exactly the kind of signal that should prompt questioning the claim itself — maybe WAU is rising among users who were never at churn risk to begin with, or maybe usage frequency isn't actually the retention driver for this feature. Concluding "the north star is up, so we're fine" ignores the very discipline this lesson introduces: a metric tree's causal claims are hypotheses to keep testing, not facts to stop examining once the number moves in the hoped-for direction.
Build
Build it
The code artifact lesson_08_02_metric_tree_builder.py implements a metric tree and guardrail-pairing tool:
MetricCategory enum (QUALITY, ADOPTION, COST) and MetricNode — a dataclass for one metric: name, category, current value, and the causal claim linking it to the node above it in the treeMetricTree — a north star metric plus a list of guardrail MetricNodes and input MetricNodes feeding it, plus the top-level business outcomecheck_guardrail_coverage() — flags whether the north star has at least one guardrail from a category other than its own (an adoption north star with no quality guardrail is flagged, mirroring the code-review-assistant checkpoint)render_metric_tree() — produces a markdown report showing the full tree with causal claims and any guardrail-coverage gaps- A worked example: the AI writing assistant from this lesson's opening, redesigned with separated quality/adoption/cost tracks and a guardrail specifically catching the "usage rises while quality drops" failure mode
"""
lesson_08_02_metric_tree_builder.py
A metric tree and guardrail-pairing tool for AI features.
Builds an explicit tree connecting a business outcome to a north star
metric to its quality/adoption/cost input metrics, with a stated causal
claim at every level, and checks whether the north star has guardrail
coverage from categories other than its own. This is the artifact that
replaces a single blended "AI health score" with a diagnosable structure
that shows which specific track broke when something goes wrong.
Usage:
python lesson_08_02_metric_tree_builder.py
No external dependencies, no network calls, no API keys required.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from enum import Enum
class MetricCategory(Enum):
QUALITY = "Quality"
ADOPTION = "Adoption"
COST = "Cost"
@dataclass
class MetricNode:
name: str
category: MetricCategory
current_value: str
causal_claim: str # why this metric is believed to affect the node above it
@dataclass
class MetricTree:
feature_name: str
business_outcome: str
north_star: MetricNode
guardrails: list[MetricNode] = field(default_factory=list)
input_metrics: list[MetricNode] = field(default_factory=list)
def check_guardrail_coverage(self) -> list[str]:
"""Flags if the north star lacks a guardrail from a category other
than its own -- the specific gap that let usage rise while quality
silently regressed in the lesson's opening example."""
warnings = []
guardrail_categories = {g.category for g in self.guardrails}
other_categories = {c for c in MetricCategory if c != self.north_star.category}
missing = other_categories - guardrail_categories
if missing:
missing_names = ", ".join(c.value for c in missing)
warnings.append(
f"North star '{self.north_star.name}' ({self.north_star.category.value}) has "
f"no guardrail metric from: {missing_names}. It could rise for reasons "
f"unrelated to genuine improvement in those dimensions without anyone noticing."
)
return warnings
def render_metric_tree(tree: MetricTree) -> str:
lines = [
f"# Metric Tree — {tree.feature_name}",
"",
f"**Business outcome:** {tree.business_outcome}",
"",
f"## North star: {tree.north_star.name} ({tree.north_star.category.value})",
f"- Current value: {tree.north_star.current_value}",
f"- Causal claim: {tree.north_star.causal_claim}",
"",
"## Guardrail metrics",
"",
]
for g in tree.guardrails:
lines.append(f"- **[{g.category.value}] {g.name}** ({g.current_value}) — {g.causal_claim}")
lines += ["", "## Input metrics feeding the north star", ""]
for m in tree.input_metrics:
lines.append(f"- **[{m.category.value}] {m.name}** ({m.current_value}) — {m.causal_claim}")
warnings = tree.check_guardrail_coverage()
lines += ["", "## Guardrail coverage check", ""]
if not warnings:
lines.append("North star has guardrail coverage from every other metric category.")
else:
for w in warnings:
lines.append(f"- ⚠️ {w}")
return "\n".join(lines)
# ── Worked example ───────────────────────────────────────────────────────────
# The AI writing assistant from the lesson's opening, redesigned with
# separated tracks and a guardrail catching the usage-up-quality-down gap.
def build_example_tree() -> MetricTree:
return MetricTree(
feature_name="AI Writing Assistant",
business_outcome="Increase subscription retention by reducing time-to-draft for "
"paid-tier content creators.",
north_star=MetricNode(
name="Weekly active users of the writing assistant",
category=MetricCategory.ADOPTION,
current_value="14,200 WAU (up from 9,800 two months ago)",
causal_claim="We believe sustained weekly usage predicts retention because "
"content creators who rely on the tool weekly are less likely to "
"churn -- but this claim is exactly what the guardrails below exist "
"to keep honest.",
),
guardrails=[
MetricNode(
name="Human-eval quality score on sampled generated drafts",
category=MetricCategory.QUALITY,
current_value="6.1/10 (down from 7.8/10 two months ago)",
causal_claim="If this drops while WAU rises, usage is climbing for a reason "
"other than the tool getting better -- exactly the gap the "
"opening example's blended score hid for weeks.",
),
MetricNode(
name="Cost per active user per month",
category=MetricCategory.COST,
current_value="$0.41/user/month",
causal_claim="Must stay well under the subscription tier's margin per user; "
"watched independently so a cost spike isn't masked by strong "
"adoption numbers.",
),
],
input_metrics=[
MetricNode(
name="Edit distance (draft to final)",
category=MetricCategory.ADOPTION,
current_value="38% of draft content changed on average (up from 22%)",
causal_claim="Rising edit distance alongside rising WAU is a diagnostic flag: "
"users may be using the tool more while trusting its raw output "
"less, needing more manual correction.",
),
MetricNode(
name="Regeneration rate",
category=MetricCategory.ADOPTION,
current_value="31% of sessions include at least one regenerate click "
"(up from 12%)",
causal_claim="A proxy for first-output quality; a rising regeneration rate "
"alongside the quality-score guardrail drop corroborates a real "
"quality regression, not just noisy sentiment data.",
),
],
)
if __name__ == "__main__":
tree = build_example_tree()
print(render_metric_tree(tree))
print()
warnings = tree.check_guardrail_coverage()
print(
"Takeaway: this tree has quality and cost guardrails paired with an adoption north "
"star, so the quality-score drop and rising edit distance/regeneration rate would "
"have surfaced as a clear, separate quality-track alarm well before a blended score "
"finished averaging the regression away -- exactly the diagnosis this lesson's "
"opening example needed and didn't have."
+ ("" if not warnings else f" (Note: {len(warnings)} coverage gap(s) remain.)")
)