Phase 4: Writing AI Product Requirements · 45 min · Markdown · Python

Cost & Latency Budgeting in the PRD

A feature that's profitable at 1,000 users and bankrupts the unit economics at 100,000 wasn't actually scoped — it was guessed at.

What you will learn

Explain token economics: input/output pricing and how context window size drives cost
Build a unit cost model per request, per user, and per feature
Set a defensible latency SLA for a generative feature and explain the user-experience tradeoffs behind it
Produce a 3-stage (launch/growth/scale) cost projection table for a real AI feature

The Problem

A PM ships an AI-powered document-analysis feature bundled into a $29/month subscription tier at no usage limit, reasoning that "AI is cheap now" and the feature will drive upgrades. The feature retrieves relevant context from a user's uploaded documents and sends it, along with the user's question, to a foundation-model API on every query. At launch, with 2,000 beta users making a few queries a week, the AI spend is a rounding error against revenue. Eight months later, at 80,000 subscribers with power users running dozens of queries a day against increasingly large document sets, the AI API bill is quietly exceeding the entire tier's subscription revenue — the feature that was supposed to drive upgrades is now actively losing money on every heavy user, and nobody sized this at launch because nobody built a cost model past the beta cohort's usage pattern.

This is a completely avoidable failure. Unlike model quality or user adoption, the cost of a generative feature at any given usage level is a knowable, computable number before launch — token pricing is public, and query volume can be estimated with the same rigor a traditional PM would apply to any other capacity-planning problem. The fact that many AI PRDs skip this isn't because it's hard to model; it's because "AI is cheap" is treated as a vibe instead of the specific, checkable claim it actually is.

Token economics: the building blocks

Foundation-model APIs price by token — a token is roughly 3/4 of a word in English — with separate rates for input tokens and output tokens, and output tokens are typically priced several times higher than input tokens because generation is more computationally expensive than reading context. This asymmetry matters directly for product design: a feature that stuffs a large document into the context window (many cheap input tokens) but returns a short answer (few expensive output tokens) has a very different cost profile than a feature that takes a short prompt but generates a long report (few cheap input tokens, many expensive output tokens). A PM sizing cost needs to estimate both halves separately, not a single blended "cost per query" number pulled from a rough guess — the ratio of input to output tokens for a specific feature is usually the single biggest lever on unit cost, bigger than which model tier gets chosen.

Context window size is a recurring cost, not a one-time design choice
Every additional document, every turn of conversation history, every piece of retrieved context added to a prompt gets re-sent and re-billed on every single query in that session, not once. A RAG feature that retrieves 5,000 tokens of context per query costs roughly 5,000 input tokens every query, and if a design decision doubles the retrieval window "to improve quality," that's not a one-time engineering tradeoff — it's a permanent doubling of the input-token cost of every future query, at every future usage level. Cost modeling has to happen at the design-decision level (how much context does this feature send, every time), not just at the final feature level.

Building the unit cost model

A workable unit cost model has three layers: cost per request (input tokens × input price + output tokens × output price, plus any retrieval/embedding costs for RAG features), cost per user (cost per request × average requests per user per period), and cost per feature (cost per user × number of users, at a given usage tier). The model needs realistic, sourced token estimates for a specific feature's actual prompt and response pattern — not a generic "AI query" assumption — because the input/output ratio and volume vary enormously between, say, a one-line classification task and a long-form document generation task.

A PM builds a cost model for an AI writing assistant using an estimated 200 input tokens and 200 output tokens per request, based on a rough guess. An engineer later profiles actual production traffic and finds the real average is 3,500 input tokens (most of the user's draft document gets included as context) and 400 output tokens. Output tokens are priced 5x higher than input tokens. What's the most likely consequence of the original estimate, and what should the PM have done differently?

Even though output tokens are priced higher per-token, a 17x volume miss on input tokens (200 estimated vs. 3,500 actual) is large enough to dominate the total cost error unless the per-token price gap is extreme. The deeper lesson is about estimation method, not just this specific number: a 'rough guess' of round numbers like 200/200 is exactly the kind of unsourced estimate that produces a cost model wildly disconnected from reality. The PM should have either prototyped the actual feature with realistic inputs (a real draft document, not a placeholder) or partnered with engineering early to get a profiled estimate before the model went into the PRD, the same way a traditional PM would validate any other capacity-planning assumption with real data rather than a guess.

Latency SLAs for generative features

Generative features have a fundamentally different latency profile than a traditional API call: output is produced token-by-token, response length varies, and quality/cost/speed trade against each other in ways a PM needs to make an explicit call on, not leave to whatever the default model configuration happens to produce. A latency SLA for a generative feature should specify time-to-first-token (how long before the user sees anything, critical for perceived responsiveness) separately from total generation time (which scales with output length and matters more for batch/report-style features than for chat-style ones), and should be tied to a specific user-experience justification — "under 2 seconds to first token" is a defensible SLA for a chat interface where perceived responsiveness drives engagement; a batch document-analysis feature that runs in the background can reasonably tolerate 30+ seconds of total generation time if the UX sets that expectation clearly (a progress indicator, an async "we'll notify you" pattern) rather than leaving a user staring at a blank, unexplained loading state.

A PM sets a single latency SLA of "under 2 seconds, end to end" for two different features sharing one PRD: a chat-style Q&A assistant, and a batch feature that generates a 10-page report from a large document set. What's the problem with applying the same SLA to both?

Total generation time scales with output length — a 10-page report has an inherent floor on how fast it can be generated that a short chat answer doesn't share, and forcing a batch-style feature under a chat-appropriate SLA either forces the team to artificially truncate report quality/length to hit the number, or the SLA becomes unachievable and gets quietly ignored, which defeats the purpose of setting it at all. The fix is exactly what the lesson recommends: split the SLA into time-to-first-token (which can reasonably be fast for both) and total generation time (which should be set per-feature based on its actual output length and interaction pattern, with UX design like a progress indicator absorbing the difference rather than an arbitrary shared number).

The 3-stage cost projection table

Every serious AI PRD's cost section projects three stages, not just launch-day cost: launch (initial rollout, small or beta user base — mostly a sanity check that the feature isn't absurdly expensive even at low volume), growth (a realistic 6-12 month usage projection, where the real economics start to matter and where features with a bad input/output ratio or an unbounded usage pattern start to show strain), and scale (a bull-case usage projection, explicitly stress-testing whether the unit economics hold or break as volume grows — this is the stage that catches the failure mode from this lesson's opening example before it happens in production instead of after). Each stage needs its own query-volume assumption, tied to a specific and stated reasoning (not just "10x more users"), because query volume per user often changes non-linearly with total user count — power users found at scale often query far more heavily than early beta testers, which is exactly the miss in the opening example.

Build

Build it

The code artifact lesson_04_05_cost_latency_model.py implements a 3-stage AI feature cost and latency model:

TokenPricing — input/output price per 1,000 tokens for a given model
RequestProfile — average input tokens, average output tokens, and any fixed per-request cost (e.g., embedding/retrieval) for a specific feature's actual usage pattern
UsageStage — one of launch/growth/scale, with a users count, average requests per user per month, and a stated assumption for why
compute_stage_cost() — computes monthly cost for a stage from a RequestProfile and TokenPricing
LatencySLA — time-to-first-token and total-generation-time targets with a UX justification
render_cost_latency_report() — produces a markdown 3-stage cost projection table plus the latency SLA section
A worked example: the document-analysis feature from this lesson's opening, modeled correctly across launch/growth/scale, showing the margin compression the original team missed

"""
lesson_04_05_cost_latency_model.py

A 3-stage AI feature cost and latency projection tool.

Builds a token-economics-grounded unit cost model (cost per request, per
user, per feature) across launch/growth/scale usage stages, and pairs it
with an explicit latency SLA (time-to-first-token and total generation
time) tied to a stated UX justification. This is the artifact that
catches a margin-compression problem in a PRD review instead of eight
months into production.

Reference: Reforge, Write a PRD for a Generative AI Feature —
https://www.reforge.com/guides/write-a-prd-for-a-generative-ai-feature

Usage:
    python lesson_04_05_cost_latency_model.py

No external dependencies, no network calls.
"""

from __future__ import annotations

from dataclasses import dataclass, field

@dataclass
class TokenPricing:
    model_name: str
    input_price_per_1k: float   # USD per 1,000 input tokens
    output_price_per_1k: float  # USD per 1,000 output tokens

@dataclass
class RequestProfile:
    feature_name: str
    avg_input_tokens: float
    avg_output_tokens: float
    fixed_cost_per_request_usd: float = 0.0  # e.g. embedding/retrieval cost, not token-priced

    def cost_per_request(self, pricing: TokenPricing) -> float:
        input_cost = (self.avg_input_tokens / 1000) * pricing.input_price_per_1k
        output_cost = (self.avg_output_tokens / 1000) * pricing.output_price_per_1k
        return input_cost + output_cost + self.fixed_cost_per_request_usd

@dataclass
class UsageStage:
    stage_name: str  # "Launch" | "Growth" | "Scale"
    users: int
    avg_requests_per_user_per_month: float
    assumption: str  # stated reasoning for this stage's numbers

@dataclass
class StageCostResult:
    stage: UsageStage
    cost_per_request: float
    total_monthly_requests: float
    total_monthly_cost: float
    cost_per_user_per_month: float

def compute_stage_cost(stage: UsageStage, profile: RequestProfile, pricing: TokenPricing) -> StageCostResult:
    cpr = profile.cost_per_request(pricing)
    total_requests = stage.users * stage.avg_requests_per_user_per_month
    total_cost = total_requests * cpr
    cost_per_user = total_cost / stage.users if stage.users else 0.0
    return StageCostResult(stage, cpr, total_requests, total_cost, cost_per_user)

@dataclass
class LatencySLA:
    time_to_first_token_seconds: float
    total_generation_time_seconds: float
    ux_justification: str

def render_cost_latency_report(
    profile: RequestProfile,
    pricing: TokenPricing,
    stages: list[UsageStage],
    sla: LatencySLA,
    subscription_price_usd: float | None = None,
) -> str:
    lines = [
        f"# Cost & Latency Projection — {profile.feature_name}",
        "",
        f"**Model:** {pricing.model_name} (${pricing.input_price_per_1k}/1K input tok, "
        f"${pricing.output_price_per_1k}/1K output tok)",
        f"**Request profile:** ~{profile.avg_input_tokens:.0f} input tok, "
        f"~{profile.avg_output_tokens:.0f} output tok, "
        f"${profile.fixed_cost_per_request_usd:.4f} fixed cost/request",
        "",
        "## 3-stage cost projection",
        "",
        "| Stage | Users | Req/User/Mo | Total Req/Mo | Cost/Request | Total Cost/Mo | Cost/User/Mo | Assumption |",
        "|---|---|---|---|---|---|---|---|",
    ]
    for stage in stages:
        result = compute_stage_cost(stage, profile, pricing)
        margin_note = ""
        if subscription_price_usd is not None:
            margin_note = (
                " ⚠️ EXCEEDS SUBSCRIPTION PRICE"
                if result.cost_per_user_per_month > subscription_price_usd
                else ""
            )
        lines.append(
            f"| {stage.stage_name} | {stage.users:,} | {stage.avg_requests_per_user_per_month:.1f} | "
            f"{result.total_monthly_requests:,.0f} | ${result.cost_per_request:.4f} | "
            f"${result.total_monthly_cost:,.2f} | ${result.cost_per_user_per_month:.2f}{margin_note} | "
            f"{stage.assumption} |"
        )

    lines += [
        "",
        "## Latency SLA",
        "",
        f"- **Time to first token:** {sla.time_to_first_token_seconds:.1f}s",
        f"- **Total generation time:** {sla.total_generation_time_seconds:.1f}s",
        f"- **UX justification:** {sla.ux_justification}",
    ]

    if subscription_price_usd is not None:
        lines += ["", f"**Subscription price for comparison:** ${subscription_price_usd:.2f}/user/month"]

    return "\n".join(lines)

# ── Worked example ───────────────────────────────────────────────────────────
# The document-analysis feature from the lesson's opening: modeled correctly
# across launch/growth/scale, showing the margin compression the original
# team missed by only sizing the beta-stage usage pattern.

def build_example_report() -> tuple[RequestProfile, TokenPricing, list[UsageStage], LatencySLA, float]:
    pricing = TokenPricing(model_name="Frontier model, standard tier", input_price_per_1k=0.003, output_price_per_1k=0.015)
    profile = RequestProfile(
        feature_name="Document Analysis Assistant",
        avg_input_tokens=4000,   # retrieved document context + question
        avg_output_tokens=350,   # a focused answer, not a full report
        fixed_cost_per_request_usd=0.0008,  # embedding cost for the query
    )
    stages = [
        UsageStage(
            stage_name="Launch",
            users=2000,
            avg_requests_per_user_per_month=12,  # beta cohort: light usage
            assumption="Beta cohort, light usage: ~3 queries/week average.",
        ),
        UsageStage(
            stage_name="Growth",
            users=25000,
            avg_requests_per_user_per_month=40,
            assumption="Broader rollout; usage roughly triples per-user as habitual users "
                       "form daily-use habits, not just linear user growth.",
        ),
        UsageStage(
            stage_name="Scale",
            users=80000,
            avg_requests_per_user_per_month=90,
            assumption="Power-user segment (roughly 20% of base) drives heavy daily usage "
                       "against growing document sets; average requests/user/month more than "
                       "doubles again from Growth as document libraries grow and querying "
                       "becomes habitual, not just more users.",
        ),
    ]
    sla = LatencySLA(
        time_to_first_token_seconds=2.5,
        total_generation_time_seconds=8.0,
        ux_justification="Chat-style interface where users watch the response stream in; "
                          "2.5s to first token keeps perceived responsiveness high, and total "
                          "generation under 8s avoids the interface feeling stuck for a "
                          "focused-answer response length.",
    )
    subscription_price = 29.00
    return profile, pricing, stages, sla, subscription_price

if __name__ == "__main__":
    profile, pricing, stages, sla, sub_price = build_example_report()
    print(render_cost_latency_report(profile, pricing, stages, sla, sub_price))
    print()
    print(
        "Takeaway: at Launch, cost/user/month is a rounding error against the $29 subscription "
        "price. By Scale, heavier per-user usage (not just more users) pushes cost/user/month "
        "up sharply -- exactly the margin compression the original team in this lesson's "
        "opening example didn't catch, because they only ever sized the beta-stage usage "
        "pattern and never projected what happens when usage per user grows along with the "
        "user base."
    )

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Check Yourself — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.