Phase 4: Writing AI Product Requirements · 140 min · Product Compass AI PRD template · Markdown · Python
The Problem
A candidate applying for an AI PM role submits a portfolio with five separate exercises: a problem statement, a success-metrics table, a prompt spec, a failure-mode list, and a cost model — each polished, each from a different hypothetical feature, none of them connected. The hiring manager reading it gets five demonstrations of isolated skill and zero demonstration of the thing the job actually requires: holding all of those decisions in your head at the same time, for one real feature, where the acceptance criteria constrain the prompt spec, the prompt spec has cost implications, the cost model forces a guardrail tradeoff, and the guardrails shape what the eval plan needs to check. A PRD is not five separate deliverables stapled together — it's one coherent argument where every section is load-bearing for the others, and a portfolio that can't show that coherence hasn't actually demonstrated the skill, no matter how good any individual section is in isolation.
This project exists to fix that. You're not writing a new hypothetical PRD — you're writing the complete, real PRD for the exact feature you scoped in Phase 2's opportunity assessment, using every skill this phase built: a problem statement that survives having "AI" deleted from it (Lesson 1), acceptance criteria specified as a behavior range instead of an exact output (Lesson 2), a versioned prompt spec (Lesson 3), explicit failure-state and escalation design (Lesson 4), and a 3-stage cost and latency model grounded in real token economics (Lesson 5). This lesson adds the two pieces that turn those five sections into a shippable document: eval criteria that hand off cleanly to Phase 5, and an explicit statement of what carries forward into Phase 7's responsible AI review. When you're done, this single document should read the way a real PRD reads to an engineering team that has to build from it — not as a demonstration that you learned five separate concepts, but as one decision, fully reasoned through.
Why eval criteria belong in the PRD, not just the eval plan
It's tempting to treat evaluation as entirely Phase 5's problem — write the PRD now, figure out how to test it later. That sequencing produces PRDs whose acceptance criteria sound rigorous but were never actually checked against "could someone build a test from this." Writing eval criteria as a PRD section, even in outline form, is a forcing function: if you can't state roughly how you'd verify a given acceptance criterion or guardrail is being met, you probably haven't specified it precisely enough yet, and it's far cheaper to discover that now than after Phase 5 tries to build a golden dataset against underspecified criteria. This PRD's eval-criteria section doesn't need the full rubric, scoring methodology, or golden dataset design that Phase 5 will build — it needs to name, for each major acceptance criterion and guardrail from Lesson 2, what category of eval would check it (a rubric-scored human eval, an automated faithfulness/grounding check, a red-team-style adversarial test, a simple pass/fail assertion) so that Phase 5 starts from a mapped list instead of a blank page.
A PRD that names its eval approach per-criterion is a fundamentally different artifact than one that doesn't
Compare "the system should not leak confidential information" (a guardrail with no stated verification approach) against "the system should not leak confidential information — verified via an automated red-team suite of 50+ adversarial prompts run on every prompt-spec version before deployment" (the same guardrail, with its verification method named). The second version is checkable, assignable, and arguable in a design review; the first is a hope. You don't need Phase 5's full rubric here — you need one sentence per major criterion naming the verification category, which is exactly the discipline a hiring manager is checking for when they read this section of your portfolio PRD.
Assembling the document
Pull together, in order: (1) the problem statement and success criteria from Lesson 1, revised if your thinking has sharpened since Phase 2; (2) the fine-tune-vs-prompt decision, also from Lesson 1; (3) non-deterministic acceptance criteria, guardrails, and edge cases from Lesson 2; (4) the prompt spec (not the full version history — just the current spec) from Lesson 3; (5) the failure-state design — abstention rules and escalation tiers — from Lesson 4; (6) the 3-stage cost and latency projection from Lesson 5; (7) the new eval-criteria mapping from this lesson; and (8) a short "handoff notes" section, also new, explicitly naming what Phase 5's eval plan needs to build against (the eval-criteria mapping) and what Phase 7's responsible AI review needs to examine (anything in your guardrails or edge cases touching bias, safety, privacy, or high-stakes decisions — flag these explicitly rather than leaving Phase 7 to rediscover them from scratch).
A learner's assembled PRD has a strong Lesson 2 acceptance-criteria section requiring "no fabricated citations, verified via retrieval grounding" and a strong Lesson 5 cost model — but the Lesson 3 prompt spec never mentions grounding or citation requirements at all, and the cost model's token estimate doesn't include any retrieval/citation-lookup overhead. What does this reveal about the PRD as a whole, and why does it matter more in this capstone than it would have in the standalone Lesson 2 or Lesson 5 exercises?
This is exactly the failure mode the capstone is built to catch. A guardrail or acceptance criterion that isn't reflected in the prompt spec is a requirement with no implementation path, and a cost model that doesn't account for a requirement's actual mechanism (here, a retrieval/grounding step) will understate real cost. In five disconnected exercises, this gap is invisible because nothing forces the sections to be checked against each other. In one assembled PRD, it's a specific, fixable inconsistency: the fix is adding a grounding/citation requirement to the prompt spec and adding retrieval-lookup tokens or a fixed per-request cost to the cost model, so all three sections describe the same actual system.
A learner writes handoff notes to Phase 7 that simply say "review this feature for responsible AI concerns" with no further detail, while their guardrails section lists a specific concern (routing decisions could disparately affect customers by account tier). What's wrong with the handoff note as written, and what should it say instead?
A handoff note that just says "review for responsible AI concerns" provides no more information than not writing a handoff section at all — it forces whoever picks up Phase 7's responsible AI review to re-derive risks the PM already spotted while writing the PRD, wasting the exact institutional knowledge a handoff is supposed to transfer. The specific version — naming the disparate-impact-by-account-tier concern directly, as this lesson's worked example does — lets Phase 7 start its review from a mapped, known risk instead of starting from zero. The same discipline that makes eval criteria useful (naming the specific check, not just "test this") applies to handoff notes: specificity is what makes them actually useful to the next phase.
Build
Build it
The code artifact lesson_04_06_full_prd_assembler.py implements a full-PRD assembler that reuses the dataclasses from Lessons 1-5:
EvalCriterion — maps one acceptance criterion or guardrail to a verification category (RUBRIC_HUMAN_EVAL, AUTOMATED_CHECK, RED_TEAM_ADVERSARIAL, SIMPLE_ASSERTION) and a one-line description of the checkHandoffNotes — explicit notes for what Phase 5's eval plan and Phase 7's responsible AI review need to pick up from this PRDFullPRD — bundles the AI PRD skeleton (Lesson 1), non-deterministic spec (Lesson 2), prompt spec (Lesson 3), failure design (Lesson 4), and cost/latency model (Lesson 5) plus the new EvalCriterion list and HandoffNotescheck_cross_section_consistency() — a lightweight consistency check: flags guardrails/acceptance criteria whose key terms don't appear anywhere in the prompt spec's required/forbidden behaviors, catching gaps like the checkpoint's grounding examplerender_full_prd() — produces the complete, portfolio-ready markdown PRD as one document- A worked example assembling a complete, internally consistent PRD for the support-ticket-triage feature used across Lessons 1-5, extended with eval criteria and handoff notes
"""
lesson_04_06_full_prd_assembler.py
The Phase 4 capstone: assembles a complete AI feature PRD from the
components built across Lessons 1-5 (problem/success criteria, fine-tune
decision, non-deterministic acceptance criteria, prompt spec, failure
design, cost/latency model), adds an eval-criteria mapping and explicit
Phase 5 / Phase 7 handoff notes, and includes a lightweight cross-section
consistency check that flags guardrails not reflected in the prompt spec.
This mirrors the real deliverable a hiring manager reads: one coherent
PRD, not five disconnected exercises.
References:
- Product Compass AI PRD Template: https://www.productcompass.pm/p/ai-prd-template
- Reforge, Write a PRD for a Generative AI Feature: https://www.reforge.com/guides/write-a-prd-for-a-generative-ai-feature
Usage:
python lesson_04_06_full_prd_assembler.py
No external dependencies, no network calls.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from enum import Enum
# ── Reused component types (condensed from Lessons 1-5) ─────────────────────
class MetricType(Enum):
LEADING = "leading"
LAGGING = "lagging"
@dataclass
class SuccessMetric:
name: str
metric_type: MetricType
target: str
measurement_window: str
@dataclass
class RubricDimension:
name: str
good_description: str
boundary_description: str
@dataclass
class Guardrail:
statement: str
rationale: str
@dataclass
class PromptSpecSummary:
goal: str
required_behaviors: list[str]
forbidden_behaviors: list[str]
@dataclass
class FailureScenarioSummary:
name: str
tier: str # "Auto-handle" | "Flag for review" | "Escalate immediately"
trigger: str
fallback: str
@dataclass
class CostStageSummary:
stage_name: str
cost_per_user_per_month: float
# ── New for this capstone ────────────────────────────────────────────────────
class EvalCategory(Enum):
RUBRIC_HUMAN_EVAL = "Rubric-scored human eval"
AUTOMATED_CHECK = "Automated check"
RED_TEAM_ADVERSARIAL = "Red-team / adversarial test"
SIMPLE_ASSERTION = "Simple pass/fail assertion"
@dataclass
class EvalCriterion:
source_requirement: str # the acceptance criterion or guardrail this maps to
category: EvalCategory
check_description: str
@dataclass
class HandoffNotes:
to_phase_5_eval_plan: list[str]
to_phase_7_responsible_ai_review: list[str]
@dataclass
class FullPRD:
title: str
problem_statement: str
success_metrics: list[SuccessMetric]
fine_tune_decision: str
rubric_dimensions: list[RubricDimension]
guardrails: list[Guardrail]
prompt_spec: PromptSpecSummary
failure_scenarios: list[FailureScenarioSummary]
cost_stages: list[CostStageSummary]
eval_criteria: list[EvalCriterion]
handoff: HandoffNotes
def check_cross_section_consistency(prd: FullPRD) -> list[str]:
"""Flags guardrails whose key terms don't appear anywhere in the prompt
spec's required or forbidden behaviors -- a proxy for 'this requirement
has no implementation path yet,' the gap described in this lesson's
checkpoint."""
warnings: list[str] = []
prompt_text = " ".join(
prd.prompt_spec.required_behaviors + prd.prompt_spec.forbidden_behaviors
).lower()
for g in prd.guardrails:
# crude but useful: check whether any distinctive word from the
# guardrail statement shows up in the prompt spec at all
words = [w.strip(".,()").lower() for w in g.statement.split() if len(w) > 5]
if words and not any(w in prompt_text for w in words):
warnings.append(
f"Guardrail '{g.statement}' has no obviously matching language in the "
f"prompt spec's required/forbidden behaviors -- verify the prompt "
f"actually implements this requirement, not just the PRD's guardrail list."
)
return warnings
def render_full_prd(prd: FullPRD) -> str:
lines = [
f"# {prd.title}",
"",
"## 1. Problem Statement",
"",
prd.problem_statement,
"",
"## 2. Success Criteria",
"",
"| Metric | Type | Target | Window |",
"|---|---|---|---|",
]
for m in prd.success_metrics:
lines.append(f"| {m.name} | {m.metric_type.value} | {m.target} | {m.measurement_window} |")
lines += ["", "## 3. Fine-Tune vs. Prompt Decision", "", prd.fine_tune_decision, ""]
lines += ["## 4. Non-Deterministic Acceptance Criteria", ""]
for d in prd.rubric_dimensions:
lines += [f"- **{d.name}** — Good: {d.good_description} | Boundary: {d.boundary_description}"]
lines += ["", "## 5. Guardrails", ""]
for g in prd.guardrails:
lines.append(f"- **Must never:** {g.statement} — *{g.rationale}*")
lines += ["", "## 6. Prompt Spec (current version)", "", f"**Goal:** {prd.prompt_spec.goal}", ""]
lines.append("Required behaviors:")
for r in prd.prompt_spec.required_behaviors:
lines.append(f"- {r}")
lines.append("Forbidden behaviors:")
for f in prd.prompt_spec.forbidden_behaviors:
lines.append(f"- {f}")
lines += ["", "## 7. Failure Design & Escalation", ""]
for fs in prd.failure_scenarios:
lines.append(f"- **[{fs.tier}]** {fs.name} — trigger: {fs.trigger}; fallback: {fs.fallback}")
lines += ["", "## 8. Cost & Latency (3-stage summary)", ""]
lines.append("| Stage | Cost/User/Month |")
lines.append("|---|---|")
for c in prd.cost_stages:
lines.append(f"| {c.stage_name} | ${c.cost_per_user_per_month:.2f} |")
lines += ["", "## 9. Eval Criteria Mapping", ""]
lines.append("| Requirement | Eval Category | Check |")
lines.append("|---|---|---|")
for ec in prd.eval_criteria:
lines.append(f"| {ec.source_requirement} | {ec.category.value} | {ec.check_description} |")
lines += ["", "## 10. Handoff Notes", "", "**To Phase 5 (Evaluation Plan):**"]
for note in prd.handoff.to_phase_5_eval_plan:
lines.append(f"- {note}")
lines += ["", "**To Phase 7 (Responsible AI Review):**"]
for note in prd.handoff.to_phase_7_responsible_ai_review:
lines.append(f"- {note}")
warnings = check_cross_section_consistency(prd)
lines += ["", "## Cross-Section Consistency Check", ""]
if not warnings:
lines.append("No obvious gaps found between guardrails and the prompt spec.")
else:
for w in warnings:
lines.append(f"- ⚠️ {w}")
return "\n".join(lines)
# ── Worked example ───────────────────────────────────────────────────────────
# The support-ticket triage feature used across Lessons 1-5, assembled into
# one complete, internally consistent PRD.
def build_example_full_prd() -> FullPRD:
return FullPRD(
title="Support Ticket Triage & Routing — Full AI Feature PRD",
problem_statement=(
"New support tickets sit in a shared inbox for a median of 4.2 hours before a "
"human agent reads, categorizes, and routes them. 18% are mis-routed on first "
"attempt, adding 6+ hours of delay. Paid-tier customers are contractually owed a "
"2-hour first-response SLA and miss it on roughly 1 in 4 tickets during the "
"mis-route path."
),
success_metrics=[
SuccessMetric("First-pass routing accuracy", MetricType.LEADING, ">=90%", "First 2 weeks"),
SuccessMetric("Paid-tier SLA compliance", MetricType.LAGGING, "95% (up from 76%)", "8 weeks"),
],
fine_tune_decision=(
"Prompted foundation model with routing taxonomy in the system prompt, plus "
"retrieval over the last 90 days of resolved tickets. Rejected fine-tuning: "
"taxonomy changes quarterly and a fine-tuned classifier can't keep pace without "
"constant retraining."
),
rubric_dimensions=[
RubricDimension(
"Routing correctness",
"Ticket is routed to the queue that would resolve it fastest given current "
"taxonomy.",
"A routing decision that's defensible but not optimal is acceptable; one "
"that ignores an explicit taxonomy rule is not.",
),
],
guardrails=[
Guardrail(
"Never auto-close or auto-resolve a ticket",
"Routing only — a human always handles the actual response.",
),
Guardrail(
"Never route a ticket containing legal, security, or safety keywords anywhere "
"except the human triage queue, regardless of confidence",
"These categories require human judgment regardless of model confidence.",
),
],
prompt_spec=PromptSpecSummary(
goal="Classify and route incoming support tickets to the correct team queue "
"using the current routing taxonomy and retrieved similar resolved tickets.",
required_behaviors=[
"Route only — never draft or send a resolution to the customer",
"Check ticket text for legal, security, or safety keywords before any other "
"routing logic runs, and route to human triage queue if found",
],
forbidden_behaviors=[
"Auto-closing or auto-resolving any ticket",
"Routing a legal, security, or safety keyword ticket to a non-human queue",
],
),
failure_scenarios=[
FailureScenarioSummary(
"Routing confidence below 60%",
"Flag for review",
"Model's top routing candidate has confidence < 60%",
"Ticket flagged 'needs manual review' rather than routed silently.",
),
FailureScenarioSummary(
"Legal/security/safety keyword detected",
"Escalate immediately",
"Ticket text matches legal/security/safety keyword list",
"Routed directly to human triage queue; no automated routing logic applies.",
),
],
cost_stages=[
CostStageSummary("Launch", 0.08),
CostStageSummary("Growth", 0.19),
CostStageSummary("Scale", 0.34),
],
eval_criteria=[
EvalCriterion(
"First-pass routing accuracy >= 90%",
EvalCategory.AUTOMATED_CHECK,
"Automated comparison against a golden set of 500 human-labeled tickets, "
"run on every prompt-spec version before deployment.",
),
EvalCriterion(
"Never route legal/security/safety ticket to non-human queue",
EvalCategory.RED_TEAM_ADVERSARIAL,
"Adversarial test suite of 100+ tickets containing keyword variants and "
"obfuscated phrasing, run on every prompt-spec version.",
),
EvalCriterion(
"Never auto-close or auto-resolve a ticket",
EvalCategory.SIMPLE_ASSERTION,
"Automated assertion checking the routing action taken is always a queue "
"assignment, never a resolution/close action.",
),
],
handoff=HandoffNotes(
to_phase_5_eval_plan=[
"Golden set of 500 human-labeled tickets needs construction (referenced in "
"the accuracy eval criterion above) — this PRD assumes it exists but does not "
"build it.",
"Red-team keyword/obfuscation test suite for legal/security/safety routing "
"needs a full rubric, not just the 100+ ticket count named here.",
],
to_phase_7_responsible_ai_review=[
"Routing decisions could disparately affect customers by account tier or "
"language — flag for bias/fairness review before broad rollout.",
"Legal/security/safety keyword list is a static list and may miss adversarial "
"phrasing not in the red-team suite — flag as an open risk for the "
"responsible AI review, not a solved problem.",
],
),
)
if __name__ == "__main__":
prd = build_example_full_prd()
output = render_full_prd(prd)
print(output)
with open("full_ai_feature_prd.md", "w") as f:
f.write(output)
warnings = check_cross_section_consistency(prd)
summary_lines = [
"# PRD Assembly Summary",
"",
f"Sections assembled: 10",
f"Cross-section consistency warnings: {len(warnings)}",
"",
"Takeaway: this PRD is deliberately built so every guardrail has matching language "
"in the prompt spec's required/forbidden behaviors -- the consistency check finds no "
"gaps. Removing the safety-keyword requirement from the prompt spec while leaving it "
"in the guardrails section (try it) reproduces the exact gap this lesson's checkpoint "
"describes.",
]
with open("prd_assembly_summary.md", "w") as f:
f.write("\n".join(summary_lines))
print()
print(f"\nWrote full_ai_feature_prd.md and prd_assembly_summary.md "
f"({len(warnings)} consistency warning(s)).")