Phase 4: Writing AI Product Requirements · 40 min · Markdown · Python

Prompt Specs & Prompt-as-Interface

The prompt is the interface. Treat a silent prompt edit like you'd treat a silent API contract change.

What you will learn

Explain why the prompt should be treated as a PM-owned product artifact, not an engineering implementation detail
Write a prompt spec that separates PM-owned intent (goals, constraints, tone, examples) from engineering-owned implementation (exact tokens, system message structure)
Design a prompt versioning scheme that ties each version to a specific eval result
Produce a versioned prompt spec and changelog for a real AI feature

The Problem

A PM tells an engineer in a Slack message: "Can you make the AI customer-support responses sound a bit more empathetic? Some users are saying it feels robotic." The engineer tweaks the system prompt, ships it the same afternoon, and moves to the next ticket. Two weeks later, refund approval rates have quietly dropped 15%, and nobody connects it to the prompt change — because there's no record that a change happened, what the exact wording change was, or what the prompt looked like before. It takes a data scientist three days of digging through deployment logs to even confirm the prompt was the variable that moved, let alone which specific line caused the effect. A one-line Slack message caused a measurable business outcome and left no auditable trail.

This is the failure mode of treating the prompt as an implementation detail instead of what it actually is: the interface between the product's intent and the model's behavior. Every other interface a PM cares about — an API contract, a UI copy string, a form field's validation rule — gets a spec, a review, and a version history, because everyone understands changing it changes behavior in ways that need to be tracked and testable. A system prompt changes behavior at least as much as any of those, often more, and yet it's routinely treated as a throwaway implementation detail that engineering can tweak informally. Treating the prompt as a first-class, PM-owned product artifact — with the same rigor as any other interface — is one of the concrete ways an AI PM's job differs from a traditional PM's, and it's a skill gap that shows up immediately in how a candidate talks about iterating on a generative feature.

What the PM owns vs. what engineering owns

The PM doesn't write the literal token sequence the model receives — that's an engineering implementation concern, involving exact system-message formatting, few-shot example selection, and provider-specific syntax. What the PM does own is the prompt spec: the goal the prompt is meant to achieve, the constraints and tone it must respect, the specific behaviors it must produce or avoid (tying directly back to the acceptance criteria and guardrails from Lesson 2), and a small set of worked examples showing desired input/output pairs. Engineering translates that spec into the actual prompt implementation, and — critically — any translation choice that changes behavior in a way the spec didn't anticipate should trigger a conversation back to the PM, not a silent judgment call. This is the same division of labor as a traditional PRD and its engineering implementation: the PM specifies desired behavior, engineering decides how to achieve it technically, and anything that reinterprets the spec's intent gets flagged rather than assumed.

A prompt spec answers "what would I tell a new, very literal-minded employee on their first day," not "what exact words go in the system prompt"
If your prompt spec reads like a finished system prompt (full of exact phrasing, formatting instructions, and prompt-engineering idioms), you've written the implementation, not the spec — and you've taken over engineering's job while leaving out the actual PM job, which is being unambiguous about the goal, the tone, the hard constraints, and 3-5 worked examples of what right and wrong look like. A good test: could two different engineers, working independently from your spec, arrive at differently-worded but behaviorally equivalent prompts? If yes, you wrote a spec. If they'd have to guess your intent because you buried it in specific wording choices, you wrote an implementation and called it a spec.

Prompt versioning as a product artifact

Every prompt change is a product change and deserves the same discipline a versioned API or a feature flag gets: a version identifier, a changelog entry describing what changed and why, and — this is the part most teams skip — an eval delta showing the measured effect of the change before it ships broadly. "We think this version sounds more empathetic" is a hypothesis; "eval score for empathy-rubric dimension went from 6.2 to 7.8 on the golden set, with no regression on the factual-accuracy dimension" is evidence. A prompt version history that pairs every change with its eval delta turns "why did refund approvals drop" from a three-day forensic investigation into a five-minute lookup: check the changelog, find the version that shipped around that date, check its eval delta, see whether refund-adjacent behavior was tested at all — and if it wasn't, that's the actual gap the incident reveals.

A team's prompt changelog shows: "v1.4 (March 3): improved tone per user feedback. v1.5 (March 15): fixed formatting issue." A support ticket surge in late March turns out to trace back to a v1.5 prompt change that inadvertently made the model more willing to approve high-value refund requests. What's the specific gap in this changelog, and what should every entry have included?

The changelog entries describe what the change was intended to do, but say nothing about what was actually measured before shipping -- specifically, whether the eval suite checked dimensions beyond the one being targeted (formatting) for regressions. A formatting-focused prompt tweak plausibly touched wording that also affected refund-approval willingness, and without an eval delta covering that dimension, nobody would have caught it before it reached production. Every prompt version entry should include not just what changed and why, but what was tested and the measured before/after delta on the full relevant rubric -- not just the one dimension the change targeted.

Handoff without losing the thread

The most common breakdown in prompt-as-interface practice isn't a bad spec — it's an untracked handoff. The PM writes a good spec, engineering implements a reasonable prompt, and then three iterations later nobody can say which version is live, what the original spec intended versus what's now actually in the prompt, or whether the two have quietly diverged. The fix isn't more process for its own sake; it's making the prompt spec and the live prompt implementation live in the same version-controlled place, so a PM reviewing "what does this feature currently do" can look at one artifact instead of reconstructing it from a Slack history and a deploy log.

A PM writes a prompt spec that says: "Draft empathetic, accurate first-response replies... never approve, promise, or imply approval of a refund." An engineer implements it, and the resulting live system prompt happens to include the literal sentence "you may offer store credit up to $25 as a goodwill gesture" as a practical way to sound less robotic when declining requests. Is this a good implementation of the spec?

The spec's forbidden-behaviors section is explicit: never approve, promise, or imply approval of a refund. Granting the prompt authority to offer up to $25 in store credit is compensation-adjacent and directly touches a hard constraint the PM wrote down — this is exactly the kind of translation choice that changes behavior in a way the spec didn't anticipate, and it should trigger a conversation back to the PM rather than being decided unilaterally during implementation. Silently expanding scope here is the same category of risk as the untracked Slack-message prompt edit this lesson opens with, just introduced through the engineering side of the handoff instead of the PM side.

Build

Build it

The code artifact lesson_04_03_prompt_spec_generator.py implements a prompt spec and version-history tool:

PromptSpec — a dataclass for the PM-owned spec: goal, constraints, tone, required behaviors, forbidden behaviors, and worked examples (input/expected-output pairs)
PromptVersion — one version in the history: version id, date, change summary, the spec it implements, and an EvalDelta
EvalDelta — before/after scores per rubric dimension, so a regression on an untargeted dimension is visible, not hidden
render_prompt_spec() — renders a single spec as a markdown document engineering can implement against
render_version_history() — renders the full changelog with eval deltas, flagging any version whose changelog entry doesn't mention testing a dimension that later regressed
A worked example: the customer-support empathy prompt from this lesson, with a version history showing the v1.5 refund-approval regression this lesson describes

"""
lesson_04_03_prompt_spec_generator.py

A prompt spec and version-history tool for AI PRDs.

Separates the PM-owned prompt spec (goal, constraints, tone, required and
forbidden behaviors, worked examples) from a versioned implementation
history, where each version is paired with an EvalDelta showing what was
actually measured before shipping -- not just a one-line "improved tone"
changelog note. This is the artifact that turns "why did behavior change"
from a multi-day forensic investigation into a changelog lookup.

References:
  - Anthropic, Prompt engineering overview: https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/overview
  - OpenAI, Prompt engineering guide: https://platform.openai.com/docs/guides/prompt-engineering

Usage:
    python lesson_04_03_prompt_spec_generator.py

No external dependencies, no network calls.
"""

from __future__ import annotations

from dataclasses import dataclass, field

@dataclass
class WorkedExample:
    input_text: str
    desired_output: str
    notes: str = ""

@dataclass
class PromptSpec:
    feature_name: str
    goal: str
    constraints: list[str]
    tone: str
    required_behaviors: list[str]
    forbidden_behaviors: list[str]
    worked_examples: list[WorkedExample]

@dataclass
class EvalDelta:
    dimension_scores_before: dict[str, float]
    dimension_scores_after: dict[str, float]

    def regressions(self, threshold: float = -0.3) -> list[str]:
        """Dimensions that dropped by more than `threshold` (default: any drop > 0.3)."""
        out = []
        for dim, before in self.dimension_scores_before.items():
            after = self.dimension_scores_after.get(dim)
            if after is not None and (after - before) < threshold:
                out.append(dim)
        return out

@dataclass
class PromptVersion:
    version_id: str
    date: str
    change_summary: str
    dimensions_tested: list[str]  # which rubric dimensions this change was actually evaled against
    eval_delta: EvalDelta

def render_prompt_spec(spec: PromptSpec) -> str:
    lines = [
        f"# Prompt Spec — {spec.feature_name}",
        "",
        f"**Goal:** {spec.goal}",
        f"**Tone:** {spec.tone}",
        "",
        "## Constraints",
    ]
    for c in spec.constraints:
        lines.append(f"- {c}")
    lines += ["", "## Required behaviors"]
    for r in spec.required_behaviors:
        lines.append(f"- {r}")
    lines += ["", "## Forbidden behaviors"]
    for f in spec.forbidden_behaviors:
        lines.append(f"- {f}")
    lines += ["", "## Worked examples", ""]
    for i, ex in enumerate(spec.worked_examples, 1):
        lines += [
            f"**Example {i}**",
            f"- Input: {ex.input_text}",
            f"- Desired output: {ex.desired_output}",
        ]
        if ex.notes:
            lines.append(f"- Notes: {ex.notes}")
        lines.append("")
    return "\n".join(lines)

def render_version_history(feature_name: str, versions: list[PromptVersion]) -> str:
    lines = [f"# Prompt Version History — {feature_name}", ""]
    for v in versions:
        lines += [
            f"## {v.version_id} — {v.date}",
            "",
            f"**Change:** {v.change_summary}",
            f"**Dimensions tested before shipping:** {', '.join(v.dimensions_tested) or 'none recorded'}",
            "",
            "| Dimension | Before | After | Delta |",
            "|---|---|---|---|",
        ]
        all_dims = sorted(set(v.eval_delta.dimension_scores_before) | set(v.eval_delta.dimension_scores_after))
        for dim in all_dims:
            before = v.eval_delta.dimension_scores_before.get(dim)
            after = v.eval_delta.dimension_scores_after.get(dim)
            delta = (after - before) if (before is not None and after is not None) else None
            lines.append(
                f"| {dim} | {before if before is not None else '—'} | "
                f"{after if after is not None else '—'} | "
                f"{f'{delta:+.1f}' if delta is not None else '—'} |"
            )

        regressions = v.eval_delta.regressions()
        untested_regressions = [r for r in regressions if r not in v.dimensions_tested]
        if untested_regressions:
            lines.append("")
            lines.append(
                f"**WARNING:** regression detected on dimension(s) not listed as tested "
                f"before shipping: {', '.join(untested_regressions)}. This is exactly the "
                f"blind-spot pattern that causes untraceable behavior changes."
            )
        lines.append("")
    return "\n".join(lines)

# ── Worked example ───────────────────────────────────────────────────────────
# The customer-support empathy prompt from the lesson body: v1.5's formatting
# fix inadvertently regressed refund-approval-caution behavior, and it wasn't
# a tested dimension for that version.

def build_example() -> tuple[PromptSpec, list[PromptVersion]]:
    spec = PromptSpec(
        feature_name="Customer Support Response Generator",
        goal="Draft empathetic, accurate first-response replies to customer support tickets "
             "for a human agent to review before sending.",
        constraints=[
            "Never approve, promise, or imply approval of a refund — drafting only, a human "
            "agent makes all refund decisions",
            "Match the customer's stated tone (frustrated, confused, neutral) without "
            "escalating or minimizing it",
        ],
        tone="Warm, direct, and specific to the customer's actual issue — never generic "
             "corporate boilerplate.",
        required_behaviors=[
            "Acknowledge the specific problem described before offering next steps",
            "Use the customer's name and reference specific order/account details when available",
        ],
        forbidden_behaviors=[
            "Stating or implying a refund, credit, or compensation amount has been approved",
            "Using dismissive phrases ('per our policy', 'unfortunately we cannot')  without "
            "an alternative offered",
        ],
        worked_examples=[
            WorkedExample(
                input_text="Customer: 'This is the third time my order has arrived broken. "
                            "I'm done with this company.'",
                desired_output="Acknowledges the repeated failure specifically (not generically), "
                                "expresses genuine apology, offers to escalate to a human agent "
                                "for resolution -- does not promise a refund or replacement.",
                notes="Tests: empathy tone + refund-approval guardrail together.",
            ),
        ],
    )

    versions = [
        PromptVersion(
            version_id="v1.4",
            date="2026-03-03",
            change_summary="Improved tone per user feedback -- added instruction to lead with "
                            "acknowledgment of the customer's specific issue before next steps.",
            dimensions_tested=["empathy", "specificity"],
            eval_delta=EvalDelta(
                dimension_scores_before={"empathy": 6.2, "specificity": 5.8, "refund_caution": 9.1},
                dimension_scores_after={"empathy": 7.8, "specificity": 7.4, "refund_caution": 9.0},
            ),
        ),
        PromptVersion(
            version_id="v1.5",
            date="2026-03-15",
            change_summary="Fixed formatting issue -- responses were sometimes missing a "
                            "closing line; added a closing-line instruction to the prompt.",
            dimensions_tested=["formatting"],
            eval_delta=EvalDelta(
                dimension_scores_before={"formatting": 6.0, "refund_caution": 9.0},
                dimension_scores_after={"formatting": 8.5, "refund_caution": 6.1},
            ),
        ),
    ]
    return spec, versions

if __name__ == "__main__":
    spec, versions = build_example()
    print(render_prompt_spec(spec))
    print()
    print(render_version_history(spec.feature_name, versions))
    print()
    print(
        "Takeaway: v1.5's changelog entry only mentions formatting, and only 'formatting' was "
        "listed as tested -- but refund_caution silently dropped from 9.0 to 6.1. The warning "
        "block makes that regression visible immediately instead of requiring a three-day "
        "forensic investigation after a support ticket surge, which is exactly the incident "
        "this lesson opens with."
    )

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Check Yourself — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.

Key terms in this lesson

Prompt Engineering