Phase 4: Writing AI Product Requirements · 45 min · Markdown · Python
Specifying Non-Deterministic Behavior
You can't write 'the button turns blue' acceptance criteria for a system that doesn't produce the same output twice.
Hiring signal: Research grounding this course is explicit that AI PRDs 'define desired behavior rather than exact spec' — this is the single technical skill that most differentiates an AI PM from a traditional PM in a take-home exercise or interview loop, since it's the section most reviewers check first after the problem statement to see whether the candidate understands probabilistic systems at all.
What you will learn
- Write acceptance criteria as a behavior range or a rubric, not a single exact expected output
- Plan edge cases systematically for a generative feature instead of discovering them in QA
- Specify guardrails as explicit 'must never' statements distinct from ordinary acceptance criteria
- Produce a non-deterministic acceptance-criteria spec for a real AI feature that an engineer or QA reviewer could actually test against
The Problem
A PM writes an acceptance criterion for a new AI meeting-summarizer feature: "The system generates an accurate summary of the meeting transcript." Engineering builds it, QA opens the feature, runs it against a test transcript, reads the output, and marks the ticket "cannot verify — what does 'accurate' mean here, exactly, and what happens if I run it again and get a differently-worded but equally accurate summary?" The ticket bounces back to the PM. This isn't QA being difficult — the acceptance criterion, as written, is untestable, because it was written the way you'd write a deterministic spec ("clicking submit saves the record") for a system that will never produce byte-identical output twice and where "accurate" is a spectrum with real judgment calls at the edges, not a boolean.
Traditional acceptance criteria specify an exact expected output: given this input, the system produces exactly this result, and a test either passes or fails. Generative AI features can't be specified that way, because the same prompt against the same model can legitimately produce several different, equally acceptable outputs — different wording, different length, different emphasis — and also, less legitimately, a range of unacceptable outputs the spec needs to explicitly rule out. Writing testable acceptance criteria for a probabilistic system requires a different mental model: define a behavior range (what's acceptable, what's the boundary, what's definitely not), not a single expected output.
From Exact Output to Behavior Range
A workable non-deterministic acceptance criterion has three parts: a rubric describing what "good" looks like along the dimensions that matter (not a single example — a set of properties any acceptable output must have), a boundary describing what's borderline-acceptable and what tips into unacceptable, and an explicit guardrail list of things the system must never do regardless of how the rest of the output reads. For the meeting-summarizer example: the rubric might require every action item to name an owner and be traceable to a specific point in the transcript; the boundary might allow summaries to vary in length and phrasing but requires 100% of stated action items to be verifiable against the transcript (a plausible-sounding but unverifiable action item is a failure, not a stylistic variation); the guardrail might state the system must never summarize a transcript segment marked confidential-legal into the shareable output, regardless of how well-written the rest of the summary is.
A useful acceptance criterion for a generative feature answers "how would a human reviewer score this," not "what exact text comes out"
If you can't describe the specific dimensions a human reviewer would check (factual grounding, completeness against source material, tone, formatting, absence of specific forbidden content) and roughly where the pass/fail line falls on each, you haven't actually specified the behavior — you've specified a vibe. The test: hand your acceptance criteria to someone unfamiliar with the feature along with three sample outputs of varying quality, and see if they can independently sort them into pass/fail/borderline the same way you would. If they can't, the criteria need more specificity, usually in the form of a concrete rubric dimension you hadn't written down.
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers Edge Case Planning for Generative Features, Guardrails Are Not the Same as Acceptance Criteria, Build It — plus a hands-on lab, quiz, and project artifact.
Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.
Browse all courses · View pricing · DeVenture Academy