Phase 6: Prompt Engineering & GenAI Feature Design · 45 min · System prompts · Few-shot prompting · Prompt versioning
The Problem
A PM specs a prompt for an AI feature that drafts customer-facing email replies: "You are a helpful customer support assistant. Write a reply to this customer email." It works on the three examples the PM tried in a playground. In production, it drifts: some replies are two sentences, some are four paragraphs; some open with "Dear Customer," others with "Hey!"; occasionally it invents a policy detail nobody confirmed. The PM's instinct is to ask engineering to "make it more consistent" — a request with no concrete action behind it. This is the same eval-first gap from Phase 5, but one level earlier: before you can evaluate a prompt's outputs, you have to know how to actually write and iterate one, or every improvement request stays vague forever.
Prompt engineering is not a one-shot writing exercise. It's an iterative loop: write a version, observe specific failures, apply a specific fix targeted at that failure, and re-test — the same before/after eval delta discipline from Phase 5, applied at the prompt-drafting stage instead of the launch-review stage.
Anatomy of a system prompt
A system prompt that produces consistent behavior separates four things a single vague sentence conflates:
- Role — who is the model acting as, and for whom. "You are a customer support assistant for [Product], writing on behalf of the support team" is more constraining, and more useful, than "you are helpful."
- Task — the specific action, stated precisely. Not "write a reply" but "draft a reply that addresses the customer's specific question, in 2-4 sentences, matching the tone of the original message."
- Constraints — the things the output must never do: never promise a refund or timeline not explicitly confirmed in the provided context, never invent a policy detail, never exceed a stated length.
- Output format — the structural shape of the response: plain text, a specific greeting/sign-off convention, a JSON schema if the output feeds a downstream system (Lesson 2 goes deeper on this).
Most prompt drift traces back to one of these four being left implicit. "Make it more consistent" is not actionable; "the task doesn't specify a length constraint, so length varies" is.
Few-shot examples: fixing a specific failure, not guessing
Few-shot prompting means including 2-5 example input/output pairs directly in the prompt, demonstrating the exact behavior you want. It is the single highest-leverage fix for a specific, observed failure mode, because it shows rather than describes — a model shown three examples of a 2-4 sentence reply with a consistent sign-off will match that pattern far more reliably than a model told "keep it brief" in the abstract.
The discipline that separates effective few-shot iteration from guessing: each example should target a specific failure you actually observed, not a hypothetical one. If the observed failure is inconsistent length, the few-shot examples should demonstrate the target length directly. If the observed failure is fabricated policy details, the few-shot examples should demonstrate correctly declining to answer when the provided context doesn't contain the answer. Adding examples that don't correspond to an observed failure mode bloats the prompt (with real cost and latency consequences — Phase 4's cost/latency budgeting applies here too) without necessarily fixing anything.
Diagnose the failure mode before writing the fix
The instinct when a prompt produces a bad output is to immediately rewrite the whole prompt. The higher-leverage move is to first name the specific failure: is it ambiguity (the instruction allows multiple valid interpretations), a missing constraint (nothing said not to do the bad thing), format drift (the output shape wanders across calls), or instruction conflict (two parts of the prompt implicitly contradict each other, e.g. 'be concise' and 'be thorough')? Each failure mode has a different fix -- ambiguity needs a more specific task statement, a missing constraint needs an explicit constraint added, format drift often needs a few-shot example or a structured output schema (Lesson 2), and instruction conflict needs the contradictory instructions resolved, not more instructions piled on top.
Common failure modes, named
Being able to name these on the spot, in an interview or a real prompt review, is the actual skill this lesson builds:
- Ambiguity — the instruction has more than one reasonable reading, and the model picked a different one than intended. Fix: make the task statement more specific, or add an example disambiguating the intended reading.
- Missing constraint — the prompt never said not to do the thing it's now doing (inventing details, using an inappropriate tone, exceeding a length). Fix: add the constraint explicitly; don't assume it's implied.
- Format drift — output structure is inconsistent across calls (sometimes bulleted, sometimes prose; sometimes with a greeting, sometimes without). Fix: few-shot examples demonstrating the exact format, or a structured output schema for anything feeding a downstream system.
- Instruction conflict — two parts of the prompt pull in different directions (concise vs. thorough, formal vs. friendly) and the model resolves the tension inconsistently across calls. Fix: resolve the conflict explicitly in the prompt (state which wins, or give a rule for when each applies) rather than leaving both instructions in tension.
A support-reply prompt says: "Write a helpful, thorough reply that fully addresses the customer's question, but keep it brief." Output length varies wildly between two-sentence and eight-sentence replies across similar inputs. What is the most likely root cause, and the correct fix?
'Thorough' and 'brief' are in direct tension with no guidance on how to balance them, so the model has to resolve that ambiguity fresh on every call, producing inconsistent length. This is a textbook instruction-conflict failure, not a model malfunction or a vocabulary problem. The fix is to replace the vague, conflicting adjectives with something concrete and unambiguous -- e.g., '2-4 sentences, addressing only the customer's specific question' -- which also happens to be a case where a few-shot example demonstrating that exact length would reinforce the fix effectively.
Prompt versioning as a product artifact
A prompt is not a one-time creative writing exercise that gets frozen at launch — it's a product artifact that changes over time, the same way code does, and it needs the same discipline: every version gets a label, every change has a reason, and every change should ship with an eval delta (Phase 5, Lesson 1's "every prompt change ships with an eval delta" practice, now applied at the source). A prompt change log entry that just says "improved the prompt" is exactly as useless as a commit message that says "fixed stuff" — it should say what specific failure mode prompted the change and what specific technique addressed it: "v6: added few-shot examples targeting length drift (observed failure: 40% of replies exceeded 4 sentences); eval delta: length-compliance 61% -> 94%, no faithfulness regression."
A team's prompt change log reads: "v4: made the prompt better. v5: fixed some issues. v6: further improvements." A new PM joins and needs to understand why the prompt is currently structured the way it is. What is the primary problem with this change log, and what should it have included?
Vague change log entries destroy the same institutional memory that a golden dataset (Phase 5, Lesson 1) is built to preserve -- if nobody recorded which failure mode 'v5: fixed some issues' addressed and what the eval delta was, a new team member has no way to know whether a given prompt section is load-bearing (fixing a real observed problem) or vestigial, and no way to safely predict what might break if they change it. A useful entry states the observed failure, the specific technique applied, and the resulting eval delta -- exactly the discipline code commit messages and Phase 5's eval-delta practice both apply, now extended to the prompt itself.