The Problem
Two engineers call the same model with the same question. One gets a hallucinated answer in an unusable format. The other gets a structured, accurate response they can parse directly into their application. The difference is entirely in the prompt.
The first engineer wrote: "Summarize this customer feedback."
The second wrote:
You are a customer insights analyst. Given raw customer feedback, extract the key sentiment and themes.
Customer feedback: [FEEDBACK]
Respond with ONLY valid JSON matching this schema:
{"sentiment": "POSITIVE|NEGATIVE|NEUTRAL", "themes": ["theme1", "theme2"], "priority": "HIGH|MEDIUM|LOW"}
No explanation. No preamble. Only the JSON object.
The second prompt is not longer for the sake of it — every component exists to prevent a specific failure mode.
What you'll build
A PromptBuilder class that assembles prompts from named components, plus a validator that retries on parse failures. By the end of this lesson you can diagnose any broken prompt and fix it in under 5 minutes.
The anatomy of a production prompt
Every reliable prompt has five components. Remove any one and you introduce a failure mode.
| Component | What it does | Common failure when missing |
|---|
| Role | Tells the model who it is and its expertise level | Generic, unfocused responses |
| Context | Background info the model needs to do the task | Hallucination, wrong assumptions |
| Task | Exactly what to do (verb-led, unambiguous) | Model guesses at what you want |
| Format | How to structure the output | Unparseable output, wrong structure |
| Constraints | What NOT to do, limits, guardrails | Model adds commentary, violates rules |
Here's the before/after for a contract analysis task:
Vague prompt (fails):
Analyze this contract.
Structured prompt (reliable):
You are a legal contract analyst specializing in SaaS agreements. [Role]
The following is a software licensing agreement between a vendor and enterprise customer. [Context]
Identify and list all clauses that could expose the customer to financial liability beyond the subscription fee. For each clause, quote the relevant text. [Task]
Respond as a JSON array of objects with keys: "clause_name", "quote", "risk_level" (HIGH/MEDIUM/LOW), "explanation". [Format]
Do not summarize the contract. Do not add legal advice. Only identify liability clauses. [Constraints]
The structured prompt takes 20 seconds longer to write and saves hours of debugging. Each component addresses a distinct failure mode:
You're extracting structured data from unstructured legal text. The model keeps adding commentary outside the JSON. What's the most reliable fix?
Explicit format constraints at the end of the prompt (where the model pays most attention) are the most reliable fix. Adding the exact schema the model should follow removes ambiguity. JSON mode in the API (if available) is an additional guardrail, but explicit instructions come first.
Zero-shot, few-shot, and chain-of-thought
Zero-shot
Just describe the task — no examples. Use when the task is unambiguous and the model has strong priors for it (common language tasks, standard formats).
prompt = """
Classify the sentiment of this review as POSITIVE, NEGATIVE, or NEUTRAL.
Review: "The battery lasts all day but the camera is disappointing."
Sentiment:
"""
Few-shot
Provide 2–5 input → output examples before the real query. Use when the output format is unusual, the vocabulary must match exactly, or zero-shot is producing inconsistent results.
prompt = """
Classify sentiment as POSITIVE, NEGATIVE, or NEUTRAL.
Review: "Best laptop I've ever owned. Totally worth it."
Sentiment: POSITIVE
Review: "Wouldn't buy again. Broke after two weeks."
Sentiment: NEGATIVE
Review: "It's fine. Does what it says."
Sentiment: NEUTRAL
Review: "The battery lasts all day but the camera is disappointing."
Sentiment:
"""
Few-shot examples anchor the model to your exact output vocabulary. The model pattern-matches on your examples before generating.
Chain-of-thought
Ask the model to reason step-by-step before answering. Use for multi-step problems, math, logic, or whenever accuracy matters more than speed.
prompt = """
A store has 48 items. 25% are electronics, and 1/3 of those are laptops.
How many laptops does the store have?
Think through this step by step before giving the final number.
"""
Adding "think step by step" or "let's reason through this" significantly improves accuracy on complex tasks because it forces the model to use its context window as working memory.
When to use which
- Zero-shot: unambiguous tasks, standard formats, speed matters
- Few-shot: custom output format, exact vocabulary required, zero-shot is inconsistent
- Chain-of-thought: multi-step reasoning, math, logic, compliance where you need an audit trail
You need a model to classify customer sentiment as POSITIVE, NEGATIVE, or NEUTRAL. The model sometimes returns "Mixed" or "Unclear." What technique fixes this?
Few-shot examples anchor the model to your exact output vocabulary. Showing 3–5 examples of input→POSITIVE, input→NEGATIVE, input→NEUTRAL makes "Mixed" and "Unclear" vanish because they don't appear in any example. This is more reliable than instructions alone.
Controlling output format
Getting reliable structured output is one of the most important practical skills. Three layers of control:
1. Structured outputs (API-level) The strongest control is to hand the API your schema and let it enforce it. With OpenAI's structured outputs, decoding is constrained so the response always matches the schema, not just "is JSON":
from pydantic import BaseModel
class Review(BaseModel):
sentiment: str
themes: list[str]
# OpenAI: the SDK turns the Pydantic model into a strict JSON schema
completion = client.chat.completions.parse(
model="gpt-4.1-mini", # any model with structured-output support
messages=[{"role": "user", "content": prompt}],
response_format=Review,
)
review = completion.choices[0].message.parsed # a Review instance
With Anthropic, define a tool whose input_schema is your JSON schema and force the model to call it (tool_choice); the tool call's arguments are your structured output.
Older "JSON mode" (response_format={"type": "json_object"} on OpenAI) only guarantees syntactically valid JSON. It does NOT guarantee the JSON matches your schema, so prefer schema-enforced outputs wherever your model supports them.
2. Schema in prompt Tell the model exactly what keys and types you expect:
Output ONLY valid JSON with this exact schema:
{"name": str, "score": int (0-100), "tags": list[str]}
No explanation. No markdown fences. Only the JSON object.
3. Pydantic validation with retry Parse the output and retry if it doesn't match your schema. You still need this: for providers or models without schema enforcement, and for rules a schema cannot express ("themes must come from this list of 12"):
from typing import Literal
from pydantic import BaseModel, ValidationError
import json
class ReviewAnalysis(BaseModel):
sentiment: Literal["POSITIVE", "NEGATIVE", "NEUTRAL"]
themes: list[str]
priority: Literal["HIGH", "MEDIUM", "LOW"]
def parse_with_retry(raw: str, schema: type[BaseModel], max_retries: int = 3):
for attempt in range(max_retries):
try:
data = json.loads(raw)
return schema(**data)
except (json.JSONDecodeError, ValidationError) as e:
if attempt == max_retries - 1:
raise
# Re-prompt with the error as feedback
raw = re_prompt_with_error(raw, str(e))
Pydantic validation is your safety net: it catches schema mismatches before they silently corrupt your application.
Build
Build it
Before building the full PromptBuilder, try a minimal version that assembles a prompt from its five components and shows what happens when pieces are missing.
def build_prompt(role="", context="", task="", format_spec="", constraints=""):
parts = [p for p in (role, context, task, format_spec, constraints) if p]
return "\n\n".join(parts)
vague = build_prompt(task="Analyze this contract.")
structured = build_prompt(
role="You are a legal contract analyst specializing in SaaS agreements.",
context="The following is a licensing agreement between a vendor and enterprise customer.",
task="Identify clauses that could expose the customer to financial liability.",
format_spec='Respond as JSON: [{"clause": str, "risk_level": "HIGH|MEDIUM|LOW"}]',
constraints="Do not summarize the contract. Only identify liability clauses.",
)
print("=== VAGUE PROMPT ===")
print(vague)
print(f"\n({len(vague.split())} words, missing 4 of 5 components)\n")
print("=== STRUCTURED PROMPT ===")
print(structured)
print(f"\n({len(structured.split())} words, all 5 components present)")
The code artifact for this lesson is a PromptBuilder class with all five components and output validation with automatic retry.
"""
lesson_00_02_prompts.py
PromptBuilder class implementing the role/context/task/format/constraints pattern.
Includes zero-shot/few-shot factory methods, Pydantic output validation with retry,
and a comparison runner that tests the same query across prompt variants.
Usage:
export OPENAI_API_KEY=sk-...
python lesson_00_02_prompts.py
Requirements:
pip install openai pydantic
"""
from __future__ import annotations
import json
import os
import re
from dataclasses import dataclass, field
from typing import Any, Literal
import openai
from pydantic import BaseModel, ValidationError
# ─── Prompt builder ───────────────────────────────────────────────────────────
@dataclass
class PromptBuilder:
"""
Assembles a prompt from named components.
Missing components are omitted (not padded with generic text).
"""
role: str = ""
context: str = ""
task: str = ""
format_spec: str = ""
constraints: str = ""
examples: list[dict[str, str]] = field(default_factory=list)
def add_example(self, input_text: str, output_text: str) -> "PromptBuilder":
self.examples.append({"input": input_text, "output": output_text})
return self
def build(self) -> str:
parts: list[str] = []
if self.role:
parts.append(self.role)
if self.context:
parts.append(self.context)
if self.examples:
parts.append("\n--- Examples ---")
for ex in self.examples:
parts.append(f"Input: {ex['input']}\nOutput: {ex['output']}")
parts.append("--- End examples ---")
if self.task:
parts.append(self.task)
if self.format_spec:
parts.append(self.format_spec)
if self.constraints:
parts.append(self.constraints)
return "\n\n".join(p.strip() for p in parts if p.strip())
@classmethod
def zero_shot(cls, task: str, format_spec: str = "", constraints: str = "") -> "PromptBuilder":
return cls(task=task, format_spec=format_spec, constraints=constraints)
@classmethod
def few_shot(
cls, task: str, examples: list[dict[str, str]],
format_spec: str = "", constraints: str = "",
) -> "PromptBuilder":
b = cls(task=task, format_spec=format_spec, constraints=constraints)
b.examples = examples
return b
# ─── Output schema ────────────────────────────────────────────────────────────
class SentimentResult(BaseModel):
sentiment: Literal["POSITIVE", "NEGATIVE", "NEUTRAL"]
themes: list[str]
confidence: float # 0.0 – 1.0
summary: str
# ─── Parser with retry ────────────────────────────────────────────────────────
class OutputValidator:
def __init__(self, client: openai.OpenAI, model: str = "gpt-4o-mini"):
self.client = client
self.model = model
def complete(self, prompt: str, max_tokens: int = 512) -> str:
resp = self.client.chat.completions.create(
model=self.model,
messages=[{"role": "user", "content": prompt}],
max_tokens=max_tokens,
temperature=0.1,
response_format={"type": "json_object"},
)
return resp.choices[0].message.content
def parse_with_retry(
self,
prompt: str,
schema: type[BaseModel],
max_retries: int = 3,
) -> BaseModel:
raw = self.complete(prompt)
for attempt in range(max_retries):
try:
data = json.loads(raw)
return schema(**data)
except (json.JSONDecodeError, ValidationError) as exc:
if attempt == max_retries - 1:
raise RuntimeError(
f"Failed to parse after {max_retries} attempts.\n"
f"Last error: {exc}\nLast raw output: {raw}"
) from exc
# Re-prompt with error feedback
error_prompt = (
f"{prompt}\n\n"
f"Your previous response could not be parsed:\n"
f"Error: {exc}\n"
f"Response was: {raw}\n\n"
f"Please output ONLY valid JSON matching the required schema. "
f"No explanation, no markdown, just the JSON object."
)
raw = self.complete(error_prompt)
raise RuntimeError("Unreachable")
# ─── Side-by-side comparison ──────────────────────────────────────────────────
def compare_prompts(
client: openai.OpenAI,
query: str,
variants: dict[str, str],
runs_per_variant: int = 3,
) -> dict[str, list[str]]:
"""Run each prompt variant N times and collect raw outputs."""
results: dict[str, list[str]] = {}
for label, prompt in variants.items():
outputs = []
for _ in range(runs_per_variant):
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": prompt.replace("{QUERY}", query)}],
max_tokens=200,
temperature=0.7,
)
outputs.append(resp.choices[0].message.content.strip())
results[label] = outputs
return results
def estimate_token_cost(text: str, model: str = "gpt-4o-mini") -> float:
"""Rough token cost estimate based on character count (4 chars ≈ 1 token)."""
tokens = len(text) / 4
input_price_per_1m = 0.15 # gpt-4o-mini
return (tokens / 1_000_000) * input_price_per_1m
# ─── Demo ─────────────────────────────────────────────────────────────────────
VAGUE_PROMPT = "Analyze this review: {QUERY}"
STRUCTURED_PROMPT = (
"You are a customer experience analyst.\n\n"
"Given a product review, extract the sentiment and key themes.\n\n"
"Review: {QUERY}\n\n"
'Output ONLY valid JSON with this exact schema:\n'
'{"sentiment": "POSITIVE|NEGATIVE|NEUTRAL", "themes": ["theme1"], '
'"confidence": 0.0-1.0, "summary": "one sentence"}\n\n'
"No explanation. No markdown. Only the JSON object."
)
FEW_SHOT_PROMPT = (
"You are a customer experience analyst. "
"Classify each review's sentiment and themes.\n\n"
"Review: 'Amazing battery, love the design, but the camera is mediocre.'\n"
'{"sentiment": "POSITIVE", "themes": ["battery", "design", "camera"], '
'"confidence": 0.8, "summary": "Mostly positive with camera concern."}\n\n'
"Review: 'Arrived broken. Customer service was useless.'\n"
'{"sentiment": "NEGATIVE", "themes": ["shipping", "customer service"], '
'"confidence": 0.95, "summary": "Negative experience on all fronts."}\n\n'
"Review: {QUERY}\n"
"Output ONLY the JSON object. No explanation."
)
def main() -> None:
client = openai.OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
test_review = (
"The laptop runs fast and the keyboard feels great, "
"but the fan noise is distracting and the price is too high."
)
print("\n" + "=" * 60)
print("PROMPT VARIANTS COMPARISON")
print("=" * 60)
variants = {
"zero_shot_vague": VAGUE_PROMPT,
"zero_shot_structured": STRUCTURED_PROMPT,
"few_shot": FEW_SHOT_PROMPT,
}
results = compare_prompts(client, test_review, variants, runs_per_variant=2)
for label, outputs in results.items():
print(f"\n[{label}]")
for i, out in enumerate(outputs, 1):
print(f" Run {i}: {out[:120]}")
# Validated parsing demo
print("\n\n" + "=" * 60)
print("PYDANTIC VALIDATION WITH RETRY")
print("=" * 60)
validator = OutputValidator(client)
structured_prompt_filled = STRUCTURED_PROMPT.replace("{QUERY}", test_review)
try:
result: SentimentResult = validator.parse_with_retry(structured_prompt_filled, SentimentResult)
print(f"\nSentiment: {result.sentiment}")
print(f"Themes: {result.themes}")
print(f"Confidence: {result.confidence:.2f}")
print(f"Summary: {result.summary}")
except RuntimeError as e:
print(f"Parse failed: {e}")
# Token cost estimate
print(f"\n\nToken cost estimates for test_review prompt variants:")
for label, prompt in variants.items():
cost = estimate_token_cost(prompt.replace("{QUERY}", test_review))
print(f" {label:<30} ~${cost:.8f} per call")
if __name__ == "__main__":
main()