Phase 1: Advanced Prompt Engineering for Agents · 55 min · Python · Anthropic SDK

Prompt Versioning & Management

Prompts are code — they need version control, CI, and a rollback plan.

What you will learn

Version control prompts as code artifacts with semantic versioning
Build a prompt registry that tracks versions, environments, and performance metrics
Implement A/B testing for prompts with statistical significance

The Problem

A prompt that worked perfectly in February starts producing wrong outputs in May. You don't know which version of the prompt is in production — it was edited directly in the codebase three months ago and nobody left a comment. After a model version update you didn't notice, output quality degraded. You can't roll back because there's no previous version to roll back to.

This is not a hypothetical. It happens to every team that treats prompts as strings rather than artifacts. The symptoms are always the same: "something changed," no one knows what, and reproducing the failure is nearly impossible without version history.

The root cause is treating prompts differently than code. If a configuration file changed and broke production, you'd check git blame, roll back, and write a test to catch the regression. Prompts deserve exactly the same treatment.

A word change is a breaking change
A single word change in a prompt can shift output distribution dramatically. Changing "summarize" to "extract key points" in a summarization prompt doubled average output length in one production system. Without a golden test set run before deployment, this change shipped to users and wasn't caught for 2 weeks.

Why prompts need version control

Prompts are code. They have bugs (wrong behavior under edge cases), they break with model updates (a prompt tuned for claude-3-opus may not work the same on claude-3-5-sonnet), and they need rollback (you shipped a bad prompt, now what?).

Minimum viable prompt version control:

Store prompts in git alongside code — not in a database, not in environment variables, not hardcoded in application files
Semantic versioning — v1.2.3 where major = breaking behavior change, minor = improvement, patch = typo/wording fix
Tag with the model — every prompt version records which model it was tuned for. gpt-4o-2024-11-20 and claude-sonnet-4-5 have different tokenization, context windows, and instruction-following characteristics
Never deploy untested — run a golden test set before any prompt goes to production

Practical directory structure:

prompts/
  summarizer/
    v1.0.0.yaml
    v1.1.0.yaml    ← current production
    v1.2.0.yaml    ← staging (under test)
  classifier/
    v2.0.0.yaml
    v2.1.0.yaml

Each YAML file contains the prompt text, target model, creation date, and eval scores:

id: summarizer
version: 1.1.0
model: claude-sonnet-4-5
created: 2026-03-15
author: darian
change_reason: Reduced average output length by 30% with tighter instruction
eval_score: 0.87
prompt: |
  Summarize the following document in 3–5 sentences.
  Focus on: main argument, key supporting evidence, conclusion.
  Do not include introductory phrases like "This document discusses..."

The prompt registry pattern

A prompt registry is a central store with an API for managing prompt versions:

registry.get(id="summarizer", version="production")  # → v1.1.0 text
registry.get(id="summarizer", version="1.2.0")        # → specific version
registry.promote(id="summarizer", version="1.2.0")    # → sets production pointer
registry.rollback(id="summarizer")                     # → reverts to prior production

The production pointer is a separate metadata record (not duplicated in the YAML):

# prompts/summarizer/production.yaml
production: 1.1.0
previous: 1.0.0
promoted_at: 2026-03-20T14:30:00Z
promoted_by: darian

This separation means you can promote or roll back without touching the prompt files themselves — the versions are immutable artifacts.

Managed services like LangSmith and Langfuse provide this pattern as a hosted service with UI, team collaboration, and automatic evaluation tracking. For teams with < 10 prompts, a YAML-in-git approach is often sufficient and avoids adding an external dependency.

You changed one word in your production summarization prompt and the average summary length doubled. How could you have caught this before it affected users?

A golden test set (fixed inputs with known good outputs) run before every prompt deployment catches regressions. If average summary length doubles on your test set, you catch it in staging — not in production. This is prompt CI/CD: run the same eval pipeline used in production, against a fixed reference dataset, before every deployment. The comparison baseline is always the current production prompt's scores.

A/B testing prompts

When you have two candidate prompts and need to decide which to promote, A/B testing provides a data-driven answer.

Setup:

Split traffic 50/50 between prompt v1 (control) and v2 (treatment)
Collect N outputs from each — minimum ~200 for reliable results
Evaluate each output with a consistent metric (LLM-as-judge, human rating, or automated format check)
Compute win rate and test for statistical significance

Statistical significance test:

from scipy import stats

control_scores = [0.87, 0.91, 0.83, ...]   # v1 outputs
treatment_scores = [0.92, 0.88, 0.95, ...]  # v2 outputs

t_stat, p_value = stats.ttest_ind(control_scores, treatment_scores)
significant = p_value < 0.05
winner = "treatment" if mean(treatment_scores) > mean(control_scores) else "control"

With n=200 samples per variant, a 5–10 point quality improvement is detectable at p < 0.05. With n=50, only large effects are detectable — too noisy for production decisions.

Defining success metrics per use case:

Summarization: output length within ±20% of target, no hallucinated facts, format compliance
Classification: accuracy against labeled holdout set
Code generation: code executes without error, passes unit tests
Customer support: resolution rate, escalation rate, user satisfaction

Your prompt registry should include which of the following for each prompt version?

A prompt version is a software artifact that needs to be auditable: what it says (text), what model it was tuned for (a prompt tuned for claude-sonnet may behave differently on GPT-4o), when and by whom it was created (audit trail), why it was changed (change log), and what eval scores it achieved (the data for promotion decisions). Without eval scores, you can't make data-driven decisions about whether to promote a new version. Without model tracking, you'll redeploy a model-incompatible prompt after a model update.

Build

Build it

Here's a minimal in-memory version of the promote/rollback mechanics — no YAML or filesystem needed to see how the production pointer moves.

registry = {}       # id -> {version: prompt_text}
production = {}      # id -> {"current": version, "previous": version}

def save(prompt_id, version, text):
    registry.setdefault(prompt_id, {})[version] = text

def promote(prompt_id, version):
    prev = production.get(prompt_id, {}).get("current")
    production[prompt_id] = {"current": version, "previous": prev}
    print(f"Promoted {prompt_id} to v{version} (was v{prev})")

def rollback(prompt_id):
    prev = production[prompt_id]["previous"]
    if prev is None:
        print(f"No previous version for {prompt_id}, cannot roll back")
        return
    promote(prompt_id, prev)

save("summarizer", "1.0.0", "Summarize concisely.")
save("summarizer", "1.1.0", "Summarize in 2-4 sentences, no preamble.")

promote("summarizer", "1.0.0")
promote("summarizer", "1.1.0")
print("Current production:", production["summarizer"]["current"])

rollback("summarizer")
print("After rollback:", production["summarizer"]["current"])

Build lesson_01_04_prompt_registry.py yourself — this is the lab. A complete implementation contains:

PromptRegistry class backed by a YAML file store with get/save/promote/rollback operations
Semantic version management with production pointer tracking
PromptABTest class: run two prompt versions on a test set, compute win rate and p-value
Model-version pinning in each stored prompt version
Drift detector that alerts when a prompt's eval score drops below a threshold

"""
lesson_01_04_prompt_registry.py

Prompt registry with YAML-backed versioning, A/B testing, and drift detection.

Usage:
    export ANTHROPIC_API_KEY=sk-ant-...
    python lesson_01_04_prompt_registry.py

Requirements:
    pip install anthropic pyyaml scipy
"""

import os
import yaml
import json
import math
from dataclasses import dataclass, asdict
from pathlib import Path
from datetime import datetime
from typing import Optional
import anthropic

client = anthropic.Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY", ""))
REGISTRY_DIR = Path("prompts_registry")

@dataclass
class PromptVersion:
    id: str
    version: str
    model: str
    prompt: str
    created: str
    author: str
    change_reason: str
    eval_score: float = 0.0

class PromptRegistry:
    """YAML-backed prompt version store."""

    def __init__(self, base_dir: Path = REGISTRY_DIR):
        self.base_dir = base_dir
        self.base_dir.mkdir(exist_ok=True)

    def _prompt_dir(self, prompt_id: str) -> Path:
        d = self.base_dir / prompt_id
        d.mkdir(exist_ok=True)
        return d

    def save(self, pv: PromptVersion) -> None:
        path = self._prompt_dir(pv.id) / f"v{pv.version}.yaml"
        with open(path, "w") as f:
            yaml.dump(asdict(pv), f, default_flow_style=False)
        print(f"Saved {pv.id} v{pv.version}")

    def get(self, prompt_id: str, version: str = "production") -> PromptVersion:
        if version == "production":
            meta_path = self._prompt_dir(prompt_id) / "production.yaml"
            if not meta_path.exists():
                raise FileNotFoundError(f"No production pointer for {prompt_id}")
            with open(meta_path) as f:
                meta = yaml.safe_load(f)
            version = meta["version"]

        path = self._prompt_dir(prompt_id) / f"v{version}.yaml"
        if not path.exists():
            raise FileNotFoundError(f"Version {version} not found for {prompt_id}")
        with open(path) as f:
            data = yaml.safe_load(f)
        return PromptVersion(**data)

    def promote(self, prompt_id: str, version: str) -> None:
        # Read current production if it exists
        meta_path = self._prompt_dir(prompt_id) / "production.yaml"
        previous = None
        if meta_path.exists():
            with open(meta_path) as f:
                current = yaml.safe_load(f)
            previous = current.get("version")

        meta = {
            "version": version,
            "previous": previous,
            "promoted_at": datetime.utcnow().isoformat(),
        }
        with open(meta_path, "w") as f:
            yaml.dump(meta, f)
        print(f"Promoted {prompt_id} to v{version} (was v{previous})")

    def rollback(self, prompt_id: str) -> None:
        meta_path = self._prompt_dir(prompt_id) / "production.yaml"
        with open(meta_path) as f:
            meta = yaml.safe_load(f)
        previous = meta.get("previous")
        if not previous:
            raise ValueError(f"No previous version to roll back to for {prompt_id}")
        self.promote(prompt_id, previous)
        print(f"Rolled back {prompt_id} to v{previous}")

    def list_versions(self, prompt_id: str) -> list[str]:
        d = self._prompt_dir(prompt_id)
        return sorted(
            f.stem[1:] for f in d.glob("v*.yaml")
        )

def _llm_score(prompt_text: str, test_input: str, expected_keywords: list[str]) -> float:
    """Simple format-compliance score: check if expected keywords appear in output."""
    if not client.api_key:
        return 0.8  # Mock score in offline mode
    resp = client.messages.create(
        model="claude-haiku-4-5",
        max_tokens=200,
        system=prompt_text,
        messages=[{"role": "user", "content": test_input}],
    )
    output = resp.content[0].text.lower()
    hits = sum(1 for kw in expected_keywords if kw.lower() in output)
    return hits / len(expected_keywords) if expected_keywords else 1.0

class PromptABTest:
    """Run two prompt versions on a test set and compute win rate."""

    def run(
        self,
        registry: PromptRegistry,
        prompt_id: str,
        version_a: str,
        version_b: str,
        test_cases: list[dict],
    ) -> dict:
        pv_a = registry.get(prompt_id, version_a)
        pv_b = registry.get(prompt_id, version_b)

        scores_a, scores_b = [], []
        for tc in test_cases:
            sa = _llm_score(pv_a.prompt, tc["input"], tc.get("keywords", []))
            sb = _llm_score(pv_b.prompt, tc["input"], tc.get("keywords", []))
            scores_a.append(sa)
            scores_b.append(sb)

        mean_a = sum(scores_a) / len(scores_a)
        mean_b = sum(scores_b) / len(scores_b)

        # Simple t-test approximation (Welch's)
        n = len(scores_a)
        var_a = sum((x - mean_a) ** 2 for x in scores_a) / max(n - 1, 1)
        var_b = sum((x - mean_b) ** 2 for x in scores_b) / max(n - 1, 1)
        se = math.sqrt(var_a / n + var_b / n) + 1e-10
        t_stat = (mean_a - mean_b) / se

        winner = version_a if mean_a >= mean_b else version_b
        return {
            f"score_{version_a}": round(mean_a, 3),
            f"score_{version_b}": round(mean_b, 3),
            "t_stat": round(t_stat, 3),
            "winner": winner,
            "n": n,
        }

def check_drift(registry: PromptRegistry, prompt_id: str, threshold: float = 0.75) -> None:
    """Alert if current production prompt score is below threshold."""
    pv = registry.get(prompt_id, "production")
    if pv.eval_score < threshold:
        print(
            f"[DRIFT ALERT] {prompt_id} production score {pv.eval_score:.2f} "
            f"is below threshold {threshold:.2f} — consider rolling back or retuning."
        )
    else:
        print(f"[OK] {prompt_id} production score {pv.eval_score:.2f} (threshold: {threshold:.2f})")

if __name__ == "__main__":
    registry = PromptRegistry()

    # Save two versions of a summarizer prompt
    v1 = PromptVersion(
        id="summarizer", version="1.0.0", model="claude-haiku-4-5",
        prompt="Summarize the following text concisely.",
        created="2026-01-15", author="darian",
        change_reason="Initial version", eval_score=0.72,
    )
    v2 = PromptVersion(
        id="summarizer", version="1.1.0", model="claude-haiku-4-5",
        prompt=(
            "Summarize the following text in 2–4 sentences. "
            "Focus on: main claim, key evidence, conclusion. "
            "Do not start with 'This text...' or 'The author...'"
        ),
        created="2026-02-01", author="darian",
        change_reason="Tighter instructions reduced length and improved format compliance",
        eval_score=0.89,
    )
    registry.save(v1)
    registry.save(v2)
    registry.promote("summarizer", "1.0.0")

    print("\nA/B Test: v1.0.0 vs v1.1.0")
    test_cases = [
        {"input": "Explain quantum entanglement in simple terms.", "keywords": ["quantum", "particles"]},
        {"input": "What is the capital of France and why is it famous?", "keywords": ["paris", "france"]},
        {"input": "Describe the water cycle briefly.", "keywords": ["evaporation", "precipitation"]},
    ]
    ab = PromptABTest()
    result = ab.run(registry, "summarizer", "1.0.0", "1.1.0", test_cases)
    print(json.dumps(result, indent=2))

    print("\nDrift check before promotion:")
    check_drift(registry, "summarizer", threshold=0.80)

    print("\nPromoting v1.1.0 to production...")
    registry.promote("summarizer", "1.1.0")

    print("\nDrift check after promotion:")
    check_drift(registry, "summarizer", threshold=0.80)

    print("\nRollback test:")
    registry.rollback("summarizer")
    print(f"Available versions: {registry.list_versions('summarizer')}")

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Check Yourself — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.

Key terms in this lesson

A/B Testing