Phase 1: Advanced Prompt Engineering for Agents · 55 min · Python · Anthropic SDK

Prompt Versioning & Management

Prompts are code — they need version control, CI, and a rollback plan.

Hiring signal: Teams at scale (Notion, Stripe, Intercom) treat prompts as first-class code artifacts with their own CI/CD pipelines. Candidates who have built or worked with a prompt registry — and can describe how they test before deploying — signal production engineering experience that most bootcamp graduates lack.

What you will learn

Version control prompts as code artifacts with semantic versioning
Build a prompt registry that tracks versions, environments, and performance metrics
Implement A/B testing for prompts with statistical significance

The Problem

A prompt that worked perfectly in February starts producing wrong outputs in May. You don't know which version of the prompt is in production — it was edited directly in the codebase three months ago and nobody left a comment. After a model version update you didn't notice, output quality degraded. You can't roll back because there's no previous version to roll back to.

This is not a hypothetical. It happens to every team that treats prompts as strings rather than artifacts. The symptoms are always the same: "something changed," no one knows what, and reproducing the failure is nearly impossible without version history.

The root cause is treating prompts differently than code. If a configuration file changed and broke production, you'd check git blame, roll back, and write a test to catch the regression. Prompts deserve exactly the same treatment.

A word change is a breaking change
A single word change in a prompt can shift output distribution dramatically. Changing "summarize" to "extract key points" in a summarization prompt doubled average output length in one production system. Without a golden test set run before deployment, this change shipped to users and wasn't caught for 2 weeks.

Why Prompts Need Version Control

Prompts are code. They have bugs (wrong behavior under edge cases), they break with model updates (a prompt tuned for claude-3-opus may not work the same on claude-3-5-sonnet), and they need rollback (you shipped a bad prompt, now what?).

Minimum viable prompt version control:

Store prompts in git alongside code — not in a database, not in environment variables, not hardcoded in application files
Semantic versioning — v1.2.3 where major = breaking behavior change, minor = improvement, patch = typo/wording fix
Tag with the model — every prompt version records which model it was tuned for. gpt-4o-2024-11-20 and claude-sonnet-4-5 have different tokenization, context windows, and instruction-following characteristics
Never deploy untested — run a golden test set before any prompt goes to production

Practical directory structure:

prompts/
  summarizer/
    v1.0.0.yaml
    v1.1.0.yaml    ← current production
    v1.2.0.yaml    ← staging (under test)
  classifier/
    v2.0.0.yaml
    v2.1.0.yaml

Each YAML file contains the prompt text, target model, creation date, and eval scores:

id: summarizer
version: 1.1.0
model: claude-sonnet-4-5
created: 2026-03-15
author: darian
change_reason: Reduced average output length by 30% with tighter instruction
eval_score: 0.87
prompt: |
  Summarize the following document in 3–5 sentences.
  Focus on: main argument, key supporting evidence, conclusion.
  Do not include introductory phrases like "This document discusses..."

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Prompt Registry Pattern, A/B Testing Prompts, Build It, What to Practice — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.

Key terms in this lesson

A/B Testing