Phase 4: Harness Engineering · 45 min · Claude Code · Python

Harness Regression Testing

A rules file, a skill, and a tool config are infrastructure. Infrastructure that isn't tested regresses silently, exactly like code does.

Hiring signal: Treating CLAUDE.md and skill definitions as versioned, tested infrastructure — not just prose you edit and hope for the best — is the harness-engineering maturity this whole phase has been building toward.

What you will learn

Explain why a harness change (a rule, a skill, a tool config) can regress agent behavior the same way a code change can regress a product
Identify when a new skill silently conflicts with an existing rule for a specific task shape
Design a harness regression suite that re-runs representative scenarios after any harness change
Catch a real regression: a previously-passing scenario failing after a new skill is added

Introduction

Harness Regression Testing

A team's rules file has stated, for months, "always ask for confirmation before writing any file to disk" — a sensible default that's kept the agent from silently overwriting things it shouldn't. This week, someone adds a new skill: a batch-export helper meant to speed up a specific reporting task, whose instructions say "for batch export tasks, write output files directly without confirmation to maximize throughput." Both pieces of the harness are individually reasonable. Together, for exactly one task shape — batch exports — the more specific skill instruction wins out over the general rule, and the agent starts silently writing files it used to ask permission for. Nobody changed the product code. Nobody touched a single test in the product's test suite. The regression is entirely in the harness, and a test suite that only checks product behavior has nothing to say about it.

The harness is infrastructure, and infrastructure regresses

Everything covered in this phase — rules, skills, tools, feedback loops, approval gates — is itself a piece of engineering infrastructure, changed over time by people who are usually thinking about the specific problem in front of them, not every downstream interaction their change might have with something else already in place. A new skill can silently override a rule for a specific task shape, exactly like the batch-export example. A tool description edited to be "clearer" can change which tool the agent reaches for in an ambiguous case. A rules file tightened in one section can loosen behavior in an unrelated one, if the agent's interpretation of the whole file shifts. None of these are code bugs in the traditional sense — they're harness regressions, and the fix is the same one the software industry already has for code: a regression suite that runs whenever the infrastructure changes.

Why product tests don't catch harness regressions
The product's test suite checks product behavior against product acceptance criteria — does the discount calculation cap at 100%, does the timestamp parser handle the hours format. None of that touches whether a new skill silently overrides an existing rule for a specific task. A harness regression only shows up in a test that specifically checks harness behavior across representative task scenarios, which is a different suite with a different purpose.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Building a harness regression suite, Versioned, tested, reviewed like any other infrastructure, Choosing what belongs in the regression suite, Build It — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.