Phase 3: The Generate-Validate Loop · 45 min · Claude Code · Python

Test-Driven AI Generation

Write the tests before the agent writes the code, and 'what did you mean' stops being a question anyone has to guess at.

Hiring signal: Encoding requirements as failing tests before prompting generation is concrete evidence of test-driven discipline applied to AI work, not just a claim that you 'write tests.'

What you will learn

Explain why writing failing tests before generation prevents the 'AI guessed what I meant' problem
Apply the red-green-refactor loop to AI-assisted development
Encode a spec's acceptance criteria as concrete, runnable test cases before prompting for an implementation
Use passing tests as a safety net while refactoring generated code

Introduction

Test-Driven AI Generation

Marcus asks an agent to "add a function to parse podcast timestamps like [12:34] into total seconds." It generates something that works on [12:34]. He asks the same question again the next day, slightly rephrased, and gets an implementation that handles [1:02:03] (with hours) differently than the first version did — one truncates unexpected formats silently, the other raises an exception. Neither version was wrong, exactly; "parse a timestamp" was never specific enough to have one correct answer, so each generation attempt filled the gap with a different, equally plausible guess. This is the "AI guessed what I meant" problem, and it isn't really about the AI — it's about what happens whenever the requirement was never made concrete before generation started.

The frustrating part is that this doesn't announce itself as a problem. Both versions of Marcus's function run without error on the inputs he happens to try. The divergence only becomes visible later, when a transcript with the hours format shows up in production and behaves differently than it did in yesterday's test run — at which point it looks like a regression, even though nothing regressed. Nothing was ever specified precisely enough to regress from. The fix isn't asking more carefully in prose; prose descriptions, however careful, still leave room for two equally reasonable readings. The fix is making the requirement mechanically checkable before generation starts.

Tests as the concrete version of "what I meant"

A test case is a spec's acceptance criteria made executable: instead of saying "parse timestamps correctly," you write parse_timestamp("12:34") == 754 (12 minutes, 34 seconds), parse_timestamp("1:02:03") == 3723 (1 hour, 2 minutes, 3 seconds), and parse_timestamp("garbage") raises ValueError. There's no ambiguity left for an agent to guess at — either the function returns 754 for "12:34" or it doesn't, and that's checkable mechanically, not by re-reading the code and deciding if it "looks right." Writing these tests before any implementation exists means the requirement gets pinned down while it's still cheap to think clearly about — including edge cases (the hours format, malformed input) that are easy to forget once you're looking at code that already handles the common case convincingly.

Why tests-first, not tests-eventually
Writing tests after an implementation exists tends to describe what the implementation already does, which is the same mirror problem from earlier in this course's spec-driven development lessons — the test just confirms the code agrees with itself. Writing tests first, from the requirement, means the test can actually disagree with whatever gets generated, which is the entire point.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Red, green, refactor — adapted for AI generation, Encoding a spec's acceptance criteria as tests, The refactor stage isn't optional cleanup, Build It — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.

Key terms in this lesson

Agent