Phase 3: Property-Based Testing for AI Code · 55 min · Python · Hypothesis · pytest

PBT for AI-Generated Code

The AI writes the code. The AI writes the properties. Hypothesis finds the contradictions.

Hiring signal: Engineers who can implement the dual-mode workflow (AI writes code + AI writes properties + Hypothesis finds contradictions) demonstrate the ability to create self-improving verification loops for AI-generated code.

What you will learn

Implement the dual-mode workflow: AI writes code + AI writes properties + Hypothesis finds contradictions
Generate properties from function specifications and type hints
Verify AI-generated functions using PBT as a verification layer
Handle the case where AI code and AI properties disagree -- which is wrong?

The Problem

You've learned PBT concepts (Lesson 1), Hypothesis mechanics (Lesson 2), property derivation (Lesson 3), and the agentic approach (Lesson 4). Now let's put it all together into a practical workflow for testing AI-generated code.

The dual-mode workflow is:

AI writes the code (producer mode)
AI writes the properties (verifier mode)
Hypothesis finds contradictions (objective arbiter)

When Hypothesis finds a contradiction (a property fails), one of two things is wrong:

The code is wrong (the property is correct, the code violates it)
The property is wrong (the code is correct, the property doesn't apply)

The human reviews the contradiction and decides which is wrong. This is the self-improving loop.

The Dual-Mode Workflow

┌──────────────────────────────────────────┐
│  Step 1: AI generates code (producer)    │
│  "Write a function that normalizes       │
│   scores to [0, 1]"                      │
│  → ai_normalize(scores)                  │
└─────────────────┬────────────────────────┘
                  ↓
┌──────────────────────────────────────────┐
│  Step 2: AI generates properties         │
│  (verifier mode -- separate context)     │
│  "What properties should normalize       │
│   satisfy?"                              │
│  → bounded_0_1, length_preserved,        │
│    idempotent, monotonic                 │
└─────────────────┬────────────────────────┘
                  ↓
┌──────────────────────────────────────────┐
│  Step 3: Hypothesis runs properties      │
│  against code (objective arbiter)        │
│  → 200 random inputs per property        │
│  → Find contradictions                   │
└─────────────────┬────────────────────────┘
                  ↓
┌──────────────────────────────────────────┐
│  Step 4: Human reviews contradictions    │
│  Is the code wrong or the property?      │
│  → Fix code (if code is wrong)           │
│  → Fix property (if property is wrong)   │
└─────────────────┬────────────────────────┘
                  ↓
┌──────────────────────────────────────────┐
│  Step 5: Re-run Hypothesis               │
│  → All properties pass? Done.            │
│  → New contradictions? Go to Step 4.     │
└──────────────────────────────────────────┘

Why separate contexts for code and properties
The AI that writes the code should NOT also write the properties in the same context. If the AI writes both in the same session, it has a cognitive bias: it writes properties that its own code passes. By using separate contexts (separate prompts, separate sessions), the verifier AI approaches the function with fresh eyes -- it reads the spec, not the implementation, and derives properties from the spec. This is analogous to why you shouldn't write tests for your own code: you test what you think the code does, not what it actually does.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Property Generation from Specifications, Build It — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.