Phase 3: Property-Based Testing for AI Code · 55 min · Claude Code · Hypothesis · pytest

The Anthropic Agentic PBT Approach

The AI reads the code. The AI writes the properties. Hypothesis finds the bugs.

Hiring signal: Engineers who can apply the Anthropic agentic PBT 5-step process demonstrate the ability to leverage AI for scalable property generation -- the frontier of AI-assisted testing.

What you will learn

Apply the 5-step agentic PBT process: read, propose, write, run, reflect
Understand the 56% valid bug discovery rate across 100 Python packages
Use Claude Code's /hypothesis command for automated property generation
Evaluate agent-generated properties for quality and coverage

The Problem

Writing good properties is hard. It requires understanding the code's semantics, identifying invariants, and choosing appropriate strategies. Most developers write 2-3 properties and stop. But what if an AI agent could read the code, propose properties, write Hypothesis tests, run them, and reflect on the results -- autonomously?

Anthropic did exactly this. Claude Code's /hypothesis command implements a 5-step agentic process that found valid bugs in 56% of reports across 100 Python packages, including bugs in NumPy, SciPy, Pandas, Google SDKs, and Amazon SDKs.

The 5-Step Agentic PBT Process

Step	What the agent does	Output
1. Read	Read the target function's source code, docstring, and type hints	Understanding of function semantics
2. Propose	Propose 3-5 properties the function should satisfy	List of candidate properties
3. Write	Write Hypothesis property tests for each proposed property	Python test code with @given and strategies
4. Run	Run the property tests with Hypothesis	Pass/fail results with counterexamples
5. Reflect	Analyze failures: is it a real bug or a property error?	Bug report or property revision

The 56% valid bug rate
Anthropic ran this process on 100 Python packages. 56% of bug reports were valid -- meaning more than half of the bugs the agent found were real defects in production code used by millions of developers. The bugs were in NumPy (array operations), SciPy (statistical functions), Pandas (data manipulation), Google Cloud SDK, and Amazon boto3. This is not a toy demo -- it's real bugs in real production code found by an AI agent writing property tests.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Bugs Found in Production Code, Claude Code /hypothesis Command, The Reflect Step: Distinguishing Bugs from Property Errors, Build It — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.

Key terms in this lesson

Agent