Phase 3: Property-Based Testing for AI Code · 60 min · Python · Hypothesis · pytest

Advanced PBT: Stateful Testing

Stateful systems have invariants too. The state machine never lies.

Hiring signal: Engineers who understand stateful property testing can catch concurrency bugs in AI-generated code (like the 847 duplicate charges incident) that sequential unit tests fundamentally cannot detect.

What you will learn

Understand stateful property testing and RuleMachine
Test stateful systems AI generates (caches, databases, state machines)
Use concurrent strategies to detect race conditions
Understand how PBT with concurrent strategies would have caught the 847 duplicate charges bug

The Problem

All the PBT we've done so far tests stateless functions: sort(items), normalize(scores), parse_date(s). These functions take inputs and produce outputs without maintaining state. But many AI-generated systems are stateful: caches, databases, state machines, shopping carts, payment processors. Stateful systems have invariants that depend on the sequence of operations, not just individual inputs.

The 847 duplicate charges incident (from Phase 0) is a perfect example: a race condition in a stateful payment system caused 847 duplicate charges. Sequential unit tests couldn't catch this because they test one operation at a time. Stateful property testing generates sequences of operations and checks invariants across the entire sequence.

Stateful Property Testing

Stateful PBT extends regular PBT to test sequences of operations:

Regular PBT	Stateful PBT
Tests individual function calls	Tests sequences of operations
`@given(st.lists(st.integers()))`	`@given(st.lists(rules()))`
Invariant: `f(x) satisfies property`	Invariant: `after any sequence of operations, system state satisfies property`
Catches function-level bugs	Catches state-level bugs (race conditions, ordering bugs, state corruption)

Hypothesis Stateful Testing

Hypothesis provides RuleBasedStateMachine for stateful testing:

from hypothesis.stateful import RuleBasedStateMachine, rule, invariant

class CacheMachine(RuleBasedStateMachine):
    def __init__(self):
        self.cache = {}
        self.model = {}  # reference implementation

    @rule(key=st.text(), value=st.integers())
    def set(self, key, value):
        self.cache[key] = value
        self.model[key] = value

    @rule(key=st.text())
    def get(self, key):
        result = self.cache.get(key)
        assert result == self.model.get(key)

    @invariant()
    def cache_matches_model(self):
        assert self.cache == self.model

TestCache = CacheMachine.TestCase

Hypothesis generates random sequences of set and get operations and checks the invariant cache_matches_model after each step.

The 847 duplicate charges incident and stateful PBT
The 847 duplicate charges bug was a race condition in a payment system: two concurrent requests for the same charge both passed the 'has this charge already been processed?' check, and both processed the charge. Sequential unit tests tested the charge function in isolation -- each test called charge() once and verified the result. No test called charge() twice concurrently for the same order. Stateful PBT with concurrent strategies would have generated sequences like [charge(order_123), charge(order_123)] and the invariant 'total charges for order_123 == 1' would have failed. The state machine catches what sequential tests fundamentally cannot: bugs that only manifest across sequences of operations.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Model/Oracle Pattern for Stateful Systems, Concurrent Stateful Testing, Build It — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.