Phase 3: Property-Based Testing for AI Code · 50 min · Python · Hypothesis · pytest

Introduction to Property-Based Testing

Don't test examples. Test invariants.

Hiring signal: Engineers who can explain PBT vs example-based testing and why PBT catches AI logic errors demonstrate understanding of testing at a deeper level than most QA engineers.

What you will learn

Explain the difference between example-based and property-based testing
Identify invariants in AI-generated code that property tests should verify
Describe the history of PBT from QuickCheck to Hypothesis
Understand why PBT catches AI logic errors that example tests miss

The Problem

An AI generates a sort_list function:

def sort_list(items):
    return sorted(items)

You write 3 example-based tests:

def test_sort_basic():
    assert sort_list([3, 1, 2]) == [1, 2, 3]

def test_sort_empty():
    assert sort_list([]) == []

def test_sort_single():
    assert sort_list([1]) == [1]

All pass. But what about [3, 1, 2, 1] (duplicates)? [-1, 0, 1] (negatives)? [3, -1, float('inf'), 0] (mixed types)? [1] * 10000 (large input)? Example-based tests only check the specific inputs you thought of. AI code has edge cases you didn't think of.

Property-based testing flips the approach: instead of specifying individual inputs and expected outputs, you specify invariants -- properties that must hold for all inputs -- and the testing framework generates hundreds of random inputs to try to break them.

Example-Based vs Property-Based Testing

Aspect	Example-Based	Property-Based
What you specify	Input + expected output	Invariant (property) that must hold
Number of test cases	3-10 (what you thought of)	100-1000 (randomly generated)
Edge case coverage	Only edges you considered	Framework finds edges you missed
AI code relevance	Misses AI-specific edge cases	Catches AI logic errors at scale
Failure reporting	"test_sort_basic failed"	Minimal counterexample (shrunk)
Maintenance	Add new test for each edge case	Property covers all cases automatically

Example: Sorting

# Example-based: 3 test cases
def test_sort_basic():
    assert sort_list([3, 1, 2]) == [1, 2, 3]

# Property-based: 1 property, 200+ test cases
@given(st.lists(st.integers()))
def test_sort_produces_sorted_output(items):
    result = sort_list(items)
    assert result == sorted(result)  # invariant: output is sorted

@given(st.lists(st.integers()))
def test_sort_preserves_length(items):
    result = sort_list(items)
    assert len(result) == len(items)  # invariant: length preserved

@given(st.lists(st.integers()))
def test_sort_preserves_elements(items):
    result = sort_list(items)
    assert sorted(result) == sorted(items)  # invariant: same elements

PBT catches what example tests miss
The AI generates a sort function that works for [3,1,2] but fails for duplicates [3,1,2,1] (it removes duplicates). Example tests with [3,1,2] pass. But the property "sort preserves elements" fails for [3,1,2,1] because the result [1,2,3] is missing the duplicate 1. Hypothesis finds this edge case automatically by generating random lists including ones with duplicates. The property catches the AI logic error that no example test covered.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers History: QuickCheck to Hypothesis, Build It — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.