Phase 4: Mutation Testing & Behavioral Verification · 50 min · Python · pytest · mutmut

Why Coverage Lies

92% coverage. 100% bugs. Coverage measures execution, not verification.

Hiring signal: Engineers who can explain why 92% code coverage doesn't mean 92% tested demonstrate understanding that coverage measures execution, not verification.

What you will learn

Explain why code coverage measures execution, not verification
Understand the 92% coverage with deduplication bug case study
Define mutation score: killed mutants / total mutants
Identify the 40% untested behavioral paths hidden behind green coverage

The Problem

A team has 92% code coverage on their AI-generated deduplication module. They feel confident. But there's a bug: the function removes duplicates AND sorts the output, changing the order. The tests execute 92% of the lines, but they don't verify that the order is preserved. Coverage is green. The bug is in production.

Code coverage measures which lines were executed. It does NOT measure whether the tests actually check the correct behavior. A test that calls deduplicate([3, 1, 2, 1]) and asserts the result is [1, 2] executes 100% of the function's lines. But it doesn't catch the order-changing bug because the expected output [1, 2] happens to be sorted -- the test author didn't realize the function was sorting in addition to deduplicating.

Coverage vs Mutation Score

Metric	What it measures	What it misses
Code coverage	Which lines were executed	Whether assertions are correct
Branch coverage	Which branches were taken	Whether branch outcomes are verified
Mutation score	What % of code changes are caught by tests	Nothing -- it's the gold standard

Mutation Score Formula

mutation_score = killed_mutants / (total_mutants - equivalent_mutants)

Killed: the test suite catches the mutation (a test fails)
Surviving: the mutation passes all tests (the tests don't catch this change)
Equivalent: the mutation doesn't change behavior (can't be killed)
Timeout: the mutation causes an infinite loop

The 92% coverage / 40% mutation score gap
A function with 92% line coverage can have a 40% mutation score. This means 60% of behavioral changes to the code go undetected by the test suite. The tests execute the lines but don't verify the behavior. For AI-generated code, this gap is critical: AI code has subtle behavioral bugs (wrong order, missing edge cases, incorrect error handling) that pass coverage checks but fail mutation testing. Mutation testing is the only metric that measures what your tests actually TEST, not just what they EXECUTE.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Deduplication Bug Case Study, Build It — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.