Phase 0: The AI Code Quality Crisis · 50 min · Python · pytest

The Defect Landscape in Statistics

If you cannot cite the numbers, you cannot argue for the budget.

Hiring signal: Engineers who can cite specific statistics (55.8% vulnerability rate, 97.8% detection gap, CWE-1236 at 50.7%) demonstrate they have read the research and can make a data-driven case for AI code testing investment to engineering leadership.

What you will learn

Cite the 55.8% Z3-proven vulnerability rate and explain its methodology
Identify the top CWE categories in AI-generated code (CWE-1236, CWE-916, CWE-20, CWE-117)
Explain the 97.8% detection gap between Z3 formal verification and industry tools
Compare vulnerability rates across programming languages (Python vs Java vs C++)

The Problem

A startup CTO in early 2026 dismissed the need for a dedicated AI code testing pipeline. "Our AI assistant writes good code," he said. "We have SonarQube and CodeQL in CI. We're covered." Six months later, a Z3 formal verification scan of their AI-generated codebase found that 52% of their AI-generated functions contained provable vulnerabilities -- vulnerabilities that SonarQube and CodeQL had both reported as clean.

The CTO's mistake was not in trusting his tools. His mistake was in not knowing the research. The 2026 "Broken by Default" study (arXiv 2604.05292) formally verified 3,500 AI-generated code artifacts from 7 different models using Z3 SMT solvers and found a 55.8% vulnerability rate. Critically, it also tested 6 industry-standard vulnerability detection tools against the same artifacts and found that combined, they detected only 7.6% of the Z3-proven vulnerabilities -- a 97.8% detection gap.

If you are building a testing strategy for AI-generated code, these numbers are your starting point. Not opinions. Not vibes. Peer-reviewed, reproducible numbers.

The 55.8% Number and What It Means

The "Broken by Default" study used Z3 SMT solvers to formally verify AI-generated code artifacts. Z3 does not pattern-match or heuristic-scan -- it mathematically proves whether a code path can be exploited. Here is what the study found:

Metric	Value	What it means
Overall vulnerability rate	55.8%	More than half of AI-generated code artifacts contain at least one Z3-proven vulnerability
CWE-1236 (improper auth)	50.7%	Half of all vulnerable artifacts have improper authentication handling
CWE-916 (password hash)	43.1%	43% use insecure password hashing practices
CWE-20 (input validation)	38.2%	Over a third lack proper input validation
CWE-117 (logging injection)	29.4%	Nearly a third have log injection vulnerabilities
OWASP Top 10 coverage	45%	45% of AI code introduces at least one OWASP Top 10 pattern

These are not theoretical vulnerabilities
Z3 SMT solvers do not flag "suspicious patterns" or "code smells." They mathematically prove that a specific input exists that will cause the code to behave incorrectly. When Z3 says a function has a CWE-1236 vulnerability, it means there is a concrete input that bypasses the authentication check. These are not false positives or heuristic warnings -- they are proven exploit paths.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The 97.8% Detection Gap, Language-Specific Vulnerability Patterns, The Debt Behind the AI Boom, Build It — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.