Testing Framework & Continuous Execution · 40 min · Python · YAML · GitHub Actions
The 8 Checklist Categories
| # | Category | Items | Phase | Gate |
|---|
| 1 | Testing | 12 | 1, 3, 4 | Unit, PBT, Mutation |
| 2 | Static Analysis | 6 | 2 | Lint, Type, SAST |
| 3 | Security | 8 | 5, 6 | SAST, SCA, AI Review |
| 4 | Supply Chain | 6 | 6 | Allowlist, Lockfile, SBOM |
| 5 | CI/CD | 7 | 7 | Pipeline, Caching, Gates |
| 6 | Documentation | 5 | 8 | Docstrings, README, Changelog |
| 7 | Performance | 4 | 8 | Benchmarks, Profiling |
| 8 | Compliance | 4 | 8 | License, Privacy, Audit |
The master checklist as a CI gate
The master checklist is not just a document -- it's an automated CI gate. Each item is checked automatically by the verification framework. If any REQUIRED item fails, the merge is blocked. OPTIONAL items generate warnings but don't block. The checklist runs as part of the verify command: verify --config=verify.yml --checklist=master_checklist.yml. This ensures no AI code is merged without passing all required checklist items.
Why is a master checklist important for AI-generated code specifically?
A master checklist is particularly important for AI-generated code for 6 reasons: (1) AI code has a 55.8% vulnerability rate: Studies show 55.8% of AI-generated code contains Z3-proven vulnerabilities, compared to ~15% for human-written code. This means AI code needs more security checks than human code. The checklist includes 8 security items (SAST, SCA, AI review, allowlist, lockfile, SBOM, secrets scan, dependency audit) to catch these vulnerabilities. Without a checklist, engineers might run SAST but forget SCA, or run SCA but forget AI review. The checklist ensures all 8 security items are checked. (2) AI code has a 92% coverage with bugs problem: Phase 4 showed that 92% code coverage can coexist with behavioral bugs (the deduplication bug). Coverage alone doesn't guarantee code quality. The checklist includes mutation testing (mutation score >= 80%) to verify that tests actually catch bugs, not just cover lines. Without a checklist, engineers might check coverage (92%) and think the code is well-tested, missing the mutation score check. (3) AI code has supply chain risks (slopsquatting): AI models can hallucinate package names, leading to slopsquatting attacks. The checklist includes supply chain items (allowlist check, lockfile with hashes, SBOM generation, dependency audit) to prevent these attacks. Without a checklist, engineers might install dependencies without checking the allowlist or generating a lockfile. (4) AI code often lacks documentation: AI models generate code but often skip docstrings, README updates, and changelog entries. The checklist includes documentation items (docstrings on all functions, README updated, changelog entry, API documentation, inline comments for complex logic) to ensure documentation is complete. Without a checklist, AI code can be merged without any documentation, making it hard to maintain. (5) AI code can have intent mismatches: The Gemini 30K-line deletion incident showed that AI can do something very different from what was requested. The checklist includes a PR description vs diff check (does the diff match what the PR claims to do?) to catch intent mismatches. Without a checklist, intent mismatches can be merged, causing catastrophic issues. (6) AI code has repetitive logic errors: AI models frequently copy-paste logic, leading to repetitive logic errors (e.g., handling type A correctly but type B incorrectly because the logic was copied without updating the variable). The checklist includes mutation analysis targeting AI-specific patterns (variable assignment errors, missing edge case handling, repetitive logic errors, missing exception handling) to catch these errors. Without a checklist, these AI-specific patterns go unchecked. In summary, the master checklist is important for AI-generated code because it systematically addresses the specific risks of AI code: high vulnerability rate, coverage-with-bugs, supply chain attacks, missing documentation, intent mismatches, and repetitive logic errors. Each of these risks has a corresponding checklist item that prevents it from reaching production."