Testing Framework & Continuous Execution · 40 min · Python · Claude API · GPT-4 API
The Problem
SAST tools (Bandit, Semgrep) catch known vulnerability patterns. But AI-generated code introduces novel patterns that SAST can't catch: hallucinated APIs, intent mismatches, architecture violations, and duplicate logic. The review prompt library addresses these gaps with structured, reusable prompts for AI code review.
The 6 Review Prompts
| # | Prompt | Purpose | Input | Output | Est. Time |
|---|
| 1 | Security Review | Find vulnerabilities SAST misses | Code, SAST results | Findings with CWE, severity, fix | 90s |
| 2 | Intent Verification | Check code matches PR description | PR description, diff | Intent-mismatch, silent-scope, unbacked-claim findings | 60s |
| 3 | Architecture Review | Check code follows project patterns | Code, architecture doc | Violations with recommendations | 45s |
| 4 | Complexity Assessment | Identify overly complex AI code | Code | Complexity report with refactoring suggestions | 30s |
| 5 | Dependency Verification | Check for hallucinated packages | Imports, requirements | Hallucinated package list with verification | 25s |
| 6 | Duplicate Code Detection | Find copy-paste patterns in AI code | Code | Duplicate blocks with deduplication suggestions | 35s |
Review prompts vs testing prompts
Lesson 4 covered testing prompts (generate tests, find edge cases, mutation analysis). This lesson covers review prompts (review code, verify intent, check architecture). Testing prompts answer: 'Is the code correct?' Review prompts answer: 'Is the code well-designed, secure, and aligned with intent?' Both are needed -- testing prompts verify behavior, review prompts verify quality and alignment.
What are the 6 review prompts, and how do they complement SAST tools?
The 6 review prompts complement SAST tools by catching issues that pattern-matching tools cannot detect. SAST tools (Bandit, Semgrep) work by matching code against known vulnerability patterns (e.g., 'eval() is dangerous', 'SQL string concatenation is dangerous'). This is effective for known patterns but misses novel patterns, design issues, and AI-specific problems. The 6 review prompts address these gaps: (1) Security Review: SAST catches known vulnerability patterns (CWE-89 SQL injection, CWE-79 XSS, CWE-78 OS command injection). But AI-generated code introduces novel vulnerabilities that don't match existing patterns: (a) SSRF (Server-Side Request Forgery): AI code fetches URLs from user input without validation. SAST may not flag this if the URL construction doesn't match known patterns. (b) Business logic flaws: AI code applies a 10% discount to all orders, but the business rule is '10% discount only for orders > $100'. SAST can't understand business logic. (c) Novel injection patterns: AI code uses a new library's API in an unsafe way that SAST doesn't know about. The Security Review prompt asks the AI to review code for vulnerabilities, including patterns that SAST might miss. It produces findings with CWE ID, severity, and fix recommendation. (2) Intent Verification: SAST doesn't know what the PR is supposed to do. It just checks the code for known patterns. But AI-generated code can have intent mismatches: the PR description says 'Fix login bug' but the diff deletes 30,000 lines of code (the Gemini incident). SAST would pass this code (no known vulnerabilities), but the intent is wrong. The Intent Verification prompt compares the PR description with the code diff and identifies: (a) Intent-mismatch: Code does something different from what the PR claims. (b) Silent-scope: Code changes files or functionality not mentioned in the PR. (c) Unbacked-claim: PR claims to fix a bug but doesn't include a test for the bug. (d) Doc-drift: Code changes don't update documentation. (e) Missing-impl: PR claims to add a feature but the implementation is incomplete. (3) Architecture Review: SAST doesn't understand project architecture. It doesn't know that 'all database access should go through the repository layer' or 'all API endpoints should use the auth middleware.' AI-generated code may violate architecture patterns: (a) Direct database access in a controller (should go through repository). (b) API endpoint without auth middleware. (c) Business logic in a data model. (d) Circular dependencies between modules. The Architecture Review prompt checks code against the project's architecture document and identifies violations with recommendations. (4) Complexity Assessment: SAST doesn't assess code complexity. AI-generated code can be overly complex: (a) Deeply nested conditionals (5+ levels of if/else). (b) Long functions (200+ lines). (c) High cyclomatic complexity (> 15). (d) God classes (classes with too many responsibilities). (e) Long parameter lists (7+ parameters). The Complexity Assessment prompt identifies overly complex code and suggests refactoring (extract method, extract class, simplify conditional). (5) Dependency Verification: SAST doesn't check if packages actually exist. AI models can hallucinate package names (slopsquatting): (a) AI suggests 'import numpyy' (hallucinated, attacker may have registered it). (b) AI suggests 'import request' (hallucinated, real package is 'requests'). (c) AI suggests 'import python-dateutil' (real but different from 'dateutil'). The Dependency Verification prompt checks all imports against package registries (PyPI, npm) and flags hallucinated packages. (6) Duplicate Code Detection: SAST doesn't detect duplicate code. AI models frequently copy-paste logic: (a) Same 20-line block repeated in 3 functions. (b) Similar error handling code in 5 endpoints. (c) Copy-pasted validation logic with one variable changed. The Duplicate Code Detection prompt identifies duplicate blocks and suggests deduplication (extract shared function, use a helper). Together, these 6 review prompts catch issues that SAST cannot: novel vulnerabilities (1), intent mismatches (2), architecture violations (3), complexity (4), hallucinated packages (5), and duplicate code (6). They complement SAST -- SAST catches known patterns, review prompts catch novel and AI-specific issues. In the CI pipeline, both run: SAST first (fast, known patterns), then AI review prompts (slower, novel patterns). If either finds critical issues, the merge is blocked."