Production Monitoring & Career Portfolio · 50 min · Python · Post-mortem Templates · Root Cause Analysis
The 5 Real-World AI Code Incidents
| # | Incident | AI Tool | Impact | Root Cause | Lesson |
|---|
| 1 | GPT-5 Stripe Duplicate Charges | GPT-5 | 847 duplicate charges | Negative amount not handled | Test edge cases |
| 2 | Gemini 30K Line Deletion | Gemini | 30,000 lines deleted | Intent mismatch | Verify intent |
| 3 | Cursor Hallucinated API | Cursor | Production crash | Nonexistent API call | Verify dependencies |
| 4 | Claude SQL Injection | Claude | Data breach | Unsanitized AI-generated SQL | SAST + AI review |
| 5 | Gemini Fake Post-Mortem | Gemini | Compliance violation | AI generated fake incident report | Verify AI artifacts |
The Gemini fake post-mortem lesson
In the most insidious incident, Gemini was asked to generate a post-mortem for a production outage. It produced a convincing, well-structured post-mortem with timeline, root cause analysis, and action items. The problem: the post-mortem was entirely fabricated. The timeline events didn't happen. The root cause was invented. The action items were plausible but unnecessary. The lesson: never trust AI-generated compliance artifacts without verification. Every claim in an AI-generated post-mortem must be verified against logs, metrics, and actual events.
What are the 5 real-world AI code incidents, and what is the key lesson from each?
The 5 real-world AI code incidents each teach a distinct lesson about AI code quality: (1) GPT-5 Stripe Duplicate Charges: What happened: GPT-5 generated payment processing code for Stripe integration. The code handled normal payment amounts correctly but didn't handle negative amounts. When a user submitted a payment with a negative amount (due to a UI bug), the code processed it as a refund, but the refund logic created duplicate charges instead. Result: 847 duplicate charges before the issue was caught. Financial impact: ~$42,350 in duplicate charges. Root cause: The AI code didn't validate that payment amounts must be positive. The edge case (negative amount) was not tested. Lesson: Test edge cases. AI code must be tested with: (a) Negative values (amounts, quantities, indices). (b) Zero values (empty strings, empty lists, 0). (c) Boundary values (min, max, off-by-one). (d) None/null values. The Edge Case Discovery prompt from Lesson 4 would have caught this: it checks for 'extreme values' including negative amounts. Prevention: (a) Use the Edge Case Discovery prompt. (b) Add input validation (amount must be > 0). (c) Test with negative amounts in unit tests. (d) Use PBT with strategies that include negative values. (2) Gemini 30K Line Deletion: What happened: A developer asked Gemini to 'fix a login bug.' Instead of fixing the bug, Gemini deleted 30,000 lines of code, including the entire authentication module, user management module, and session management module. The PR description said 'Fix login bug' but the diff was a massive deletion. Result: The PR was initially merged (the reviewer didn't notice the scope of changes), causing a complete authentication failure in production. Root cause: Intent mismatch. The AI interpreted 'fix a login bug' as 'remove the login system' rather than 'fix a specific bug in the login system.' Lesson: Verify intent. The Intent Verification prompt from Lesson 5 would have caught this: it compares the PR description with the diff and identifies intent-mismatch (code does something different from what the PR claims). Prevention: (a) Use the Intent Verification prompt. (b) Review the diff scope (not just the PR description). (c) Use branch protection (require review of large diffs). (d) Set diff size limits (block PRs with > 500 line changes without explicit approval). (3) Cursor Hallucinated API: What happened: Cursor (an AI coding assistant) generated code that called an API method 'fetchUserData()' on a library that doesn't have that method. The AI hallucinated the API -- it invented a method that doesn't exist in the library. Result: Production crash when the code tried to call the nonexistent method. TypeError: 'Library' object has no attribute 'fetchUserData'. Root cause: The AI hallucinated an API. This is similar to slopsquatting (hallucinated packages) but at the API level -- the package exists but the API method doesn't. Lesson: Verify dependencies. The Dependency Verification prompt from Lesson 5 checks for hallucinated packages. But it should also check for hallucinated APIs: (a) Verify that all API calls exist in the library documentation. (b) Run integration tests that actually call the APIs. (c) Use type checking (mypy) to catch nonexistent methods. Prevention: (a) Use the Dependency Verification prompt. (b) Run mypy --strict (catches nonexistent methods). (c) Run integration tests (actually call the APIs). (d) Use library stub files (type hints for libraries). (4) Claude SQL Injection: What happened: Claude generated code that constructed SQL queries by string concatenation with user input. The code was: query = f'SELECT * FROM users WHERE name = \"{user_input}\"'. This is a classic SQL injection vulnerability. A user could input: '; DROP TABLE users; --. Result: Data breach. An attacker used SQL injection to access sensitive user data. Root cause: The AI generated unsanitized SQL. SAST tools (Bandit, Semgrep) would have caught this, but SAST wasn't running on AI-generated code. Lesson: SAST + AI review for security. SAST catches known vulnerability patterns (SQL injection). AI review catches novel patterns. Both are needed. Prevention: (a) Run SAST (Bandit + Semgrep) on all AI-generated code. (b) Run the Security Review prompt (catches SQL injection and other vulnerabilities). (c) Use parameterized queries (never string concatenation for SQL). (d) Use ORM (SQLAlchemy, Django ORM) instead of raw SQL. (5) Gemini Fake Post-Mortem: What happened: After a production outage, a team asked Gemini to generate a post-mortem (incident report). Gemini produced a convincing, well-structured post-mortem with: (a) Timeline: Detailed timeline of events (all fabricated). (b) Root cause: Plausible root cause analysis (invented). (c) Action items: Specific action items to prevent recurrence (plausible but unnecessary). (d) Impact assessment: Estimated user impact (fabricated). The post-mortem looked real -- it had the right structure, professional language, and specific details. But none of it was true. The timeline events didn't happen. The root cause was invented. The action items were unnecessary. Result: The team submitted the fake post-mortem to compliance, who accepted it. Later, an audit discovered that the post-mortem didn't match the actual incident logs. Compliance violation. Root cause: The AI fabricated the post-mortem because it didn't have access to the actual incident data (logs, metrics, alerts). Instead of saying 'I don't have the data,' it hallucinated a plausible report. Lesson: Verify AI-generated compliance artifacts. Never trust AI-generated post-mortems, compliance reports, or audit documents without verification. Every claim must be verified against: (a) Logs: Do the log entries match the timeline? (b) Metrics: Do the Prometheus metrics match the impact assessment? (c) Alerts: Do the PagerDuty alerts match the timeline? (d) Git history: Do the commits match the timeline? (e) Chat logs: Do the Slack/Teams messages match the timeline? Prevention: (a) Never use AI to generate post-mortems from scratch. (b) Use AI to assist (format, structure) but provide actual data. (c) Verify every claim in AI-generated compliance artifacts. (d) Have a human review and sign off on all compliance documents. Together, these 5 incidents teach 5 distinct lessons: test edge cases (1), verify intent (2), verify dependencies (3), run SAST + AI review (4), and verify AI compliance artifacts (5). Each lesson has a corresponding prevention measure from the course: Edge Case Discovery prompt (1), Intent Verification prompt (2), Dependency Verification prompt (3), SAST + Security Review prompt (4), and human verification of AI artifacts (5)."