Phase 7: AI Governance, Compliance & Risk · 55 min · MITRE ATLAS · NIST AI RMF · EU AI Act Article 73

AI Incident Response & Regulatory Reporting

A prompt injection at 2am is a security problem. A prompt injection your legal team finds out about from a regulator is a career problem.

Hiring signal: The EU AI Act requires providers of high-risk AI systems to report serious incidents to market surveillance authorities within 15 days (2 days for widespread infringements or serious harm), and OpenAI, Anthropic, Microsoft, and Google all publish named incident response processes for model and product safety events. Engineers who can run detection through post-mortem on an AI-specific incident -- and who know which findings trigger a regulatory clock -- are the ones trusted to lead the response instead of just patch the bug.

What you will learn

Design an AI incident response playbook covering detection, triage, containment, eradication, recovery, and post-mortem phases, adapted for prompt injection, data leakage, and model misbehavior scenarios
Conduct a forensic post-mortem that reconstructs an AI incident's timeline, root cause, and blast radius from logs, prompts, and tool-call traces
Determine when an AI incident triggers a regulatory reporting obligation under the EU AI Act's Article 73 serious-incident regime, and assemble the documentation that reporting requires
Run a pre-launch AI security review that would have caught the incident before it shipped, and fold the finding back into that review process

The Problem

In June 2025, security researchers disclosed "EchoLeak" (CVE-2025-32711), a zero-click prompt injection vulnerability in Microsoft 365 Copilot: a malicious email, never opened by the user, could plant instructions that Copilot's retrieval pipeline picked up and acted on, exfiltrating sensitive data without any user interaction at all. Microsoft patched it server-side before public disclosure. The interesting part for this lesson isn't the vulnerability itself — indirect prompt injection is well understood by now — it's what has to happen organizationally the moment a vulnerability like this is confirmed live in production: someone has to detect it, someone has to figure out what was actually exposed and to whom, someone has to contain it without breaking the product for everyone else, someone has to figure out why it happened, and — increasingly — someone has to determine whether a regulator needs to be told, and on what clock.

Most AI security curricula stop at "here's how the attack works" and "here's the defense." This lesson covers the part that starts after an attack succeeds anyway, because one will, eventually, no matter how good the defenses are: incident response. AI incidents have the same lifecycle as any other security incident — detect, triage, contain, eradicate, recover, learn — but with AI-specific wrinkles at almost every stage. Detection is harder because model behavior is probabilistic and a single bad output can look like noise. Triage requires reconstructing what a model "decided" from a context window and tool-call trace, not from a stack trace. And as of the EU AI Act, some AI incidents now carry a hard regulatory reporting clock that starts ticking the moment your team becomes aware of the problem — whether or not you've finished figuring out what happened.

The AI Incident Response Playbook

The five phases below are the same ones used in general security incident response (they map closely to the NIST Computer Security Incident Handling Guide, SP 800-61), adapted for what's different about AI systems.

Phase	General security version	AI-specific wrinkle
Detection	Alerting on known signatures, anomalous traffic	Behavior is probabilistic; a single bad output is often indistinguishable from a model just being wrong. Detection needs output-level monitoring (moderation classifiers, guardrail rejection rates, anomalous tool-call patterns), not just infrastructure metrics
Triage	Determine scope: which systems, which users, how bad	Requires reconstructing what the model was shown and what it decided to do — the prompt, retrieved context, and tool calls, not just server logs. Without full request/response/tool-call logging, triage on an AI incident is close to impossible
Containment	Isolate the affected system, block the attack vector	Often means disabling a specific tool, guardrail, or capability rather than taking a whole service down — e.g. disable a refund tool while leaving the rest of a support agent running
Eradication	Patch the vulnerability, remove the threat	May require prompt template changes, retraining, guardrail rule updates, or ingestion-pipeline sanitization — fixes that need their own testing (a regression suite), since prompt changes can silently break unrelated behavior
Recovery	Restore normal operation, monitor for recurrence	Re-enable capability behind a tighter control than existed before (a new approval threshold, a new guardrail rule) — recovery to the exact prior state usually means the same incident recurs

The step teams skip most often is treating post-mortem as a sixth phase rather than an afterthought. A post-mortem that only documents what was patched, without asking what allowed it to ship in the first place, guarantees a structurally similar incident under a different feature name. That connective tissue — this incident to the review process that should have caught it — is what makes an incident response program improve over time instead of just repeatedly reacting.

Detection is a monitoring design problem, not an afterthought
An AI system with no output-level monitoring cannot detect most AI-specific incidents at all, no matter how good the response process is on paper. Infrastructure metrics (latency, error rate, CPU) stay green while a model quietly leaks data, gets manipulated into an unauthorized action, or produces increasingly unsafe outputs. Real detection requires logging full prompts, retrieved context, and tool calls; running lightweight classifiers over outputs for policy violations; and alerting on guardrail rejection-rate spikes and anomalous tool-call patterns (a support agent suddenly calling a refund tool at 10x its normal rate). If detection isn't designed in at launch, the first sign of an incident is often a customer complaint or a regulator's letter — both of which start your reporting clock later than it should have started, and look worse in the post-mortem.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Forensic Analysis and the Post-Mortem Structure, Regulatory Reporting Requirements, The AI Security Review Process for Feature Launches, Build It, What to Practice — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.