Phase 7: AI Governance, Compliance & Risk · 55 min · MITRE ATLAS · NIST AI RMF · EU AI Act Article 73
The Problem
In June 2025, security researchers disclosed "EchoLeak" (CVE-2025-32711), a zero-click prompt injection vulnerability in Microsoft 365 Copilot: a malicious email, never opened by the user, could plant instructions that Copilot's retrieval pipeline picked up and acted on, exfiltrating sensitive data without any user interaction at all. Microsoft patched it server-side before public disclosure. The interesting part for this lesson isn't the vulnerability itself — indirect prompt injection is well understood by now — it's what has to happen organizationally the moment a vulnerability like this is confirmed live in production: someone has to detect it, someone has to figure out what was actually exposed and to whom, someone has to contain it without breaking the product for everyone else, someone has to figure out why it happened, and — increasingly — someone has to determine whether a regulator needs to be told, and on what clock.
Most AI security curricula stop at "here's how the attack works" and "here's the defense." This lesson covers the part that starts after an attack succeeds anyway, because one will, eventually, no matter how good the defenses are: incident response. AI incidents have the same lifecycle as any other security incident — detect, triage, contain, eradicate, recover, learn — but with AI-specific wrinkles at almost every stage. Detection is harder because model behavior is probabilistic and a single bad output can look like noise. Triage requires reconstructing what a model "decided" from a context window and tool-call trace, not from a stack trace. And as of the EU AI Act, some AI incidents now carry a hard regulatory reporting clock that starts ticking the moment your team becomes aware of the problem — whether or not you've finished figuring out what happened.
The AI incident response playbook
The five phases below are the same ones used in general security incident response (they map closely to the NIST Computer Security Incident Handling Guide, SP 800-61), adapted for what's different about AI systems.
| Phase | General security version | AI-specific wrinkle |
|---|
| Detection | Alerting on known signatures, anomalous traffic | Behavior is probabilistic; a single bad output is often indistinguishable from a model just being wrong. Detection needs output-level monitoring (moderation classifiers, guardrail rejection rates, anomalous tool-call patterns), not just infrastructure metrics |
| Triage | Determine scope: which systems, which users, how bad | Requires reconstructing what the model was shown and what it decided to do — the prompt, retrieved context, and tool calls, not just server logs. Without full request/response/tool-call logging, triage on an AI incident is close to impossible |
| Containment | Isolate the affected system, block the attack vector | Often means disabling a specific tool, guardrail, or capability rather than taking a whole service down — e.g. disable a refund tool while leaving the rest of a support agent running |
| Eradication | Patch the vulnerability, remove the threat | May require prompt template changes, retraining, guardrail rule updates, or ingestion-pipeline sanitization — fixes that need their own testing (a regression suite), since prompt changes can silently break unrelated behavior |
| Recovery | Restore normal operation, monitor for recurrence | Re-enable capability behind a tighter control than existed before (a new approval threshold, a new guardrail rule) — recovery to the exact prior state usually means the same incident recurs |
The step teams skip most often is treating post-mortem as a sixth phase rather than an afterthought. A post-mortem that only documents what was patched, without asking what allowed it to ship in the first place, guarantees a structurally similar incident under a different feature name. That connective tissue — this incident to the review process that should have caught it — is what makes an incident response program improve over time instead of just repeatedly reacting.
Detection is a monitoring design problem, not an afterthought
An AI system with no output-level monitoring cannot detect most AI-specific incidents at all, no matter how good the response process is on paper. Infrastructure metrics (latency, error rate, CPU) stay green while a model quietly leaks data, gets manipulated into an unauthorized action, or produces increasingly unsafe outputs. Real detection requires logging full prompts, retrieved context, and tool calls; running lightweight classifiers over outputs for policy violations; and alerting on guardrail rejection-rate spikes and anomalous tool-call patterns (a support agent suddenly calling a refund tool at 10x its normal rate). If detection isn't designed in at launch, the first sign of an incident is often a customer complaint or a regulator's letter — both of which start your reporting clock later than it should have started, and look worse in the post-mortem.
Forensic analysis and the post-mortem structure
Reconstructing what actually happened in an AI incident is different from a traditional post-mortem because the "decision" under investigation was made by a model, not by a line of deterministic code. A rigorous AI incident post-mortem needs to answer, in order:
- Timeline — what happened, in what order, established from logs (not memory or assumption). For an AI incident this specifically means: what was the full prompt sent to the model (including system prompt and any retrieved/injected content), what did the model output, what tool calls did it make and with what arguments, and what did those tool calls actually do.
- Root cause — the specific mechanism that allowed the bad outcome. "The model was manipulated" is not a root cause; "an indirect prompt injection in a retrieved document was not structurally separated from trusted instructions, so the model treated it as a command" is.
- Blast radius — how many users, systems, and how much data were actually affected — not how much could have been affected in the worst case, but the specific measured scope, established by querying logs across the full window the vulnerability was live, not just the reported case.
- Contributing factors — the structural gaps that let the root cause matter (missing approval gate, missing regression test, missing defense-in-depth check) — this is the list that feeds back into the pre-launch review, and it's usually more valuable than the root cause itself for preventing recurrence.
- Remediation — both the immediate patch and, separately, the structural fix (a new CI gate, a new approval threshold, a new item on the launch checklist).
A post-mortem that stops at step 2 treats the incident as a one-off bug. A post-mortem that reaches steps 4 and 5 treats it as a finding about the system that produced the bug — which is the difference between an incident response program that gets better over time and one that just keeps reacting to the same class of problem with a new name each time.
A team patches the specific prompt injection payload that caused an incident and closes the post-mortem. Three months later, a structurally similar injection succeeds against a different feature built by a different team. What does this most likely indicate was missing from the first post-mortem?
A post-mortem that documents only the specific patch (e.g. "we now strip hidden text from this one ingestion pipeline") fixes an instance, not a class of vulnerability. The contributing-factors analysis is what should generalize: "there was no structural separation between trusted instructions and untrusted retrieved content" and "there was no approval gate on high-blast-radius tool calls" are findings that apply to any team building a similar feature. Feeding those findings into a company-wide pre-launch security review (or a shared guardrail library, or a shared regression suite) is what prevents the second team from shipping the same underlying gap under a different feature name. Without that step, incident response stays reactive no matter how good any single post-mortem is.
Regulatory reporting requirements
Incident response for AI systems increasingly has a legal clock attached, and getting the reporting determination wrong — in either direction — has real consequences. The most significant new obligation is the EU AI Act's Article 73 serious-incident reporting regime:
- Who it applies to: providers of AI systems classified high-risk under Annex III (employment, credit, essential public/private services, law enforcement, migration, justice administration, and several other categories) — not every AI system, and not GPAI models by default.
- What triggers it: a "serious incident" as defined in Article 3(49) — the death of a person or serious harm to a person's health caused by the AI system, a serious and irreversible disruption of critical infrastructure, an infringement of obligations intended to protect fundamental rights, or serious harm to property or the environment.
- The clock: report to the market surveillance authority of the relevant Member State without undue delay and no later than 15 days after becoming aware of the incident. That window compresses to 2 days for a widespread infringement or for a serious incident involving death or serious harm to health. The clock starts at awareness, not at the incident's occurrence — which makes fast, accurate triage a regulatory requirement, not just an operational nicety.
- What has to be in the report: a description of the incident, the systems and territories affected, an initial assessment of likely cause and possible remediation, and corrective actions taken or planned.
The EU AI Act's regime doesn't replace other reporting obligations that can apply to the same incident — a bias-audit failure or discriminatory-outcome incident can separately trigger obligations under employment or credit law (e.g. NYC Local Law 144's audit requirements, covered in the previous lesson); a data exposure can trigger GDPR Article 33/34 breach notification (72 hours to the supervisory authority) or US state breach notification statutes; a publicly traded company may have SEC materiality disclosure obligations if the incident is financially significant. A mature incident response process screens against all of these independently rather than assuming one regulatory framework covers the case. NIST's AI RMF doesn't impose a legal reporting clock, but its Manage function expects organizations to track and respond to AI incidents as an ongoing risk management practice, and ISO/IEC 42001-certified organizations are expected to maintain an incident register as part of their AI management system regardless of whether any external report is triggered.
None of this is a substitute for legal advice, and the regulatory text itself — artificialintelligenceact.eu — and counsel are the authoritative sources on any specific determination. What incident responders can and should do is build a fast, accurate screening step into triage, so the question "does this need to go to legal today" gets asked and answered early, rather than three weeks into eradication when part of the reporting window has already passed by default.
The AI security review process for feature launches
The most leveraged place to apply everything an incident teaches you is before the next feature ships, not after the next incident. A pre-launch AI security review is a structured gate a feature has to pass before it goes to production — the direct answer to "how do we stop shipping the same category of incident under a new name." At minimum, a review should require a documented answer to:
- Has this feature been threat-modeled against the OWASP LLM Top 10 and MITRE ATLAS tactics relevant to its architecture (RAG, tool use, agents)?
- Have prompt injection and jailbreak test suites been run against the exact system prompt and tool set shipping?
- Is there a guardrail or output filter between model output and any consumer of that output that has write/execute/financial capability?
- Is every tool call the model can make scoped to least privilege, with an approval gate for irreversible or high-blast-radius actions?
- Is there logging sufficient to reconstruct a full incident timeline after the fact?
- Has legal/compliance confirmed whether this feature's risk classification changes under the EU AI Act, sector regulation, or company policy?
- Is there a rollback/kill-switch path that doesn't require a full deploy?
Every "no" answer on this list is a specific, nameable gap that a future incident will eventually find. The review process only compounds in value if post-mortems feed new questions back into it — a review checklist that never changes is a review checklist that stopped learning from the incidents it was supposed to prevent.
Build
Build it
The code artifact lesson_07_05_incident_report.py is a structured AI incident report generator — it takes the basic facts of an incident (type, severity, a timeline of events, affected systems and users) and produces a full post-mortem document covering detection through post-mortem and regulatory reporting screening, plus a retroactive pre-launch security review. It's standard-library only, deliberately: incident response tooling has to run reliably during an actual incident, including from a locked-down environment, and shouldn't depend on whatever happens to be pip install-able that day.
IncidentReport — a dataclass holding the full incident record (type, severity, timeline, root cause, blast radius, contributing factors, remediation actions) with methods that render each report section as markdowntriggers_eu_ai_act_reporting() and reporting_deadline() — a conservative Article 73 screening heuristic based on severity, incident type, and exposure, explicitly documented as a screening aid rather than a legal determinationbuild_example_incident() — two fully worked example incidents (an indirect prompt injection causing an unauthorized refund tool call, and a RAG retrieval scope bug leaking cross-tenant documents), each with a realistic multi-event timeline, root cause, blast radius, and remediation planrender_pre_launch_review() — applies the seven-question pre-launch security review retroactively to the incident, showing which "no" answer would have let it happen
"""
lesson_07_05_incident_report.py
A structured AI incident report generator: given the basic facts of an AI
security incident (type, severity, timeline events, affected systems), this
produces a full post-mortem document covering detection, triage, containment,
eradication, recovery, root-cause analysis, and a regulatory reporting
checklist -- the same shape as the internal incident reports used at
companies with mature AI safety/security programs (OpenAI, Anthropic,
Microsoft all publish variants of this structure).
This has no required external dependencies -- it is standard-library only,
because incident report tooling has to run reliably during an actual
incident, not depend on whatever happens to be `pip install`-able that day.
Reference: NIST AI RMF (https://www.nist.gov/itl/ai-risk-management-framework),
EU AI Act Article 73 serious-incident reporting
(https://artificialintelligenceact.eu/), MITRE ATLAS (https://atlas.mitre.org/).
Usage:
python lesson_07_05_incident_report.py
python lesson_07_05_incident_report.py --incident-type data_leakage --severity critical
"""
from __future__ import annotations
import argparse
import json
from dataclasses import dataclass, field
from datetime import datetime, timedelta, timezone
from enum import Enum
class Severity(str, Enum):
LOW = "low"
MEDIUM = "medium"
HIGH = "high"
CRITICAL = "critical"
class IncidentType(str, Enum):
PROMPT_INJECTION = "prompt_injection"
DATA_LEAKAGE = "data_leakage"
MODEL_MISBEHAVIOR = "model_misbehavior"
EXCESSIVE_AGENCY = "excessive_agency"
MODEL_EXTRACTION = "model_extraction"
# ── EU AI Act Article 73 reporting-clock reference ───────────────────────────
# Article 73 requires providers of high-risk AI systems to report a "serious
# incident" to the relevant market surveillance authority without undue delay
# and, in any case, no later than 15 days after becoming aware of it. That
# window shortens to 2 days for a widespread infringement or a serious
# incident as defined in Art. 3(49)(a) (death or serious harm to health), and
# to immediately (with the report following without undue delay) for a
# widespread infringement causing serious harm. This lookup is a teaching
# reference, not legal advice -- always confirm current obligations with
# counsel and the text at https://artificialintelligenceact.eu/.
REPORTING_CLOCK_DAYS = {
"serious_harm_to_health_or_death": 2,
"widespread_infringement": 2,
"critical_infrastructure_disruption": 2,
"standard_serious_incident": 15,
}
@dataclass
class TimelineEvent:
timestamp: datetime
phase: str # detection | triage | containment | eradication | recovery
description: str
actor: str # who/what performed or observed this
@dataclass
class IncidentReport:
incident_id: str
title: str
incident_type: IncidentType
severity: Severity
detected_at: datetime
affected_systems: list[str]
affected_users_estimate: int
timeline: list[TimelineEvent] = field(default_factory=list)
root_cause: str = ""
blast_radius: str = ""
data_exposed: list[str] = field(default_factory=list)
remediation_actions: list[str] = field(default_factory=list)
contributing_factors: list[str] = field(default_factory=list)
# ── Regulatory triggers ──────────────────────────────────────────────────
def triggers_eu_ai_act_reporting(self) -> bool:
"""
A conservative trigger check for whether this incident would need
evaluation under EU AI Act Article 73 serious-incident reporting.
This flags for review; it does not make the legal determination.
Serious incidents under Art. 3(49) include: death or serious harm to
health, serious and irreversible disruption of critical
infrastructure, infringement of fundamental rights obligations, or
serious harm to property/environment. Only high-risk AI systems (per
Annex III, e.g. employment, credit, law enforcement, essential
services) fall under this specific obligation -- confirm system
classification before treating this as a real determination.
"""
high_severity = self.severity in (Severity.HIGH, Severity.CRITICAL)
rights_impacting = self.incident_type in (
IncidentType.DATA_LEAKAGE,
IncidentType.EXCESSIVE_AGENCY,
IncidentType.MODEL_MISBEHAVIOR,
)
material_exposure = self.affected_users_estimate >= 1 or bool(self.data_exposed)
return high_severity and rights_impacting and material_exposure
def reporting_deadline(self, reference_time: datetime | None = None) -> tuple[str, datetime]:
"""Returns (category, deadline) using the conservative 2-day category
whenever severity is CRITICAL, else the standard 15-day window."""
reference_time = reference_time or self.detected_at
if self.severity == Severity.CRITICAL:
category = "serious_harm_to_health_or_death"
else:
category = "standard_serious_incident"
days = REPORTING_CLOCK_DAYS[category]
return category, reference_time + timedelta(days=days)
# ── Report sections ──────────────────────────────────────────────────────
def render_detection(self) -> str:
detection_events = [e for e in self.timeline if e.phase == "detection"]
lines = [
"## 1. Detection",
f"- **Detected at:** {self.detected_at.isoformat()}",
f"- **Incident type:** {self.incident_type.value}",
f"- **Severity:** {self.severity.value.upper()}",
]
for e in detection_events:
lines.append(f"- `{e.timestamp.isoformat()}` ({e.actor}): {e.description}")
if not detection_events:
lines.append("- No detection-phase events logged.")
return "\n".join(lines)
def render_triage(self) -> str:
triage_events = [e for e in self.timeline if e.phase == "triage"]
lines = [
"## 2. Triage",
f"- **Affected systems:** {', '.join(self.affected_systems) or 'none identified'}",
f"- **Affected users (estimate):** {self.affected_users_estimate}",
f"- **Data potentially exposed:** {', '.join(self.data_exposed) or 'none identified'}",
]
for e in triage_events:
lines.append(f"- `{e.timestamp.isoformat()}` ({e.actor}): {e.description}")
return "\n".join(lines)
def render_containment(self) -> str:
events = [e for e in self.timeline if e.phase == "containment"]
lines = ["## 3. Containment"]
if events:
for e in events:
lines.append(f"- `{e.timestamp.isoformat()}` ({e.actor}): {e.description}")
else:
lines.append("- No containment actions logged yet.")
return "\n".join(lines)
def render_eradication(self) -> str:
events = [e for e in self.timeline if e.phase == "eradication"]
lines = ["## 4. Eradication"]
if events:
for e in events:
lines.append(f"- `{e.timestamp.isoformat()}` ({e.actor}): {e.description}")
else:
lines.append("- No eradication actions logged yet.")
return "\n".join(lines)
def render_recovery(self) -> str:
events = [e for e in self.timeline if e.phase == "recovery"]
lines = ["## 5. Recovery"]
if events:
for e in events:
lines.append(f"- `{e.timestamp.isoformat()}` ({e.actor}): {e.description}")
else:
lines.append("- No recovery actions logged yet.")
return "\n".join(lines)
def render_post_mortem(self) -> str:
lines = [
"## 6. Post-Mortem",
"",
"### Root Cause",
self.root_cause or "_Not yet determined._",
"",
"### Blast Radius",
self.blast_radius or "_Not yet determined._",
"",
"### Contributing Factors",
]
if self.contributing_factors:
lines += [f"- {f}" for f in self.contributing_factors]
else:
lines.append("_None identified._")
lines += ["", "### Remediation Actions"]
if self.remediation_actions:
lines += [f"- [ ] {a}" for a in self.remediation_actions]
else:
lines.append("_None recorded._")
lines += [
"",
"### Timeline (full)",
"",
"| Timestamp | Phase | Actor | Description |",
"|---|---|---|---|",
]
for e in sorted(self.timeline, key=lambda ev: ev.timestamp):
lines.append(f"| {e.timestamp.isoformat()} | {e.phase} | {e.actor} | {e.description} |")
return "\n".join(lines)
def render_regulatory_checklist(self) -> str:
triggers = self.triggers_eu_ai_act_reporting()
lines = [
"## 7. Regulatory Reporting Checklist",
"",
"**This is a screening aid, not a legal determination. Route to "
"legal/compliance before treating any of this as final.**",
"",
f"- **EU AI Act Article 73 screening result:** "
f"{'FLAGGED FOR LEGAL REVIEW' if triggers else 'not flagged by automated screen'}",
]
if triggers:
category, deadline = self.reporting_deadline()
lines += [
f"- **Applicable reporting category (tentative):** `{category}` "
f"({REPORTING_CLOCK_DAYS[category]}-day clock)",
f"- **Reporting deadline if confirmed:** {deadline.isoformat()} "
f"(clock starts from provider's awareness of the incident, "
f"not from the incident's occurrence)",
"- [ ] Confirm the AI system is classified high-risk under Annex III",
"- [ ] Confirm the incident meets the Art. 3(49) 'serious incident' definition",
"- [ ] Identify the market surveillance authority of the relevant Member State(s)",
"- [ ] Prepare incident report per Article 73 content requirements "
"(description, systems affected, likely cause, corrective actions)",
"- [ ] Notify affected deployers/users per the provider's contractual "
"and Article 26 deployer-cooperation obligations",
"- [ ] Legal/compliance sign-off before external submission",
]
else:
lines += [
"- [ ] Confirm system risk classification (high-risk systems under "
"Annex III carry Article 73 obligations regardless of automated screen result)",
"- [ ] Re-screen if severity, user impact, or data exposure estimates change",
]
lines += [
"",
"- [ ] Check sector-specific obligations independently of the EU AI Act "
"(e.g. GDPR Article 33/34 breach notification if personal data was "
"exposed, US state breach notification laws, SEC materiality "
"disclosure for public companies, HIPAA if health data is involved)",
"- [ ] Log this incident in the internal AI incident register regardless "
"of external reporting outcome (required under ISO/IEC 42001 clause 10.2 "
"and expected under NIST AI RMF's Manage function)",
]
return "\n".join(lines)
def render_full_report(self) -> str:
header = [
f"# AI Incident Report: {self.title}",
"",
f"**Incident ID:** {self.incident_id} ",
f"**Type:** {self.incident_type.value} ",
f"**Severity:** {self.severity.value.upper()} ",
f"**Status as of report generation:** post-mortem draft",
"",
]
sections = [
self.render_detection(),
self.render_triage(),
self.render_containment(),
self.render_eradication(),
self.render_recovery(),
self.render_post_mortem(),
self.render_regulatory_checklist(),
]
return "\n".join(header) + "\n\n" + "\n\n".join(sections) + "\n"
def to_dict(self) -> dict:
return {
"incident_id": self.incident_id,
"title": self.title,
"incident_type": self.incident_type.value,
"severity": self.severity.value,
"detected_at": self.detected_at.isoformat(),
"affected_systems": self.affected_systems,
"affected_users_estimate": self.affected_users_estimate,
"data_exposed": self.data_exposed,
"root_cause": self.root_cause,
"blast_radius": self.blast_radius,
"contributing_factors": self.contributing_factors,
"remediation_actions": self.remediation_actions,
"triggers_eu_ai_act_reporting": self.triggers_eu_ai_act_reporting(),
"timeline": [
{
"timestamp": e.timestamp.isoformat(),
"phase": e.phase,
"actor": e.actor,
"description": e.description,
}
for e in self.timeline
],
}
# ── Pre-launch AI security review checklist ──────────────────────────────────
# A minimal version of the review a feature launch should pass before it
# ships, framed as the questions that -- had they been asked -- would have
# caught the example incident below. This is intentionally short; a real
# review process is a structured gate (see phase c3-08 for the full
# production security architecture), not a substitute for one.
PRE_LAUNCH_SECURITY_REVIEW_CHECKLIST = [
"Has this feature been threat-modeled against the OWASP LLM Top 10 and "
"MITRE ATLAS tactics relevant to its architecture (RAG, tool use, agents)?",
"Have prompt injection and jailbreak test suites (e.g. garak, promptfoo "
"red-team) been run against the exact system prompt and tool set shipping?",
"Is there a guardrail or output filter between model output and any "
"consumer of that output that has write/execute/financial capability?",
"Is every tool call the model can make scoped to least privilege, and is "
"there an approval gate for irreversible or high-blast-radius actions?",
"Is there logging sufficient to reconstruct a full incident timeline "
"after the fact (prompts, retrieved context, tool calls, outputs)?",
"Has legal/compliance confirmed whether this feature's risk classification "
"changes under the EU AI Act, sector regulation, or company policy?",
"Is there a rollback/kill-switch path that does not require a full deploy?",
]
def build_example_incident(incident_type: IncidentType, severity: Severity) -> IncidentReport:
"""Builds one of a small library of realistic example incidents so the
report generator is runnable standalone with representative data."""
base_time = datetime(2026, 3, 14, 9, 12, tzinfo=timezone.utc)
if incident_type == IncidentType.PROMPT_INJECTION:
report = IncidentReport(
incident_id="INC-2026-0031",
title="Indirect prompt injection via ingested support ticket causes unauthorized refund tool call",
incident_type=incident_type,
severity=severity,
detected_at=base_time,
affected_systems=["support-agent-v3", "refund-tool", "ticket-ingestion-pipeline"],
affected_users_estimate=1,
data_exposed=[],
root_cause=(
"A customer-submitted support ticket contained hidden instructions "
"(white text on white background, rendered but not visually "
"displayed) that were ingested verbatim into the agent's context "
"window during RAG retrieval. The agent's system prompt did not "
"structurally separate retrieved ticket content from trusted "
"instructions, so the injected text was treated as an operator "
"command and triggered the refund tool with no human approval gate."
),
blast_radius=(
"One unauthorized refund ($340) processed. No other tickets in the "
"affected queue (47 tickets processed in the same window) showed "
"the same injection pattern on manual review. No customer PII beyond "
"what was already in the ticket was exposed."
),
contributing_factors=[
"No structural separation (e.g. XML/delimiter tagging) between "
"trusted system instructions and untrusted retrieved content",
"Refund tool had no dollar-amount approval threshold requiring "
"human sign-off",
"No injection-pattern regression test covered hidden-text-in-HTML "
"payloads specifically",
],
remediation_actions=[
"Add structural delimiting (XML tags) around all retrieved/ingested "
"content in the agent's prompt template",
"Require human approval for any refund tool call above $0",
"Strip non-visible text (CSS-hidden, zero-width, off-canvas) during "
"ticket ingestion before it reaches the context window",
"Add hidden-text injection payloads to the guardrail regression suite",
],
)
report.timeline = [
TimelineEvent(base_time - timedelta(minutes=18), "detection",
"Finance reconciliation flagged a refund with no matching approval record", "finance-ops"),
TimelineEvent(base_time - timedelta(minutes=10), "detection",
"On-call security engineer paged, incident opened", "pagerduty"),
TimelineEvent(base_time, "triage",
"Confirmed the refund tool call originated from support-agent-v3, traced to ticket #88213", "security-eng"),
TimelineEvent(base_time + timedelta(minutes=25), "triage",
"Extracted hidden instruction payload from ticket #88213 HTML source", "security-eng"),
TimelineEvent(base_time + timedelta(minutes=40), "containment",
"Disabled refund tool access for support-agent-v3 pending fix", "platform-eng"),
TimelineEvent(base_time + timedelta(hours=2), "containment",
"Manually reviewed all 47 tickets processed in the affected window for the same pattern", "security-eng"),
TimelineEvent(base_time + timedelta(hours=6), "eradication",
"Deployed ticket ingestion sanitizer stripping non-visible text", "platform-eng"),
TimelineEvent(base_time + timedelta(hours=8), "eradication",
"Added XML delimiting to agent system prompt template; re-ran injection regression suite", "platform-eng"),
TimelineEvent(base_time + timedelta(hours=10), "recovery",
"Re-enabled refund tool with $0 approval threshold (human sign-off required for all refunds)", "platform-eng"),
TimelineEvent(base_time + timedelta(hours=12), "recovery",
"Confirmed no recurrence over 500 synthetic injection replay tests", "security-eng"),
]
return report
if incident_type == IncidentType.DATA_LEAKAGE:
report = IncidentReport(
incident_id="INC-2026-0047",
title="RAG retrieval scope bug exposes cross-tenant documents in chat responses",
incident_type=incident_type,
severity=severity,
detected_at=base_time,
affected_systems=["rag-retrieval-service", "vector-db-shared-index"],
affected_users_estimate=214,
data_exposed=["customer document excerpts", "customer names", "internal account notes"],
root_cause=(
"A shared vector index migration dropped the tenant_id metadata "
"filter from the retrieval query for approximately 6 hours. "
"Retrieval calls returned top-k nearest neighbors across all "
"tenants rather than being scoped to the requesting tenant, so "
"some chat responses included excerpts from other customers' "
"uploaded documents."
),
blast_radius=(
"214 distinct end users received at least one response containing "
"cross-tenant content during the 6-hour window. Exposure was "
"limited to document excerpts already present in the shared index "
"(no credentials, payment data, or authentication secrets were in "
"the affected index)."
),
contributing_factors=[
"Tenant isolation was enforced only at the query-filter layer with "
"no defense-in-depth check at the response layer",
"The index migration had no automated test asserting cross-tenant "
"isolation before deploy",
"Alerting existed for retrieval latency but not for retrieval "
"result tenant-mismatch",
],
remediation_actions=[
"Add a post-retrieval assertion that every returned chunk's "
"tenant_id matches the requester before it reaches the prompt",
"Add an automated cross-tenant isolation test to the retrieval "
"service's deploy pipeline as a release gate",
"Add real-time alerting on any retrieval result set containing "
"mixed tenant_id values",
"Notify affected customers per data processing agreement and "
"applicable breach notification law",
],
)
report.timeline = [
TimelineEvent(base_time - timedelta(hours=5, minutes=50), "detection",
"Customer support ticket: user reports seeing an unfamiliar company name in a chat answer", "support"),
TimelineEvent(base_time - timedelta(hours=1), "detection",
"Second, unrelated customer report of the same symptom escalated to security", "support"),
TimelineEvent(base_time, "triage",
"Reproduced issue: retrieval query missing tenant_id filter post-migration", "security-eng"),
TimelineEvent(base_time + timedelta(minutes=20), "triage",
"Queried access logs to enumerate all affected requester/tenant pairs in the window", "data-eng"),
TimelineEvent(base_time + timedelta(minutes=35), "containment",
"Rolled back vector index to pre-migration snapshot", "platform-eng"),
TimelineEvent(base_time + timedelta(hours=1), "containment",
"Confirmed tenant_id filter restored and verified with synthetic cross-tenant probes", "security-eng"),
TimelineEvent(base_time + timedelta(hours=3), "eradication",
"Added automated cross-tenant isolation test to CI as a release gate", "platform-eng"),
TimelineEvent(base_time + timedelta(hours=5), "recovery",
"Re-ran migration with the new gate in place; verified no isolation regression", "platform-eng"),
TimelineEvent(base_time + timedelta(days=1), "recovery",
"Legal notified all 214 affected accounts per data processing agreement", "legal"),
]
return report
# MODEL_MISBEHAVIOR / EXCESSIVE_AGENCY / MODEL_EXTRACTION share a compact template
report = IncidentReport(
incident_id="INC-2026-0058",
title=f"{incident_type.value.replace('_', ' ').title()} incident",
incident_type=incident_type,
severity=severity,
detected_at=base_time,
affected_systems=["generic-ai-service"],
affected_users_estimate=1,
data_exposed=[],
root_cause="Root cause under investigation.",
blast_radius="Blast radius under investigation.",
contributing_factors=[],
remediation_actions=["Complete triage and populate root cause and blast radius."],
)
report.timeline = [
TimelineEvent(base_time, "detection", "Anomaly detected by monitoring", "observability-system"),
]
return report
def render_pre_launch_review(report: IncidentReport) -> str:
lines = [
"## Pre-Launch AI Security Review (would this have caught it?)",
"",
f"Applying the standard pre-launch checklist retroactively to "
f"`{report.incident_id}`:",
"",
]
for i, question in enumerate(PRE_LAUNCH_SECURITY_REVIEW_CHECKLIST, start=1):
lines.append(f"{i}. {question}")
lines += [
"",
"A launch review that requires a passing answer to every question above "
"before shipping a tool-using agent feature is the structural fix -- not "
"just the specific patch applied during eradication. Post-mortems that "
"stop at 'we fixed this one bug' without feeding a new check back into "
"the pre-launch review process guarantee a variant of the same incident "
"ships again under a different feature name.",
]
return "\n".join(lines)
def main() -> None:
parser = argparse.ArgumentParser(description="Generate a structured AI incident post-mortem report.")
parser.add_argument(
"--incident-type",
choices=[t.value for t in IncidentType],
default=IncidentType.PROMPT_INJECTION.value,
help="Type of AI incident to generate an example report for.",
)
parser.add_argument(
"--severity",
choices=[s.value for s in Severity],
default=Severity.HIGH.value,
help="Incident severity.",
)
parser.add_argument("--out", default="incident_report.md", help="Path to write the markdown report.")
parser.add_argument("--json-out", default="incident_report.json", help="Path to write the JSON record.")
args = parser.parse_args()
incident_type = IncidentType(args.incident_type)
severity = Severity(args.severity)
report = build_example_incident(incident_type, severity)
full_report = report.render_full_report()
review_section = render_pre_launch_review(report)
print(full_report)
print(review_section)
with open(args.out, "w") as f:
f.write(full_report + "\n" + review_section + "\n")
with open(args.json_out, "w") as f:
json.dump(report.to_dict(), f, indent=2)
print(f"\nFull report written to {args.out}")
print(f"Structured JSON record written to {args.json_out}")
print(
"\nTakeaway: the containment/eradication steps fix this incident. The "
"pre-launch review checklist, fed by this post-mortem's findings, is "
"what prevents the next one. And the regulatory checklist is a "
"screening aid only -- the actual reporting determination always goes "
"through legal/compliance, not through this script."
)
if __name__ == "__main__":
main()
An incident report's triggers_eu_ai_act_reporting() screen returns True for a critical-severity data leakage incident. What should happen next, according to the lesson?
The screening function is explicitly a triage aid, not a legal determination -- it cannot verify facts like whether the specific AI system is classified high-risk under Annex III, or whether the incident's facts actually meet the Article 3(49) definition of "serious incident." Those require legal/compliance judgment. Treating an automated flag as final in either direction -- assuming it's authoritative and reporting immediately, or dismissing it because the technical fix is done -- both skip the judgment call the law actually requires. The correct response is to route the flagged incident to legal/compliance quickly, since the reporting clock starts at awareness, not at the completion of triage.