The Problem
In 2020, a credit scoring model at a major bank started denying applications from an entire zip code. The cause: a data pipeline change had silently corrupted the income field for that region. The model had been running for 3 weeks before anyone noticed — during which time, hundreds of applicants were wrongly denied credit.
The problem wasn't the model. It was the absence of an incident response process: no detection alerting, no severity classification, no containment procedure, and no reporting mechanism. The bank had a risk register (the risk was identified) but no operational playbook for when it actually happened.
The AI incident response lifecycle
detect → classify → contain → investigate → report → learn
↑ |
└────────────────────────────────────────────────────────┘
(update risk register)
| Stage | What Happens | Timeframe |
|---|
| Detect | Monitoring triggers an alert (bias drift, performance drop, data anomaly) | Automated, real-time |
| Classify | Determine severity level and assign incident lead | Within 1 hour |
| Contain | Stop the bleeding — roll back to previous model, disable automated decisions, switch to manual review | Within 4 hours |
| Investigate | Root cause analysis — what went wrong, why, and how widespread | Within 48 hours |
| Report | Notify stakeholders, regulators (if required), and affected individuals | Within 72 hours (GDPR) |
| Learn | Post-incident review, update risk register, improve monitoring | Within 2 weeks |
Severity classification
| Level | Criteria | Response | Reporting |
|---|
| SEV-1 (Critical) | Model causing active harm to individuals; legal rights violated | Immediate rollback, executive notification, legal engagement | Regulator notification within 72h (GDPR Art. 33), public disclosure |
| SEV-2 (High) | Model producing biased or incorrect outputs affecting many users | Rollback within 4h, senior leadership notification | Internal report, risk committee notification |
| SEV-3 (Medium) | Performance degradation or bias drift beyond thresholds | Monitor closely, schedule emergency retraining | Internal report, team lead notification |
| SEV-4 (Low) | Minor anomaly, single-user issue, cosmetic problem | Log and track, address in next sprint | Team notification |
Building the incident response playbook
"""
incident_response_playbook.py
AI incident response playbook with severity classification and reporting.
"""
import json
from datetime import datetime, timedelta
SEVERITY_LEVELS = {
"SEV-1": {
"name": "Critical",
"criteria": "Model causing active harm to individuals; legal rights violated",
"response_time": "Immediate",
"containment_deadline": timedelta(hours=1),
"reporting_deadline": timedelta(hours=72),
"reporting_required": ["Executive leadership", "Legal team", "Regulator (GDPR Art. 33)", "Affected individuals", "Public disclosure"],
"actions": [
"Immediately roll back to previous model version",
"Disable all automated decisions — switch to manual review",
"Notify executive leadership within 1 hour",
"Engage legal team for regulatory assessment",
"Begin root cause investigation",
"Prepare regulator notification (72-hour GDPR deadline)",
"Prepare affected individual notifications",
],
},
"SEV-2": {
"name": "High",
"criteria": "Model producing biased or incorrect outputs affecting many users",
"response_time": "Within 1 hour",
"containment_deadline": timedelta(hours=4),
"reporting_deadline": timedelta(hours=24),
"reporting_required": ["Senior leadership", "Risk committee", "Legal team"],
"actions": [
"Roll back to previous model version within 4 hours",
"Switch to manual review for affected decisions",
"Notify senior leadership within 1 hour",
"Conduct bias audit on affected predictions",
"Document scope of impact (how many users, how long)",
"Schedule emergency retraining",
],
},
"SEV-3": {
"name": "Medium",
"criteria": "Performance degradation or bias drift beyond thresholds",
"response_time": "Within 4 hours",
"containment_deadline": timedelta(hours=24),
"reporting_deadline": timedelta(days=7),
"reporting_required": ["Team lead", "RAI engineering team"],
"actions": [
"Increase monitoring frequency to hourly",
"Assess whether rollback is needed",
"Schedule emergency retraining",
"Document drift metrics and timeline",
"Update risk register with new findings",
],
},
"SEV-4": {
"name": "Low",
"criteria": "Minor anomaly, single-user issue, or cosmetic problem",
"response_time": "Next business day",
"containment_deadline": timedelta(days=3),
"reporting_deadline": timedelta(days=14),
"reporting_required": ["Development team"],
"actions": [
"Log the incident with full details",
"Track in backlog for next sprint",
"Monitor for recurrence",
],
},
}
def classify_incident(description, affected_users, harm_level, automated_decision):
"""
Classify an AI incident into a severity level.
Args:
description: What happened
affected_users: Estimated number of affected users
harm_level: "none", "minor", "significant", "severe"
automated_decision: Whether the model made automated decisions (True/False)
Returns:
Dict with severity classification and response plan
"""
if harm_level == "severe" and automated_decision:
severity = "SEV-1"
elif harm_level in ["severe", "significant"] and affected_users > 100:
severity = "SEV-2"
elif harm_level in ["significant", "minor"] and affected_users > 10:
severity = "SEV-3"
else:
severity = "SEV-4"
level = SEVERITY_LEVELS[severity]
detected_at = datetime.now()
incident = {
"incident_id": f"INC-{detected_at.strftime('%Y%m%d')}-{severity}",
"description": description,
"detected_at": detected_at.isoformat(),
"severity": severity,
"severity_name": level["name"],
"criteria": level["criteria"],
"affected_users": affected_users,
"harm_level": harm_level,
"automated_decision": automated_decision,
"response_time": level["response_time"],
"containment_deadline": (detected_at + level["containment_deadline"]).isoformat(),
"reporting_deadline": (detected_at + level["reporting_deadline"]).isoformat(),
"reporting_required": level["reporting_required"],
"actions": level["actions"],
"status": "detected",
"framework_mapping": {
"EU AI Act Art. 73": "Serious incident reporting to market surveillance authorities",
"GDPR Art. 33": "Personal data breach notification within 72 hours" if severity in ["SEV-1", "SEV-2"] else "Not applicable",
"NIST AI RMF MANAGE 4.1": "Incident response and continuous monitoring",
"ISO 42001 Clause 10.2": "Nonconformity and corrective action",
},
}
return incident
def generate_incident_report(incident):
"""Generate a Markdown incident report."""
lines = []
lines.append(f"# AI Incident Report — {incident['incident_id']}\n")
lines.append(f"**Severity:** {incident['severity']} ({incident['severity_name']})")
lines.append(f"**Description:** {incident['description']}")
lines.append(f"**Detected:** {incident['detected_at']}")
lines.append(f"**Affected users:** ~{incident['affected_users']}")
lines.append(f"**Harm level:** {incident['harm_level']}")
lines.append(f"**Automated decisions:** {'Yes' if incident['automated_decision'] else 'No'}\n")
lines.append("## Response Timeline\n")
lines.append(f"- **Response time:** {incident['response_time']}")
lines.append(f"- **Containment deadline:** {incident['containment_deadline']}")
lines.append(f"- **Reporting deadline:** {incident['reporting_deadline']}\n")
lines.append("## Required Actions\n")
for i, action in enumerate(incident["actions"], 1):
lines.append(f"{i}. [ ] {action}")
lines.append("")
lines.append("## Reporting Requirements\n")
for party in incident["reporting_required"]:
lines.append(f"- [ ] Notify: {party}")
lines.append("")
lines.append("## Framework Mapping\n")
lines.append("| Framework | Requirement |")
lines.append("|---|---|")
for fw, req in incident["framework_mapping"].items():
lines.append(f"| {fw} | {req} |")
lines.append("")
lines.append("## Post-Incident Review Template\n")
lines.append("- [ ] Root cause identified and documented")
lines.append("- [ ] Timeline of detection → containment → resolution documented")
lines.append("- [ ] Number of affected users confirmed")
lines.append("- [ ] Corrective actions identified")
lines.append("- [ ] Risk register updated with new findings")
lines.append("- [ ] Monitoring thresholds adjusted if needed")
lines.append("- [ ] Lessons learned shared with team")
lines.append("- [ ] Regulatory notifications completed (if required)")
return "\n".join(lines)
if __name__ == "__main__":
# Example: A bias drift incident
incident = classify_incident(
description="Loan approval model's disparate impact ratio dropped from 0.78 to 0.65 for Group_B over 2 weeks",
affected_users=450,
harm_level="significant",
automated_decision=True,
)
report = generate_incident_report(incident)
with open("incident_report.md", "w") as f:
f.write(report)
with open("incident_report.json", "w") as f:
json.dump(incident, f, indent=2)
print(f"Incident: {incident['incident_id']}")
print(f"Severity: {incident['severity']} ({incident['severity_name']})")
print(f"Containment deadline: {incident['containment_deadline']}")
print(f"Reporting deadline: {incident['reporting_deadline']}")
print(f"Required actions: {len(incident['actions'])}")
print("\nReport saved to incident_report.md")
Your monitoring system flags that a hiring model's disparate impact ratio dropped from 0.82 to 0.71 for one group over the past month, affecting roughly 300 candidates. The model runs fully automated — no recruiter reviews the auto-reject decisions. Using classify_incident's logic, what severity should this get, and what does that trigger?
The classifier's first branch checks harm_level == "severe" and automated_decision before it even looks at the affected-user count — that ordering is deliberate, because a fully automated system with severe harm is the scenario where waiting to count users first would cost time you don't have. A hiring model auto-rejecting candidates with no human in the loop, where the harm is discriminatory exclusion from opportunity, qualifies as severe harm under an automated decision, which is SEV-1 regardless of whether the harm is "physical" — civil rights harm is exactly the kind of harm the EU AI Act's Article 73 serious-incident reporting and GDPR's Article 33 breach notification exist for. Treating this as SEV-3 "because it's just a metric drifting" ignores that the metric represents real people being wrongly excluded from jobs, automatically, every day the incident goes unaddressed.