Phase 1: When AI Harms — Real-World Cases & the Responsible AI Landscape · 40 min · Python · requests · pandas
The Problem
In 2016, ProPublica published an investigation that would become the most cited algorithmic bias case in history. They analyzed COMPAS — a tool used by U.S. judges to predict whether a criminal defendant would reoffend — and found it was nearly twice as likely to falsely flag Black defendants as high risk compared to white defendants. A Black man named Bernard Parker, who had two prior misdemeanors, was rated high risk. A white man named Dylan Fugett, who had three prior felonies, was rated low risk.
Northpointe (the company behind COMPAS) pushed back, arguing their tool was fair by a different metric. The debate that followed — playing out in academic papers, court cases, and eventually the Wisconsin Supreme Court — established something that every Responsible AI Engineer needs to internalize: fairness is not a single number, and two reasonable people can disagree about what "fair" even means.
COMPAS is not an isolated case. The AI Incident Database — a public, curated repository of real-world AI harms — documented 233 AI safety incidents in 2024 alone, a 56.4% increase from 149 in 2023 (Stanford AI Index Report 2025). These aren't thought experiments. They are real systems that caused real harm to real people:
- Amazon's hiring AI (2018) was scrapped after it systematically downgraded resumes containing the word "women's" — trained on 10 years of male-dominated hiring data, it learned that male candidates were preferable
- SafeRent's scoring algorithm gave a voucher-holding applicant a failing score, automatically rejecting her rental application — the DOJ joined a lawsuit claiming it disproportionately harmed protected classes
- A Chevrolet dealership chatbot (2023) was tricked into agreeing to sell a car for one dollar — a harm that was funny until you realized the same architecture powers customer service bots at banks and hospitals
Every one of these incidents is indexed, categorized, and publicly available. The AI Incident Database is where Responsible AI engineering starts — not with frameworks or regulations, but with understanding what has already gone wrong.
The AI incident database
The AI Incident Database (AID) at incidentdatabase.ai is the closest thing the AI field has to aviation's incident reporting system. It catalogs cases where AI systems caused or nearly caused real-world harm. Each incident includes:
- A description of what happened
- The AI system involved
- The harm type and severity
- The date and source
- Links to original reporting
The database classifies harms into categories that map to how organizations actually think about risk:
| Harm Type | Description | Example |
|---|
| Physical | Bodily injury or death | Autonomous vehicle collision, medical misdiagnosis |
| Financial | Economic loss to individuals or organizations | Discriminatory lending, automated trading losses |
| Reputational | Damage to an individual's or organization's reputation | False content generation, misidentification |
| Civil Rights | Violation of legally protected rights | Discriminatory hiring, biased sentencing, housing discrimination |
| Social | Harm to social systems, institutions, or trust | Election interference, misinformation at scale |
Why "near harms" matter
The AID also tracks "near harms" — incidents where harm could have occurred but was caught before impact. These are the AI equivalent of aviation near-misses. A Responsible AI Engineer studies near harms because they reveal failure modes before they become incidents. If your organization's monitoring caught a biased output before it reached a user, that's a near harm worth documenting — not a non-event to ignore.
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers The Concept, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.
Create a free account — the opening phases of 29 of 35 courses are free, no credit card.