Phase 1: When AI Harms — Real-World Cases & the Responsible AI Landscape · 40 min · Python · requests · pandas

When Algorithms Harm — The AI Incident Database

You cannot prevent what you cannot name.

Hiring signal: Responsible AI roles at companies like Citi, Marriott, and Schwab require candidates who can cite real AI incidents and their harm categories — not abstract philosophy. Being able to query the AI Incident Database and produce a structured harm taxonomy is the most basic evidence of domain literacy.

What you will learn

Query the AI Incident Database and retrieve real incident records
Categorize AI incidents by harm type: physical, financial, reputational, civil rights, social
Explain the COMPAS recidivism case and why it remains the canonical algorithmic bias example
Articulate why AI incidents increased 56.4% from 2023 to 2024 (Stanford AI Index)

The Problem

In 2016, ProPublica published an investigation that would become the most cited algorithmic bias case in history. They analyzed COMPAS — a tool used by U.S. judges to predict whether a criminal defendant would reoffend — and found it was nearly twice as likely to falsely flag Black defendants as high risk compared to white defendants. A Black man named Bernard Parker, who had two prior misdemeanors, was rated high risk. A white man named Dylan Fugett, who had three prior felonies, was rated low risk.

Northpointe (the company behind COMPAS) pushed back, arguing their tool was fair by a different metric. The debate that followed — playing out in academic papers, court cases, and eventually the Wisconsin Supreme Court — established something that every Responsible AI Engineer needs to internalize: fairness is not a single number, and two reasonable people can disagree about what "fair" even means.

COMPAS is not an isolated case. The AI Incident Database — a public, curated repository of real-world AI harms — documented 233 AI safety incidents in 2024 alone, a 56.4% increase from 149 in 2023 (Stanford AI Index Report 2025). These aren't thought experiments. They are real systems that caused real harm to real people:

Amazon's hiring AI (2018) was scrapped after it systematically downgraded resumes containing the word "women's" — trained on 10 years of male-dominated hiring data, it learned that male candidates were preferable
SafeRent's scoring algorithm gave a voucher-holding applicant a failing score, automatically rejecting her rental application — the DOJ joined a lawsuit claiming it disproportionately harmed protected classes
A Chevrolet dealership chatbot (2023) was tricked into agreeing to sell a car for one dollar — a harm that was funny until you realized the same architecture powers customer service bots at banks and hospitals

Every one of these incidents is indexed, categorized, and publicly available. The AI Incident Database is where Responsible AI engineering starts — not with frameworks or regulations, but with understanding what has already gone wrong.

The AI Incident Database

The AI Incident Database (AID) at incidentdatabase.ai is the closest thing the AI field has to aviation's incident reporting system. It catalogs cases where AI systems caused or nearly caused real-world harm. Each incident includes:

A description of what happened
The AI system involved
The harm type and severity
The date and source
Links to original reporting

The database classifies harms into categories that map to how organizations actually think about risk:

Harm Type	Description	Example
Physical	Bodily injury or death	Autonomous vehicle collision, medical misdiagnosis
Financial	Economic loss to individuals or organizations	Discriminatory lending, automated trading losses
Reputational	Damage to an individual's or organization's reputation	False content generation, misidentification
Civil Rights	Violation of legally protected rights	Discriminatory hiring, biased sentencing, housing discrimination
Social	Harm to social systems, institutions, or trust	Election interference, misinformation at scale

Why "near harms" matter
The AID also tracks "near harms" — incidents where harm could have occurred but was caught before impact. These are the AI equivalent of aviation near-misses. A Responsible AI Engineer studies near harms because they reveal failure modes before they become incidents. If your organization's monitoring caught a biased output before it reached a user, that's a near harm worth documenting — not a non-event to ignore.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The COMPAS Case: Why It Still Matters, Building an Incident Categorizer, The 56.4% Increase — Why Incidents Are Accelerating, What's Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.