Phase 2: Bias Detection & Fairness Metrics · 40 min · Python · pandas · scikit-learn

Where Bias Comes From — Data, Models, and Deployment

Bias doesn't come from the model. It comes from the world the model was trained on.

Hiring signal: Being able to identify the specific source of bias in an ML system — historical, representation, measurement, aggregation, or deployment — is the first technical skill RAI interviewers test. A candidate who can look at a dataset and say 'this is representation bias because the training data under-samples the affected group' demonstrates diagnostic capability that goes beyond running a fairness metric.

What you will learn

Identify the 5 sources of bias in ML systems: historical, representation, measurement, aggregation, deployment
Perform exploratory bias analysis on the COMPAS recidivism dataset
Explain why removing protected attributes doesn't eliminate bias (proxy variables)
Document bias sources with evidence from real data

The Problem

In 2018, Amazon scrapped an AI recruiting tool they'd been building for four years. The system was supposed to rate job candidates from 1 to 5 stars. Instead, it systematically downgraded resumes that included the word "women's" — as in "captain of the women's chess club" — and penalized graduates of two all-women's colleges. The model had been trained on 10 years of Amazon's own hiring data, which was overwhelmingly male. It learned a simple lesson: men get hired, so male-looking resumes are better.

Amazon's engineers hadn't told the model to discriminate against women. They hadn't included gender as a feature. The model found gender anyway — through proxy variables like school names, sports teams, word choices, and resume formatting. This is the fundamental challenge of bias in ML: the bias is in the data, and the data reflects the world.

The 5 Sources of Bias

Bias in ML systems comes from 5 distinct sources. Understanding which source you're dealing with determines what mitigation strategy will work.

Source	Where It Enters	Example	Mitigation Point
Historical	Training data reflects past discrimination	Amazon's hiring AI: 10 years of male-dominated hiring data taught it to prefer men	Pre-processing (reweighing) or in-processing (constraints)
Representation	Training data under-samples or over-samples certain groups	Facial recognition trained primarily on lighter-skinned faces fails on darker-skinned faces	Collect more representative data (pre-processing)
Measurement	The measurement process itself is biased	COMPAS uses arrest data, but arrest rates reflect policing patterns, not actual crime rates	Change the target variable or use proxy-corrected labels
Aggregation	A single model serves groups with different data distributions	A medical diagnosis model trained on adult data applied to children	Group-specific models or conditional models
Deployment	The model behaves differently in production than in testing	A loan model tested on historical data faces a different economic context in production	Post-deployment monitoring and drift detection

Proxy variables are the core problem
Amazon removed gender from the training data. The model still discriminated against women. Why? Because gender is encoded in dozens of other features: school names, sports teams, resume length, word choice, gap years. These are "proxy variables" — features that correlate with the protected attribute without being the attribute itself. You cannot remove all proxies without removing most of the signal in your data. This is why bias detection requires measuring outcomes across protected groups, not just removing protected attributes from the input.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The COMPAS Case: Identifying Bias Sources, What's Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.