Phase 7: AI Governance, Compliance & Risk · 55 min · AIF360 · Fairlearn · SHAP

Bias Audits, Fairness Testing & Explainability

A model that never asks about race or gender can still fail the four-fifths rule. Auditing outcomes is the only thing that catches it.

Hiring signal: NYC Local Law 144, the EU AI Act's high-risk obligations, and most enterprise AI procurement reviews all require a documented bias audit before a decision-making model ships. Candidates who can run demographic parity and equalized odds metrics with Fairlearn or AIF360, attach SHAP-backed explanations to individual decisions, and produce a model card an auditor accepts are the ones trusted to sign off on a launch, not just build the model.

What you will learn

Compute and interpret demographic parity difference, disparate impact ratio (four-fifths rule), and equalized odds difference
Run a fairness audit with Fairlearn (or AIF360) and identify when a model discriminates through a proxy feature rather than a protected attribute directly
Generate per-decision feature attributions with SHAP and explain why a specific individual was approved or denied
Write a model card and dataset card that satisfy the documentation requirements common to NIST AI RMF, the EU AI Act, and ISO/IEC 42001

The Problem

In 2021, New York City passed Local Law 144, requiring any employer using an "automated employment decision tool" to screen candidates to commission an independent bias audit — computing selection rates by race/ethnicity and sex, and publishing the results — before the tool can be used. It's one of the first laws anywhere to turn "audit your model for bias" from best practice into a legal requirement with teeth, and it's not alone: the EU AI Act classifies AI systems used in employment, credit, and essential services as high-risk and requires bias testing and monitoring; NIST's AI RMF names fairness as a core Measure-function metric; and ISO/IEC 42001 asks certified organizations to demonstrate ongoing fairness evaluation as part of their AI management system.

The common failure mode these laws are built around isn't a model that explicitly checks race or gender — that's rare, and easy to catch with a code review. It's a model that never sees a protected attribute at all, but learns the same discriminatory pattern through a correlated proxy: zip code standing in for race, a "credit tier" that encodes historical redlining, a college name that correlates with gender in ways a hiring model never intended. A bias audit that only checks "did we use a protected attribute" would pass this model. The only way to catch it is to measure outcomes by group and see if they diverge — which is exactly what demographic parity, disparate impact, and equalized odds are for.

This lesson builds a bias audit and explainability pipeline for a synthetic loan-approval model that has exactly this problem: it never explicitly used race, but it did use a proxy, and the audit catches it from the outside, by outcomes alone.

Fairness Metrics: Three Different Questions

There is no single number called "fairness." Different metrics ask different questions, and picking the wrong one for your context is itself a common audit failure.

Metric	Question it answers	Formula	Where it's codified
Demographic parity difference	Do groups get approved at the same rate, regardless of whether they qualify?	`max(selection_rate) − min(selection_rate)` across groups	General EEOC guidance context
Disparate impact ratio (four-fifths rule)	Is the lowest group's selection rate at least 80% of the highest group's?	`min(selection_rate) / max(selection_rate)`	EEOC Uniform Guidelines; the exact ratio NYC Local Law 144 audits are built around
Equalized odds difference	Among people who actually qualify (or don't), does the model treat groups the same?	`max(gap in TPR, gap in FPR)` across groups	Hardt, Price & Srebro (2016); used when ground-truth labels exist

Demographic parity and the four-fifths rule are outcome-only metrics — they don't need to know who actually deserved approval, which makes them usable in production monitoring where you rarely have ground truth. Equalized odds needs labeled outcomes (did the person who got the loan actually repay it, did the person who got hired actually perform well), which makes it more informative but only available after the fact, often on a delay, and sometimes never available at all for the people who were rejected. A mature audit reports more than one metric and is explicit about which one it's optimizing for, because a model can satisfy demographic parity while badly failing equalized odds, and vice versa — they are not the same question, and no single number replaces reporting both.

The proxy is the point, not the exception
The textbook bias-audit example uses a model with a direct coefficient on a protected attribute, because it's easy to explain. But real production bias almost never looks like that — anyone can grep training code for "race" or "gender" fields. The realistic failure is a legacy feature (a zip code, a "credit tier" inherited from a discontinued scoring system, a university field) that correlates with a protected group for historical, structural reasons the model never intended to learn. This is why a bias audit has to be an outcomes-based, black-box test — checking group-level results — rather than a code review of which features a model consumes. You cannot grep your way to a fairness guarantee.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Tooling: Fairlearn, AIF360, and SHAP, Model Cards and Dataset Cards, Build It, Intended Use, Fairness Evaluation, Known Limitations, What to Practice — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.