The Problem
In 2021, New York City passed Local Law 144, requiring any employer using an "automated employment decision tool" to screen candidates to commission an independent bias audit — computing selection rates by race/ethnicity and sex, and publishing the results — before the tool can be used. It's one of the first laws anywhere to turn "audit your model for bias" from best practice into a legal requirement with teeth, and it's not alone: the EU AI Act classifies AI systems used in employment, credit, and essential services as high-risk and requires bias testing and monitoring; NIST's AI RMF names fairness as a core Measure-function metric; and ISO/IEC 42001 asks certified organizations to demonstrate ongoing fairness evaluation as part of their AI management system.
The common failure mode these laws are built around isn't a model that explicitly checks race or gender — that's rare, and easy to catch with a code review. It's a model that never sees a protected attribute at all, but learns the same discriminatory pattern through a correlated proxy: zip code standing in for race, a "credit tier" that encodes historical redlining, a college name that correlates with gender in ways a hiring model never intended. A bias audit that only checks "did we use a protected attribute" would pass this model. The only way to catch it is to measure outcomes by group and see if they diverge — which is exactly what demographic parity, disparate impact, and equalized odds are for.
This lesson builds a bias audit and explainability pipeline for a synthetic loan-approval model that has exactly this problem: it never explicitly used race, but it did use a proxy, and the audit catches it from the outside, by outcomes alone.
Fairness metrics: three different questions
There is no single number called "fairness." Different metrics ask different questions, and picking the wrong one for your context is itself a common audit failure.
| Metric | Question it answers | Formula | Where it's codified |
|---|
| Demographic parity difference | Do groups get approved at the same rate, regardless of whether they qualify? | max(selection_rate) − min(selection_rate) across groups | General EEOC guidance context |
| Disparate impact ratio (four-fifths rule) | Is the lowest group's selection rate at least 80% of the highest group's? | min(selection_rate) / max(selection_rate) | EEOC Uniform Guidelines; the exact ratio NYC Local Law 144 audits are built around |
| Equalized odds difference | Among people who actually qualify (or don't), does the model treat groups the same? | max(gap in TPR, gap in FPR) across groups | Hardt, Price & Srebro (2016); used when ground-truth labels exist |
Demographic parity and the four-fifths rule are outcome-only metrics — they don't need to know who actually deserved approval, which makes them usable in production monitoring where you rarely have ground truth. Equalized odds needs labeled outcomes (did the person who got the loan actually repay it, did the person who got hired actually perform well), which makes it more informative but only available after the fact, often on a delay, and sometimes never available at all for the people who were rejected. A mature audit reports more than one metric and is explicit about which one it's optimizing for, because a model can satisfy demographic parity while badly failing equalized odds, and vice versa — they are not the same question, and no single number replaces reporting both.
The proxy is the point, not the exception
The textbook bias-audit example uses a model with a direct coefficient on a protected attribute, because it's easy to explain. But real production bias almost never looks like that — anyone can grep training code for "race" or "gender" fields. The realistic failure is a legacy feature (a zip code, a "credit tier" inherited from a discontinued scoring system, a university field) that correlates with a protected group for historical, structural reasons the model never intended to learn. This is why a bias audit has to be an outcomes-based, black-box test — checking group-level results — rather than a code review of which features a model consumes. You cannot grep your way to a fairness guarantee.
Tooling: fairlearn, AIF360, and SHAP
Two open-source libraries dominate production fairness work. Fairlearn (Microsoft-maintained, fairlearn.metrics) provides MetricFrame for computing any metric broken out by group, plus purpose-built functions like demographic_parity_difference and equalized_odds_difference. AIF360 (IBM/LF AI & Data) covers a broader surface — more fairness metrics, plus bias mitigation algorithms (reweighing, adversarial debiasing, reject-option classification) that Fairlearn's metrics-first API doesn't include. Teams commonly use Fairlearn for the audit-and-report workflow and reach for AIF360 when they need to actually fix a model that failed the audit, not just document that it failed.
Once an audit surfaces a disparity, the next question is why — and that's an explainability problem, not a fairness-metrics problem. SHAP (SHapley Additive exPlanations) answers "which features drove this specific decision, and by how much" using a game-theoretic attribution that has one crucial property: the attributions for a given prediction sum exactly back to the difference between that prediction and a baseline. For linear models, shap.LinearExplainer computes this in closed form — no sampling, no approximation. LIME (Local Interpretable Model-agnostic Explanations) answers a similar question differently: it fits a simple, interpretable model locally around one prediction to approximate the complex model's behavior nearby. LIME is more model-agnostic and often faster on very large models; SHAP has stronger theoretical guarantees (it's the only attribution method satisfying local accuracy, consistency, and missingness simultaneously) and is more commonly the one auditors and regulators expect to see cited.
An audit reports a demographic parity difference of 0.02 (nearly identical approval rates between groups) but an equalized odds difference of 0.31 (a large gap in true positive rate between groups). What does this combination most likely indicate?
Demographic parity only looks at overall selection rates — it can be satisfied by a model that approves qualified people in one group and unqualified people in another, as long as the totals balance out. Equalized odds looks at whether the model's true positive rate and false positive rate are the same across groups — i.e., does it treat equally-qualified people equally. A near-zero demographic parity gap alongside a large equalized odds gap is a classic sign that a model is "balancing" its overall numbers while still discriminating at the individual level: exactly the pattern a naive audit that only checks one metric would miss.
Model cards and dataset cards
A bias audit produces numbers; a model card and dataset card are what turn those numbers into documentation an auditor, a regulator, or a downstream engineer can actually use. A model card documents intended use, out-of-scope uses, the fairness evaluation performed (method, metrics, pass/fail against a stated threshold), and known limitations. A dataset card documents composition (including per-group prevalence of the outcome, which is what lets a reader spot base-rate imbalance before it becomes a bias-audit surprise), feature definitions, and provenance. Both formats are now referenced directly by name in the EU AI Act's high-risk documentation requirements and NIST's AI RMF Map function, and both are far more useful when generated automatically from the same code that ran the audit — a hand-written model card drifts out of sync with the model it describes the first time anyone retrains it.
Build
Build it
The code artifact lesson_07_04_bias_audit_shap.py runs a full bias audit and explainability pass on a synthetic loan-approval model, using the real fairlearn and shap packages when installed and mathematically equivalent from-scratch implementations otherwise:
generate_synthetic_applicants() — builds 500 synthetic applicants with a ground-truth repayment model that never uses group membership, and a deployed decision model that does (a group_b_flag coefficient standing in for a real-world correlated proxy)compute_fairness_report() — computes demographic parity difference, the four-fifths disparate impact ratio, and equalized odds difference, via fairlearn.metrics.MetricFrame and demographic_parity_difference/equalized_odds_difference when installed, or a from-scratch implementation that computes the identical formulas otherwiseexplain_decisions() — attributes individual decisions to features via shap.LinearExplainer when installed, or a closed-form linear attribution (weight * (value - background_mean)) that is exact, not an approximation, for this linear modelrender_report(), render_model_card(), render_dataset_card() — produce the audit report, model card, and dataset card as markdown
"""
lesson_07_04_bias_audit_shap.py
A bias audit + explainability tool for a binary classifier with a protected
attribute: fairness metrics (demographic parity difference, disparate impact
ratio / four-fifths rule, equalized odds difference) via the real `fairlearn`
package when installed, plus per-decision feature attribution via the real
`shap` package when installed.
Both fall back to from-scratch implementations when the library isn't
available -- and for this lesson's linear decision model, the fallback
attribution formula (coefficient * (feature_value - background_mean)) is
mathematically identical to what shap.LinearExplainer computes under its
default independent-feature masker, not an approximation of it.
Reference: Fairlearn (https://fairlearn.org/), AIF360
(https://github.com/Trusted-AI/AIF360), SHAP (https://github.com/shap/shap).
Usage:
python lesson_07_04_bias_audit_shap.py
python lesson_07_04_bias_audit_shap.py --n 1000 --out my_audit.md
Requirements (optional, for the real tools):
pip install fairlearn shap numpy
"""
from __future__ import annotations
import argparse
import math
import random
import statistics
from dataclasses import dataclass
try:
from fairlearn.metrics import (
MetricFrame,
demographic_parity_difference,
equalized_odds_difference,
false_positive_rate,
selection_rate,
true_positive_rate,
)
FAIRLEARN_INSTALLED = True
except ImportError:
FAIRLEARN_INSTALLED = False
try:
import numpy as np
import shap
SHAP_INSTALLED = True
except ImportError:
SHAP_INSTALLED = False
RANDOM_SEED = 42
N_APPLICANTS = 500
APPROVAL_THRESHOLD = 0.5 # decision boundary on the sigmoid'd score
FEATURE_NAMES = ["debt_ratio", "income_norm", "group_b_flag"]
# ── The deployed decision model ──────────────────────────────────────────────
# A simple linear-then-sigmoid model. It includes a direct coefficient on
# group membership (`group_b_flag`) -- a stand-in for the far more common
# real-world case where the model never sees a protected attribute directly,
# but a correlated proxy feature (zip code, a legacy "credit tier", school
# attended) reintroduces the same effect. Modeling it explicitly here keeps
# the audit's job -- catching disparate impact from the outside, purely by
# looking at outcomes -- honest and easy to verify by hand.
WEIGHTS = {
"debt_ratio": -6.0,
"income_norm": 2.5,
"group_b_flag": -1.4,
}
INTERCEPT = 1.0
# ── Ground truth ──────────────────────────────────────────────────────────────
# What actually determines whether an applicant would repay the loan. Group
# membership is deliberately absent -- it is not a real risk factor. Any gap
# between the deployed model's decisions and this ground truth is exactly the
# proxy discrimination a bias audit exists to surface.
TRUE_WEIGHTS = {"debt_ratio": -6.0, "income_norm": 2.5}
TRUE_INTERCEPT = 1.0
def sigmoid(x: float) -> float:
return 1.0 / (1.0 + math.exp(-x))
@dataclass
class Applicant:
applicant_id: int
group: str # "A" (privileged) | "B" (unprivileged)
debt_ratio: float
income_norm: float
true_label: int # 1 = would actually repay
predicted: int # 1 = model approves the loan
decision_score: float
def generate_synthetic_applicants(n: int = N_APPLICANTS, seed: int = RANDOM_SEED) -> list[Applicant]:
rng = random.Random(seed)
applicants = []
for i in range(n):
group = "A" if rng.random() < 0.6 else "B"
debt_ratio = rng.uniform(0.05, 0.65)
income_norm = rng.gauss(0.0, 1.0) # standardized income, mean 0
true_score = (
TRUE_INTERCEPT
+ TRUE_WEIGHTS["debt_ratio"] * debt_ratio
+ TRUE_WEIGHTS["income_norm"] * income_norm
)
true_label = 1 if rng.random() < sigmoid(true_score) else 0
group_b_flag = 1.0 if group == "B" else 0.0
decision_score = (
INTERCEPT
+ WEIGHTS["debt_ratio"] * debt_ratio
+ WEIGHTS["income_norm"] * income_norm
+ WEIGHTS["group_b_flag"] * group_b_flag
)
predicted = 1 if sigmoid(decision_score) >= APPROVAL_THRESHOLD else 0
applicants.append(
Applicant(i, group, debt_ratio, income_norm, true_label, predicted, decision_score)
)
return applicants
# ── Fairness metrics ──────────────────────────────────────────────────────────
@dataclass
class GroupStats:
group: str
n: int
selection_rate: float
tpr: float
fpr: float
def compute_group_stats_manual(applicants: list[Applicant], group: str) -> GroupStats:
subset = [a for a in applicants if a.group == group]
selection = sum(a.predicted for a in subset) / len(subset)
positives = [a for a in subset if a.true_label == 1]
negatives = [a for a in subset if a.true_label == 0]
tpr = sum(a.predicted for a in positives) / len(positives) if positives else 0.0
fpr = sum(a.predicted for a in negatives) / len(negatives) if negatives else 0.0
return GroupStats(group, len(subset), selection, tpr, fpr)
@dataclass
class FairnessReport:
method: str # "fairlearn" | "from-scratch"
group_stats: dict[str, GroupStats]
demographic_parity_difference: float
disparate_impact_ratio: float
equalized_odds_difference: float
@property
def passes_four_fifths_rule(self) -> bool:
return self.disparate_impact_ratio >= 0.8
def compute_fairness_report(applicants: list[Applicant]) -> FairnessReport:
groups = sorted({a.group for a in applicants})
y_true = [a.true_label for a in applicants]
y_pred = [a.predicted for a in applicants]
sensitive = [a.group for a in applicants]
if FAIRLEARN_INSTALLED:
method = "fairlearn"
dp_diff = demographic_parity_difference(y_true, y_pred, sensitive_features=sensitive)
eo_diff = equalized_odds_difference(y_true, y_pred, sensitive_features=sensitive)
mf = MetricFrame(
metrics={"selection_rate": selection_rate, "tpr": true_positive_rate, "fpr": false_positive_rate},
y_true=y_true,
y_pred=y_pred,
sensitive_features=sensitive,
)
group_stats = {}
for g in groups:
row = mf.by_group.loc[g]
n = sum(1 for a in applicants if a.group == g)
group_stats[g] = GroupStats(g, n, float(row["selection_rate"]), float(row["tpr"]), float(row["fpr"]))
else:
method = "from-scratch"
group_stats = {g: compute_group_stats_manual(applicants, g) for g in groups}
rates = [gs.selection_rate for gs in group_stats.values()]
dp_diff = max(rates) - min(rates)
tpr_gap = max(gs.tpr for gs in group_stats.values()) - min(gs.tpr for gs in group_stats.values())
fpr_gap = max(gs.fpr for gs in group_stats.values()) - min(gs.fpr for gs in group_stats.values())
eo_diff = max(tpr_gap, fpr_gap)
# Disparate impact ratio / four-fifths rule: the lowest group selection
# rate divided by the highest. This is the same ratio the EEOC's Uniform
# Guidelines use, and the one NYC Local Law 144 bias audits for automated
# employment decision tools are built around. Below 0.8 is the
# conventional adverse-impact signal.
selection_rates = {g: gs.selection_rate for g, gs in group_stats.items()}
disparate_impact_ratio = (
min(selection_rates.values()) / max(selection_rates.values())
if max(selection_rates.values()) > 0
else 1.0
)
return FairnessReport(method, group_stats, dp_diff, disparate_impact_ratio, eo_diff)
# ── Explainability ────────────────────────────────────────────────────────────
def _feature_vector(a: Applicant) -> dict[str, float]:
return {
"debt_ratio": a.debt_ratio,
"income_norm": a.income_norm,
"group_b_flag": 1.0 if a.group == "B" else 0.0,
}
def find_notable_cases(applicants: list[Applicant]) -> list[Applicant]:
"""One wrongly-denied qualified group-B applicant closest to the approval
boundary (the clearest individual evidence of proxy discrimination), and
one correctly-approved group-A applicant as a baseline comparison."""
wrongly_denied = sorted(
(a for a in applicants if a.group == "B" and a.true_label == 1 and a.predicted == 0),
key=lambda a: -a.decision_score,
)
baseline_approved = sorted(
(a for a in applicants if a.group == "A" and a.true_label == 1 and a.predicted == 1),
key=lambda a: abs(a.decision_score),
)
sample = []
if wrongly_denied:
sample.append(wrongly_denied[0])
if baseline_approved:
sample.append(baseline_approved[0])
return sample
def explain_from_scratch(sample: list[Applicant], background_means: dict[str, float]) -> dict[int, dict[str, float]]:
"""
Closed-form linear feature attribution: contribution_i = w_i * (x_i - mean_i).
This is not an approximation -- for a linear model, this is exactly the
Shapley-value decomposition shap.LinearExplainer computes under its
default independent-feature masker. The contributions plus the model's
score at the background mean sum exactly back to the model's output for
that input.
"""
attributions = {}
for a in sample:
x = _feature_vector(a)
attributions[a.applicant_id] = {
name: WEIGHTS[name] * (x[name] - background_means[name]) for name in FEATURE_NAMES
}
return attributions
def explain_with_shap(sample: list[Applicant], background_means: dict[str, float]) -> dict[int, dict[str, float]]:
coef = np.array([WEIGHTS[name] for name in FEATURE_NAMES])
background = np.array([[background_means[name] for name in FEATURE_NAMES]])
explainer = shap.LinearExplainer((coef, INTERCEPT), background)
X_sample = np.array([[_feature_vector(a)[name] for name in FEATURE_NAMES] for a in sample])
shap_values = explainer(X_sample)
attributions = {}
for i, a in enumerate(sample):
attributions[a.applicant_id] = {
name: float(shap_values.values[i][j]) for j, name in enumerate(FEATURE_NAMES)
}
return attributions
def explain_decisions(applicants: list[Applicant]) -> tuple[str, dict[int, dict[str, float]]]:
sample = find_notable_cases(applicants)
background_means = {
name: statistics.mean(_feature_vector(a)[name] for a in applicants) for name in FEATURE_NAMES
}
if SHAP_INSTALLED:
return "shap.LinearExplainer", explain_with_shap(sample, background_means)
return "from-scratch (closed-form linear attribution)", explain_from_scratch(sample, background_means)
# ── Reporting ─────────────────────────────────────────────────────────────────
def render_report(
applicants: list[Applicant],
fairness: FairnessReport,
explain_method: str,
attributions: dict[int, dict[str, float]],
) -> str:
by_id = {a.applicant_id: a for a in applicants}
lines = [
"BIAS AUDIT REPORT",
"=" * 70,
f"Applicants: {len(applicants)} | Fairness method: {fairness.method} | "
f"Explainability method: {explain_method}",
"",
"Group statistics:",
]
for g in sorted(fairness.group_stats):
gs = fairness.group_stats[g]
lines.append(
f" Group {g}: n={gs.n:4d} selection_rate={gs.selection_rate:.1%} "
f"TPR={gs.tpr:.1%} FPR={gs.fpr:.1%}"
)
lines += [
"",
f"Demographic parity difference: {fairness.demographic_parity_difference:.3f}",
f"Disparate impact ratio (four-fifths rule): {fairness.disparate_impact_ratio:.3f} "
f"({'PASSES' if fairness.passes_four_fifths_rule else 'FAILS'} the 0.80 threshold)",
f"Equalized odds difference: {fairness.equalized_odds_difference:.3f}",
"",
f"Feature attributions ({explain_method}) for notable cases:",
]
for applicant_id, attrib in attributions.items():
a = by_id[applicant_id]
lines.append(
f" Applicant #{applicant_id} (group {a.group}, true_label={a.true_label}, "
f"predicted={a.predicted}):"
)
for name, value in sorted(attrib.items(), key=lambda kv: -abs(kv[1])):
lines.append(f" {name:15s} contributes {value:+.3f} to the decision score")
return "\n".join(lines)
def render_model_card(fairness: FairnessReport) -> str:
verdict = "PASSES" if fairness.passes_four_fifths_rule else "FAILS"
return f"""# Model Card: Loan Approval Decision Model (synthetic demo)
## Intended Use
Approves or denies unsecured personal loan applications from applicant debt
ratio and normalized income. Not evaluated for any use outside this synthetic
teaching scenario.
## Fairness Evaluation
- Protected attribute evaluated: `group` (A = privileged, B = unprivileged)
- Method: {fairness.method}
- Demographic parity difference: {fairness.demographic_parity_difference:.3f}
- Disparate impact ratio: {fairness.disparate_impact_ratio:.3f} ({verdict} the four-fifths rule, threshold 0.80)
- Equalized odds difference: {fairness.equalized_odds_difference:.3f}
## Known Limitations
This model includes a deliberate legacy penalty on group membership
(`group_b_flag`) to demonstrate proxy discrimination for teaching purposes.
A real model card documents the actual root cause once traced -- typically a
correlated feature, not a raw group flag -- along with a remediation owner
and a re-audit date.
"""
def render_dataset_card(applicants: list[Applicant]) -> str:
groups = sorted({a.group for a in applicants})
lines = [
"# Dataset Card: Synthetic Loan Applicant Data",
"",
"## Composition",
f"- Total applicants: {len(applicants)}",
]
for g in groups:
subset = [a for a in applicants if a.group == g]
prevalence = sum(a.true_label for a in subset) / len(subset)
lines.append(f"- Group {g}: n={len(subset)}, true-label (would-repay) prevalence={prevalence:.1%}")
lines += [
"",
"## Features",
"- `debt_ratio`: uniform(0.05, 0.65), independent of group",
"- `income_norm`: standardized income, mean 0, independent of group",
"- `group`: protected attribute, not used by the ground-truth repayment model",
"",
"## Provenance",
"Synthetically generated (`generate_synthetic_applicants`, fixed seed "
f"{RANDOM_SEED}) for this lesson. No real applicant data was used.",
]
return "\n".join(lines)
def main() -> None:
parser = argparse.ArgumentParser(description="Synthetic bias audit + explainability report.")
parser.add_argument("--n", type=int, default=N_APPLICANTS, help="Number of synthetic applicants.")
parser.add_argument("--out", default="bias_audit_report.md", help="Path to write the audit report.")
args = parser.parse_args()
if not FAIRLEARN_INSTALLED:
print("fairlearn is not installed. For the real metrics library: pip install fairlearn")
print("Using an equivalent from-scratch fairness metric implementation instead.\n")
if not SHAP_INSTALLED:
print("shap is not installed. For the real explainability library: pip install shap")
print("Using a from-scratch closed-form linear attribution instead (exact for this model).\n")
applicants = generate_synthetic_applicants(args.n)
fairness = compute_fairness_report(applicants)
explain_method, attributions = explain_decisions(applicants)
report = render_report(applicants, fairness, explain_method, attributions)
print(report)
with open(args.out, "w") as f:
f.write("# Bias Audit Report\n\n```\n" + report + "\n```\n")
with open("model_card.md", "w") as f:
f.write(render_model_card(fairness))
with open("dataset_card.md", "w") as f:
f.write(render_dataset_card(applicants))
print(f"\nFull report written to {args.out}, model_card.md, and dataset_card.md")
print(
"\nTakeaway: the deployed model never asks about race, gender, or any "
"explicitly protected class -- it fails the four-fifths rule anyway, "
"because a proxy feature correlated with group membership does the "
"same work. A bias audit that only checks 'did we use a protected "
"attribute' would have passed this model. Checking outcomes is what "
"catches it."
)
if __name__ == "__main__":
main()
The audit's from-scratch feature attribution uses the formula weight * (feature_value - background_mean) for a linear model. Why is this described as exact rather than an approximation of SHAP?
SHAP's guarantee is that attributions sum exactly back to the difference between a prediction and a baseline. For a linear model f(x) = w·x + b, that decomposition has a unique closed form: each feature's contribution is its weight times its deviation from the background/baseline value. shap.LinearExplainer computes precisely this under its default assumptions -- there's no sampling or approximation involved for the linear case, so a hand-written version of the same formula produces identical numbers, not merely similar ones.