Phase 2: Bias Detection & Fairness Metrics · 50 min · Python · Fairlearn · scikit-learn

Building a Bias Audit with Fairlearn

The audit is the artifact. The artifact is the evidence.

Hiring signal: Fairlearn is the most commonly required fairness tool in RAI job postings. Being able to build a reusable audit module that produces structured JSON and visualizations — not just run a notebook — is what separates an engineer from a researcher.

What you will learn

Use Fairlearn's MetricFrame to compute subgroup metrics
Build a Fairlearn dashboard for model comparison on fairness/accuracy tradeoff
Create a reusable bias audit module that outputs structured JSON
Visualize subgroup disparities with matplotlib

The Problem

In Lesson 2, you computed fairness metrics from scratch. That's important for understanding — but in production, you need a reusable tool that any engineer on your team can run against any model. That tool needs to produce structured output (JSON for pipelines), visualizations (for presentations), and be composable (so it can be part of a larger audit pipeline).

Fairlearn is the industry standard for this. It's used by Microsoft, Accenture, and the U.S. government. It appears in 90% of RAI job postings. And its MetricFrame API is designed exactly for building reusable bias audit modules.

Fairlearn's MetricFrame

The core of Fairlearn is MetricFrame — a data structure that computes any scikit-learn metric across subgroups defined by protected attributes.

from fairlearn.metrics import MetricFrame
from sklearn.metrics import accuracy_score, selection_rate

# One call computes the metric for every subgroup
mf = MetricFrame(
    metrics={"accuracy": accuracy_score, "selection_rate": selection_rate},
    y_true=y_true,
    y_pred=y_pred,
    sensitive_features=groups,
)

# Access per-group results
print(mf.by_group)  # DataFrame: rows=groups, columns=metrics

# Compute disparities
print(mf.difference())  # max - min for each metric
print(mf.ratio())       # min / max for each metric

The power of MetricFrame is that it works with any scikit-learn metric — accuracy, precision, recall, F1, false positive rate, etc. You can compute dozens of metrics across dozens of subgroups in a single call.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Building a Reusable Audit Module, What's Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.