Phase 2: Bias Detection & Fairness Metrics · 45 min · Python · AIF360 · scikit-learn

Advanced Bias Detection with AIF360

Fairlearn tells you there's a gap. AIF360 tells you how deep it goes.

Hiring signal: AIF360 appears in 60%+ of RAI postings, often alongside Fairlearn. Being able to explain when to use AIF360 vs. Fairlearn — and combine both in a single audit — shows depth beyond a single-tool skill set.

What you will learn

Use AIF360's BinaryLabelDatasetMetric and ClassificationMetric for bias analysis
Compute consistency metric and generalized entropy index
Determine when to use AIF360 vs. Fairlearn based on the audit requirements
Combine Fairlearn and AIF360 metrics in a single bias report

The Problem

Fairlearn is excellent for computing standard fairness metrics across groups. But some bias detection tasks require more specialized tools:

Consistency: How similar are predictions for individuals who are similar except for their protected attribute?
Generalized entropy index: How unequal is the distribution of predictions across individuals?
Rich subgroup analysis: When you need to analyze intersections (e.g., Black women vs. Black men vs. white women vs. white men)
Pre/post-mitigation comparison: When you need to compare bias before and after applying a mitigation algorithm

IBM's AI Fairness 360 (AIF360) is built for these cases. It provides a different API than Fairlearn — based on BinaryLabelDataset and ClassificationMetric objects — but it covers a wider range of metrics and mitigation algorithms.

AIF360 vs. Fairlearn: When to Use Which

Criterion	Fairlearn	AIF360
Ease of use	Simpler API, sklearn-compatible	More complex, custom data structures
Metric variety	Standard metrics (DP, EO, DI)	40+ metrics including consistency, entropy
Mitigation algorithms	3 (reweighing, exponentiated gradients, threshold optimization)	10+ including adversarial debiasing, reject option classification
Intersectional analysis	Supported via multi-index	Native support via multiple protected attributes
Industry adoption	More common in job postings	Common in research and enterprise
Best for	Quick audits, dashboards, production pipelines	Deep analysis, research-grade metrics, complex mitigation

The practical pattern: Use Fairlearn for the standard audit (Phase 2, Lesson 3). Use AIF360 when you need deeper analysis or more advanced mitigation. Combine both in a single report for comprehensive coverage.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers AIF360's Core API, Building the AIF360 Extension, What's Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.