Phase 3: Explainability & Transparency · 35 min · Python · scikit-learn

The Black Box Problem — Why Explainability Matters

A model that can't explain itself can't be trusted, audited, or deployed in high-stakes contexts.

Real-world application: RAI engineers must be able to articulate the regulatory drivers for explainability (EU AI Act Article 13, GDPR right to explanation) and demonstrate why black-box models are insufficient for high-risk use cases without tooling.

What you will learn

Explain the difference between global and local explainability
Identify the regulatory requirements for explainability (EU AI Act Art. 13, GDPR)
Demonstrate that a trained model's predictions are opaque without tooling
Articulate the explainability-performance tradeoff

Objective

Test whether an explanation is faithful, understandable, decision-relevant, and appropriate for its actual audience.

The Problem

A loan applicant is denied credit by an automated system. They ask: "Why was I denied?" The bank's answer: "The model said so." This is not just bad customer service — it's a legal violation. Under GDPR Article 22, individuals have the right to "meaningful information about the logic involved" in automated decision-making. Under the EU AI Act Article 13, high-risk AI systems must provide transparency and allow users to interpret the system's output.

A gradient boosting model that considers 47 features to produce a single score is a black box. You cannot look at its 500 decision trees and explain why a specific applicant was denied. Without explainability tooling, the model is unusable in regulated contexts — regardless of how accurate it is.

Global vs. local explainability

Type	Question Answered	Tools	Example
Global	"Which features matter most across all predictions?"	SHAP summary plots, feature importance	"Income is the most important feature for loan approval overall"
Local	"Why was this specific prediction made for this specific person?"	SHAP waterfall/force plots, LIME	"This applicant was denied because their credit score of 620 pushed the prediction below the threshold, despite their income being favorable"

Both are necessary. Global explainability helps auditors and regulators understand the model's behavior patterns. Local explainability helps individuals understand their specific outcome — which is what GDPR and the EU AI Act require.

The regulatory drivers

Regulation	Requirement	What It Means in Practice
GDPR Article 22	Right to meaningful information about automated decision logic	You must be able to explain why a specific individual got a specific outcome
EU AI Act Article 13	Transparency for high-risk AI systems	Users must be able to interpret system outputs — global + local explanations
EU AI Act Annex IV	Technical documentation requirements	Model architecture, training data, performance metrics, validation results
NIST AI RMF MEASURE 2.11	Explainability measurement	Document what can and cannot be explained about the system's outputs

The explainability-performance tradeoff

Model Type	Performance	Explainability	When to Use
Linear/Logistic Regression	Lower	Fully transparent	When explainability is the top priority (regulated decisions)
Decision Tree	Medium	Interpretable (follow the path)	When you need a balance of both
Random Forest / Gradient Boosting	Higher	Black box (needs SHAP/LIME)	When performance matters and tooling can provide explanations
Deep Neural Networks	Highest	Deeply opaque	When performance is critical and explainability tooling is sufficient

A colleague argues "we don't need explainability tooling — we'll just use a decision tree so it's naturally interpretable." What's wrong with this approach for a high-stakes loan approval system?

The tradeoff is real: a shallow decision tree is interpretable but underperforms. A deep decision tree matches accuracy but becomes unreadable — a 20-level tree with hundreds of leaves is not "interpretable" in any meaningful sense. The modern RAI engineering approach is to use the best-performing model (gradient boosting, neural networks) and layer SHAP/LIME on top. This gives you the accuracy of a black-box model with the explainability of an interpretable one. The explanations are approximate (SHAP values are estimates, not exact decompositions), but they're good enough for regulatory compliance and stakeholder communication. Don't sacrifice accuracy for interpretability when tooling can give you both.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.