The Problem
In 2020, researchers showed that GPT-2 could be prompted to regurgitate verbatim passages from its training data — including names, phone numbers, and email addresses that were in the training corpus. The model wasn't supposed to memorize this. It was supposed to learn patterns. But neural networks don't distinguish between "learning a pattern" and "memorizing a specific example" — and when they overfit, they do both.
This is the fundamental privacy risk of ML: models leak information about their training data. The more a model overfits, the more it memorizes, and the more an attacker can extract.
The 3 privacy attack types
| Attack | What It Reveals | How It Works | Real-World Impact |
|---|
| Membership Inference | Whether a specific individual's data was in the training set | Train a "shadow model" to distinguish member vs. non-member predictions | "Your hospital records were used to train this model" — reveals medical status |
| Model Inversion | Reconstructs training data features from model outputs | Optimize inputs to match known outputs | Reconstructing faces from a facial recognition model |
| Data Extraction | Extracts verbatim training data from generative models | Prompt the model and filter outputs that match training data patterns | GPT-2 regurgitating PII from training corpus |
Membership inference attacks
The most practical attack for RAI engineers to understand. Here's the intuition:
- A model trained on a dataset behaves differently for examples it has seen (members) vs. examples it hasn't (non-members)
- Members tend to get higher confidence predictions (the model "remembers" them)
- An attacker trains a "shadow model" that mimics the target model's behavior, then trains an "attack model" to distinguish member vs. non-member confidence patterns
- The attack model is then applied to the target model's outputs
"""
lesson_03_01_membership_inference.py
Demonstrates a membership inference attack on a trained model.
"""
import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, roc_auc_score
def membership_inference_attack(target_model, X_train, y_train, X_test, y_test):
"""
Simple membership inference attack using prediction confidence.
The intuition: the target model is more confident on training data
(members) than on test data (non-members). We use prediction
probability as the membership signal.
Args:
target_model: Trained model with predict_proba
X_train, y_train: Training data (members)
X_test, y_test: Test data (non-members)
Returns:
Dict with attack results
"""
# Get prediction probabilities for both sets
train_proba = target_model.predict_proba(X_train)[:, 1]
test_proba = target_model.predict_proba(X_test)[:, 1]
# Membership signal: max confidence (higher = more likely a member)
train_confidence = np.maximum(train_proba, 1 - train_proba)
test_confidence = np.maximum(test_proba, 1 - test_proba)
# Labels: 1 = member (training), 0 = non-member (test)
member_labels = np.concatenate([np.ones(len(train_confidence)),
np.zeros(len(test_confidence))])
member_scores = np.concatenate([train_confidence, test_confidence])
# AUC of the attack: how well can we distinguish members from non-members?
attack_auc = roc_auc_score(member_labels, member_scores)
# Accuracy at a threshold
threshold = np.median(member_scores)
predicted_members = (member_scores >= threshold).astype(int)
attack_accuracy = accuracy_score(member_labels, predicted_members)
return {
"attack_auc": round(attack_auc, 4),
"attack_accuracy": round(attack_accuracy, 4),
"avg_member_confidence": round(float(train_confidence.mean()), 4),
"avg_nonmember_confidence": round(float(test_confidence.mean()), 4),
"privacy_risk": "High" if attack_auc > 0.65 else "Moderate" if attack_auc > 0.55 else "Low",
}
def demonstrate_overfitting_impact():
"""Show how overfitting increases membership inference vulnerability."""
np.random.seed(42)
n = 2000
X = np.random.randn(n, 10)
y = (X[:, 0] + X[:, 1] > 0).astype(int)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=42)
results = []
# Vary model complexity to control overfitting
for depth in [3, 5, 10, 20, None]:
model = RandomForestClassifier(
n_estimators=100, max_depth=depth, random_state=42
)
model.fit(X_train, y_train)
train_acc = model.score(X_train, y_train)
test_acc = model.score(X_test, y_test)
attack = membership_inference_attack(model, X_train, y_train, X_test, y_test)
results.append({
"max_depth": str(depth),
"train_accuracy": round(train_acc, 4),
"test_accuracy": round(test_acc, 4),
"overfit_gap": round(train_acc - test_acc, 4),
"attack_auc": attack["attack_auc"],
"privacy_risk": attack["privacy_risk"],
})
return results
if __name__ == "__main__":
print("Demonstrating membership inference attack...")
print("\nEffect of model complexity (overfitting) on privacy risk:\n")
results = demonstrate_overfitting_impact()
print(f"{'Max Depth':<12} {'Train Acc':<12} {'Test Acc':<12} {'Overfit Gap':<14} {'Attack AUC':<12} {'Risk':<10}")
print("-" * 72)
for r in results:
print(f"{r['max_depth']:<12} {r['train_accuracy']:<12} {r['test_accuracy']:<12} "
f"{r['overfit_gap']:<14} {r['attack_auc']:<12} {r['privacy_risk']:<10}")
print("\nKey insight: As overfitting increases (larger overfit gap),")
print("the attack AUC increases — the model leaks more training data information.")
# Generate report
lines = ["# Membership Inference Attack Analysis\n"]
lines.append("## Overfitting vs. Privacy Risk\n")
lines.append("| Max Depth | Train Acc | Test Acc | Overfit Gap | Attack AUC | Privacy Risk |")
lines.append("|---|---|---|---|---|---|")
for r in results:
lines.append(f"| {r['max_depth']} | {r['train_accuracy']} | {r['test_accuracy']} | "
f"{r['overfit_gap']} | {r['attack_auc']} | {r['privacy_risk']} |")
lines.append("\n## Key Finding")
lines.append("As model complexity increases (deeper trees), the overfit gap grows and the "
"membership inference attack AUC increases. This means the model is leaking more "
"information about its training data — an attacker can more easily determine whether "
"a specific individual's data was used in training.\n")
lines.append("## Mitigation")
lines.append("- Regularization (limit tree depth, increase min_samples_leaf)")
lines.append("- Differential privacy (add noise during training — see Lesson 2)")
lines.append("- Dropout and early stopping for neural networks")
with open("membership_inference_report.md", "w") as f:
f.write("\n".join(lines))
print("\nReport saved to membership_inference_report.md")
Your model has a train accuracy of 99% and test accuracy of 85%. The membership inference attack AUC is 0.72. What does this mean, and what should you do?
An attack AUC of 0.72 is significantly above the 0.50 baseline (random guessing). It means the model's behavior is noticeably different for training examples vs. unseen examples — an attacker can exploit this difference. In a healthcare context, this could reveal that a specific person was a patient at a hospital (their data was in the training set), which is a GDPR violation. The 14% overfitting gap is the root cause — the model memorized training examples rather than learning generalizable patterns. The fix is to reduce overfitting (regularization, simpler models, early stopping) and/or add differential privacy noise during training to mask the difference between members and non-members.