Phase 4: Privacy-Preserving AI · 40 min · Python · scikit-learn · numpy

How Models Leak Training Data — Privacy Attacks on AI

Your model memorized the training data. An attacker can extract it.

Hiring signal: Understanding how models leak training data — through membership inference, model inversion, and extraction attacks — is a core privacy risk assessment skill. RAI engineers must demonstrate this threat model in interviews.

What you will learn

Explain membership inference attacks and how they reveal whether an individual's data was in the training set
Explain model inversion attacks and how they reconstruct training data from model outputs
Demonstrate a membership inference attack on a trained model
Articulate why overfitting increases privacy risk

The Problem

In 2020, researchers showed that GPT-2 could be prompted to regurgitate verbatim passages from its training data — including names, phone numbers, and email addresses that were in the training corpus. The model wasn't supposed to memorize this. It was supposed to learn patterns. But neural networks don't distinguish between "learning a pattern" and "memorizing a specific example" — and when they overfit, they do both.

This is the fundamental privacy risk of ML: models leak information about their training data. The more a model overfits, the more it memorizes, and the more an attacker can extract.

The 3 Privacy Attack Types

Attack	What It Reveals	How It Works	Real-World Impact
Membership Inference	Whether a specific individual's data was in the training set	Train a "shadow model" to distinguish member vs. non-member predictions	"Your hospital records were used to train this model" — reveals medical status
Model Inversion	Reconstructs training data features from model outputs	Optimize inputs to match known outputs	Reconstructing faces from a facial recognition model
Data Extraction	Extracts verbatim training data from generative models	Prompt the model and filter outputs that match training data patterns	GPT-2 regurgitating PII from training corpus

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Membership Inference Attacks, What's Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.