Phase 3: Adversarial Machine Learning · 55 min · PyTorch

Model Inversion & Membership Inference Attacks

You never touched the training data, and the model still told you who was in it.

Hiring signal: Privacy attacks against ML models are a named risk category in NIST AI RMF and a required control area under GDPR/HIPAA-adjacent data protection regimes for any team that fine-tunes on sensitive data (health records, financial records, proprietary customer data). Being able to run a shadow-model membership inference attack and explain the train/test overfitting gap it exploits is a concrete, demonstrable skill for AI Security Engineer and privacy-engineering roles, and is the same technique regulators and auditors use to test whether a released model leaks its training set.

What you will learn

Implement a model inversion attack that reconstructs a representative training input for a target class using only gradient ascent on a trained model's confidence output
Implement a shadow-model membership inference attack, training an attack model on shadow-model in/out confidence data and applying it to a real target model
Explain why the train/test accuracy gap (overfitting) is the mechanism that makes membership inference possible, and why it succeeds well above a 50% random-guess baseline
Describe the privacy and regulatory implications of both attacks for models trained on sensitive or proprietary data

The Problem

Every attack so far in this phase has assumed the attacker wants to change what the model does — flip a prediction (evasion), or corrupt what it learns (poisoning). This lesson is about attacks that don't touch the model's behavior at all. They exploit what the model's outputs reveal — about the specific people, records, and documents that trained it.

In 2016, Reza Shokri, Congzheng Song, Vitaly Shmatikov, and colleagues published Membership Inference Attacks Against Machine Learning Models (Shokri et al., 2016, arXiv:1610.05820), showing that an attacker with nothing but query access to a trained model's confidence scores — no training data, no weights, no gradients in the black-box case — could determine, well above a 50% coin flip, whether a specific record was used to train that model. Around the same time, Fredrikson et al. showed a related but distinct attack: given white-box access to a trained model, you can run gradient ascent on the model's own confidence output and reconstruct a representative input for a target class, without ever reading a single training example.

Both attacks matter because they break an assumption a lot of teams still make implicitly: "we don't publish the training data, so the training data is private." A model trained on medical records, financial histories, or proprietary customer data and then exposed via API is not automatically privacy-preserving just because the raw dataset stays locked away. The model itself — specifically, the way it behaves differently on data it memorized versus data it merely generalizes to — is a side channel. Under GDPR, HIPAA-adjacent frameworks, and NIST's AI Risk Management Framework, that side channel is a named, auditable risk category, not a theoretical curiosity.

Model Inversion: Reconstructing Training Inputs from Outputs

Model inversion asks: if I have white-box access to a trained classifier and I want to know "what does a typical example of class c look like to this model," can I get an answer without ever seeing the training set?

The attack is direct optimization. Start from random noise x, freeze the model's weights, and run gradient ascent to maximize the model's own predicted log-probability for the target class:

x* = argmax_x  log P(y = c | x; theta)

At each step, compute grad_x log P(y=c | x; theta) and take an ascent step (Adam works fine here — this isn't a fixed-budget attack like FGSM, it's an unconstrained optimization run to convergence). A small L2 penalty toward the origin acts as a mild prior, keeping the reconstruction from wandering to an unbounded, meaningless point in input space — Fredrikson et al.'s original formulation uses an analogous prior (a natural-image smoothness term, in their case) for the same reason.

What makes this attack work is a fact worth sitting with: the model's decision boundary is itself a compressed description of the training distribution. A model didn't have to store training examples verbatim to leak information about them — the boundary it learned to draw between classes was shaped by those examples, and gradient ascent on the model's confidence is just walking uphill on that boundary until you land near where the real class-conditional density was concentrated. In this lesson's demo, gradient ascent on a target class's confidence reconstructs a point roughly 23-56% closer to the true (attacker-inaccessible) class centroid than a random guess — with zero training examples ever read.

The attacker never needs the training data — that's the whole point
Both attacks in this lesson work purely from query access to a trained model's outputs (confidence scores, or gradients if white-box). This is what makes them a model-level privacy property, not an infrastructure access-control problem. Locking down the database, encrypting data at rest, and restricting who can query the training pipeline does nothing to prevent either attack — a model behind a perfectly secured database can still leak information about its own training set through nothing but ordinary prediction requests.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Membership Inference: The Shadow-Model Approach, Privacy Implications and Regulatory Relevance, Build It, What to Practice — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.