Phase 3: Adversarial Machine Learning · 55 min · PyTorch
Model Inversion & Membership Inference Attacks
You never touched the training data, and the model still told you who was in it.
Real-world application: Privacy attacks against ML models are a named risk category in NIST AI RMF and a required control area under GDPR/HIPAA-adjacent data protection regimes for any team that fine-tunes on sensitive data (health records, financial records, proprietary customer data). Being able to run a shadow-model membership inference attack and explain the train/test overfitting gap it exploits is a concrete, demonstrable skill for AI Security Engineer and privacy-engineering roles, and is the same technique regulators and auditors use to test whether a released model leaks its training set.
What you will learn
- Implement a model inversion attack that reconstructs a representative training input for a target class using only gradient ascent on a trained model's confidence output
- Implement a shadow-model membership inference attack, training an attack model on shadow-model in/out confidence data and applying it to a real target model
- Explain why the train/test accuracy gap (overfitting) is the mechanism that makes membership inference possible, and why it succeeds well above a 50% random-guess baseline
- Describe the privacy and regulatory implications of both attacks for models trained on sensitive or proprietary data
The Problem
Every attack so far in this phase has assumed the attacker wants to change what the model does — flip a prediction (evasion), or corrupt what it learns (poisoning). This lesson is about attacks that don't touch the model's behavior at all. They exploit what the model's outputs reveal — about the specific people, records, and documents that trained it.
In 2016, Reza Shokri, Congzheng Song, Vitaly Shmatikov, and colleagues published Membership Inference Attacks Against Machine Learning Models (Shokri et al., 2016, arXiv:1610.05820), showing that an attacker with nothing but query access to a trained model's confidence scores — no training data, no weights, no gradients in the black-box case — could determine, well above a 50% coin flip, whether a specific record was used to train that model. Around the same time, Fredrikson et al. showed a related but distinct attack: given white-box access to a trained model, you can run gradient ascent on the model's own confidence output and reconstruct a representative input for a target class, without ever reading a single training example.
Both attacks matter because they break an assumption a lot of teams still make implicitly: "we don't publish the training data, so the training data is private." A model trained on medical records, financial histories, or proprietary customer data and then exposed via API is not automatically privacy-preserving just because the raw dataset stays locked away. The model itself — specifically, the way it behaves differently on data it memorized versus data it merely generalizes to — is a side channel. Under GDPR, HIPAA-adjacent frameworks, and NIST's AI Risk Management Framework, that side channel is a named, auditable risk category, not a theoretical curiosity.
Model inversion: reconstructing training inputs from outputs
Model inversion asks: if I have white-box access to a trained classifier and I want to know "what does a typical example of class c look like to this model," can I get an answer without ever seeing the training set?
The attack is direct optimization. Start from random noise x, freeze the model's weights, and run gradient ascent to maximize the model's own predicted log-probability for the target class:
x* = argmax_x log P(y = c | x; theta)
At each step, compute grad_x log P(y=c | x; theta) and take an ascent step (Adam works fine here — this isn't a fixed-budget attack like FGSM, it's an unconstrained optimization run to convergence). A small L2 penalty toward the origin acts as a mild prior, keeping the reconstruction from wandering to an unbounded, meaningless point in input space — Fredrikson et al.'s original formulation uses an analogous prior (a natural-image smoothness term, in their case) for the same reason.
What makes this attack work is a fact worth sitting with: the model's decision boundary is itself a compressed description of the training distribution. A model didn't have to store training examples verbatim to leak information about them — the boundary it learned to draw between classes was shaped by those examples, and gradient ascent on the model's confidence is just walking uphill on that boundary until you land near where the real class-conditional density was concentrated. In this lesson's demo, gradient ascent on a target class's confidence reconstructs a point roughly 23-56% closer to the true (attacker-inaccessible) class centroid than a random guess — with zero training examples ever read.
The attacker never needs the training data — that's the whole point
Both attacks in this lesson work purely from query access to a trained model's outputs (confidence scores, or gradients if white-box). This is what makes them a model-level privacy property, not an infrastructure access-control problem. Locking down the database, encrypting data at rest, and restricting who can query the training pipeline does nothing to prevent either attack — a model behind a perfectly secured database can still leak information about its own training set through nothing but ordinary prediction requests.
Membership inference: the shadow-model approach
Model inversion answers "what does a class look like." Membership inference answers a sharper, more consequential question: was this specific record used to train the model? Given a real record — "was patient X's file in the training set of this diagnostic model?" — this is the question that turns into a genuine privacy violation, because a "yes" answer can itself be sensitive information (e.g., confirming someone was a patient at a specific clinic that trains models on its own patient population).
The key insight from Shokri et al. is turning an unsupervised problem into a supervised one. The attacker cannot directly label which of the target's real training records were "in" versus "out" — that's exactly what they're trying to determine. But the attacker can train their own shadow models on data drawn from the same general distribution as the target's training data (a distributional assumption, not access to the actual training set). Because the attacker trained the shadow models themselves, they know with certainty which records were in each shadow model's training set and which were held out. That gives them a labeled dataset:
(confidence_vector, membership_label) pairs, membership_label in {in, out}
collected by querying each shadow model on its own training data (label = "in") and on fresh, never-seen data from the same distribution (label = "out"). An attack model — here, a small binary classifier — is trained on this labeled data to map a confidence vector to a membership prediction. Once trained, the attack model is pointed at the real target model's confidence outputs and used to guess, per record, whether that record was a training member.
Why does this transfer from shadow models to the real target at all? Because the signal the attack model learns isn't specific to any one model's weights — it's a general behavioral pattern of overfitting: models tend to produce more confident, lower-entropy predictions on examples they memorized during training than on examples they only generalize to. Any model trained with enough capacity relative to its dataset size exhibits some version of this gap, and the attack model learns to detect the gap's signature in the confidence vector itself (which classes get near-zero probability, how peaked the distribution is, and so on) rather than memorizing anything specific to the shadow models.
In the lesson's demo, the target model reaches 100% train accuracy but only 71% test accuracy on a small, deliberately overfitting-prone dataset. A colleague argues this is an unrealistic setup that inflates the membership inference attack's apparent effectiveness. How should you respond?
Small, sensitive, high-value datasets fine-tuned into capacity-rich models are not a contrived edge case -- they are the realistic scenario membership inference is most relevant to (a hospital fine-tuning on a few hundred patient records, a company fine-tuning on proprietary customer data). Larger, more heavily regularized training runs typically show a smaller train/test gap and a correspondingly weaker membership inference signal, but that's a difference in degree, not in kind: the same mechanism (overfitting produces detectably different confidence behavior on members vs. non-members) is present any time a model has more capacity than its training data can constrain, which is common in real fine-tuning workflows.
Privacy implications and regulatory relevance
Both attacks point to the same underlying defense principle: reduce the behavioral gap between memorized and generalized examples.
| Mitigation | What it targets | Limitation |
|---|
| Stronger regularization (weight decay, dropout, early stopping) | Reduces overfitting, shrinking the train/test confidence gap both attacks exploit | Weakens but does not eliminate the gap; determined attackers with enough shadow-model queries can still find residual signal |
| Differential privacy (DP-SGD) during training | Mathematically bounds how much any single training example can influence the final model's parameters | Real privacy/utility trade-off — meaningful DP guarantees typically cost measurable accuracy |
| Reduced output granularity (return only top-1 label, not full confidence vector) | Removes the fine-grained confidence signal both attacks rely on | Blunts but doesn't fully stop attacks that only need the top label plus repeated queries |
| Query rate limiting / anomaly detection on query patterns | Raises the cost of collecting enough queries to train an effective attack or shadow models | Doesn't help against a single well-crafted batch of queries within normal usage patterns |
None of these are a silencing "fix" — rounding confidence scores to fewer decimal places, for instance, is a superficial change that raises the bar only slightly rather than closing the gap; the attack model can often still separate members from non-members from coarser signal. The mitigations that actually work are the ones that change the model's training behavior (regularization, DP) or restrict the information exposed at inference (output granularity, rate limiting) — not cosmetic changes to output formatting.
A team defends against membership inference by rounding all returned confidence scores to one decimal place, arguing this removes the fine-grained signal the attack model needs. Why is this mitigation weaker than differential privacy (DP-SGD), even though both appear in the same mitigation table?
This is the general pattern behind why some mitigations in the table are "cosmetic" and others are structural: rounding confidence scores changes the output format but leaves the underlying cause untouched — the model still behaves detectably differently on memorized versus generalized examples, and an attack model trained on coarser signal (or even just the top-1 label across repeated queries) can often still find that gap. DP-SGD instead constrains what the model is allowed to learn about any individual training example during training itself, which shrinks the memorization/generalization behavioral gap that both model inversion and membership inference exploit at its source. That's why the table lists DP-SGD's limitation as a genuine accuracy/utility trade-off, while rounding's limitation is that it barely raises the bar at all.
This is why NIST's AI Risk Management Framework and GDPR-adjacent data protection guidance both treat "does this model leak membership or reconstruction information about its training set" as a question that has to be actively tested, not assumed away because the training data itself is stored securely. If your team fine-tunes on health records, financial data, or any dataset where "was this record used to train the model" is itself sensitive, running the attack in this lesson against your own model — before an auditor or attacker does — is the concrete, demonstrable version of that control.
Build
Build it
Before the full PyTorch version, here's the membership-inference intuition on a toy model: a model that "memorized" its training set produces sharper (lower-entropy) confidence than one that only generalizes.
import math
import random
random.seed(0)
def entropy(probs):
return -sum(p * math.log(p + 1e-9) for p in probs)
def confidence_vector(is_member):
# Members (memorized) get a sharp, confident distribution.
# Non-members (generalized) get a flatter, less confident one.
if is_member:
top = random.uniform(0.85, 0.99)
else:
top = random.uniform(0.4, 0.7)
rest = (1 - top) / 2
return [top, rest, rest]
members = [confidence_vector(True) for _ in range(50)]
nonmembers = [confidence_vector(False) for _ in range(50)]
member_entropy = sum(entropy(v) for v in members) / len(members)
nonmember_entropy = sum(entropy(v) for v in nonmembers) / len(nonmembers)
print(f"Mean entropy on training members: {member_entropy:.3f}")
print(f"Mean entropy on non-members (test): {nonmember_entropy:.3f}")
# A simple attack: guess "member" if entropy is below the midpoint threshold.
threshold = (member_entropy + nonmember_entropy) / 2
correct = sum(1 for v in members if entropy(v) < threshold)
correct += sum(1 for v in nonmembers if entropy(v) >= threshold)
total = len(members) + len(nonmembers)
print(f"Simple entropy-threshold attack accuracy: {correct/total:.0%} (baseline: 50%)")
The code artifact lesson_03_03_inversion_membership.py implements both attacks from scratch against a small PyTorch MLP trained on synthetic tabular data (class-conditional Gaussians in 12 dimensions), so every step of the attack math is visible:
model_inversion_attack(): gradient ascent on log P(y=c | x) from random noise, with an L2 prior, evaluated against the true (attacker-inaccessible) per-class centroid and a random-guess baselinetrain_shadow_models(): trains 8 shadow models on independent samples from the population distribution, collecting labeled (confidence vector, in/out) pairs from eachAttackModel + train_attack_model(): a small binary classifier trained on the shadow data to predict membership from a confidence vectormembership_inference_accuracy(): applies the trained attack model to the real target's actual train and test sets, reporting accuracy against the 50% random-guess baseline
"""
lesson_03_03_inversion_membership.py
Two privacy attacks implemented from scratch against a small PyTorch MLP
classifier trained on synthetic tabular data. Neither attack requires
modifying the training pipeline -- both work purely from query access to
a trained model's outputs.
1. Model inversion (Fredrikson et al., 2015-style gradient reconstruction):
given only white-box access to a trained classifier and a target class
label, reconstruct a representative input for that class by gradient
ascent on the model's predicted class-confidence, starting from noise.
No training data is ever touched directly -- the reconstruction is
built entirely from the gradient signal the model exposes. We measure
reconstruction quality as distance to the true (attacker-inaccessible)
per-class feature centroid, versus a random-guess baseline.
2. Membership inference via shadow models (Shokri et al., 2016 --
"Membership Inference Attacks Against Machine Learning Models"):
https://arxiv.org/abs/1610.05820
an attacker who can query a target model's confidence scores, and who
knows the general data distribution (but not the exact training set),
trains several "shadow" models on data drawn from that same
distribution. Because the attacker trained the shadow models, they know
exactly which examples were "in" each shadow model's training set and
which were "out." This labeled (confidence_vector -> in/out) dataset
trains an attack model that is then pointed at the real target model to
guess whether specific records were part of ITS training set.
Metrics reported:
- Model inversion: L2 distance from the reconstructed input to the true
per-class centroid, compared against a random-vector baseline distance
- Membership inference: attack accuracy on the target model's real train
set (members) vs. real test set (non-members), compared to a 50%
random-guess baseline, and correlated with the target's train/test
accuracy gap (overfitting is what makes membership inference work)
Usage:
python lesson_03_03_inversion_membership.py
Requirements:
pip install torch
"""
from __future__ import annotations
from dataclasses import dataclass
import torch
import torch.nn as nn
import torch.nn.functional as F
torch.manual_seed(0)
NUM_FEATURES = 12
NUM_CLASSES = 3
CLASS_SEP = 2.5 # distance between per-class Gaussian centroids
# ---------------------------------------------------------------------------
# Synthetic tabular dataset: class-conditional Gaussians in NUM_FEATURES dims
# ---------------------------------------------------------------------------
def _class_centroid(label: int, num_features: int = NUM_FEATURES) -> torch.Tensor:
"""A fixed, deterministic centroid per class, spread out along the first
few feature dimensions so classes are separable but not trivially so."""
g = torch.Generator().manual_seed(1000 + label)
direction = torch.randn(num_features, generator=g)
direction = direction / direction.norm()
return direction * CLASS_SEP
def sample_records(n_per_class: int, noise_std: float = 1.6, seed: int | None = None) -> tuple[torch.Tensor, torch.Tensor]:
"""Draw n_per_class fresh samples per class from the population distribution."""
if seed is not None:
torch.manual_seed(seed)
xs, ys = [], []
for label in range(NUM_CLASSES):
centroid = _class_centroid(label)
noise = torch.randn(n_per_class, NUM_FEATURES) * noise_std
xs.append(centroid.unsqueeze(0) + noise)
ys.append(torch.full((n_per_class,), label, dtype=torch.long))
x = torch.cat(xs)
y = torch.cat(ys)
perm = torch.randperm(len(y))
return x[perm], y[perm]
# ---------------------------------------------------------------------------
# Small MLP classifier
# ---------------------------------------------------------------------------
class TinyMLP(nn.Module):
def __init__(self, num_features: int = NUM_FEATURES, num_classes: int = NUM_CLASSES):
super().__init__()
self.net = nn.Sequential(
nn.Linear(num_features, 64),
nn.ReLU(),
nn.Linear(64, 64),
nn.ReLU(),
nn.Linear(64, num_classes),
)
def forward(self, x: torch.Tensor) -> torch.Tensor:
return self.net(x)
def train_model(model: nn.Module, x: torch.Tensor, y: torch.Tensor, epochs: int = 150, lr: float = 1e-2) -> None:
optimizer = torch.optim.Adam(model.parameters(), lr=lr)
model.train()
for _ in range(epochs):
optimizer.zero_grad()
loss = F.cross_entropy(model(x), y)
loss.backward()
optimizer.step()
@torch.no_grad()
def accuracy(model: nn.Module, x: torch.Tensor, y: torch.Tensor) -> float:
model.eval()
return (model(x).argmax(1) == y).float().mean().item()
# ---------------------------------------------------------------------------
# Attack 1: Model inversion via gradient ascent on class confidence
# ---------------------------------------------------------------------------
def model_inversion_attack(
model: nn.Module,
target_class: int,
num_features: int = NUM_FEATURES,
steps: int = 300,
lr: float = 0.1,
l2_weight: float = 1e-3,
) -> torch.Tensor:
"""Reconstruct a representative input for `target_class` using only
white-box gradient access to the trained model -- no training data.
We maximize log P(target_class | x) w.r.t. x directly, starting from
random noise, with a small L2 penalty toward the origin acting as a
mild prior that keeps the reconstruction from diverging to an
unbounded, meaningless vector (Fredrikson et al. use an analogous prior
-- in their case a natural-image prior -- for the same reason: raw
gradient ascent on confidence alone can wander far from anything
resembling real data).
"""
model.eval()
for p in model.parameters():
p.requires_grad_(False)
x = torch.randn(1, num_features, requires_grad=True)
optimizer = torch.optim.Adam([x], lr=lr)
target = torch.tensor([target_class])
for _ in range(steps):
optimizer.zero_grad()
logits = model(x)
log_prob = F.log_softmax(logits, dim=1)[0, target_class]
loss = -log_prob + l2_weight * x.pow(2).sum()
loss.backward()
optimizer.step()
for p in model.parameters():
p.requires_grad_(True)
return x.detach()[0]
@dataclass
class InversionReport:
target_class: int
dist_to_true_centroid: float
dist_random_baseline: float
def evaluate_inversion(model: nn.Module, target_class: int) -> InversionReport:
reconstructed = model_inversion_attack(model, target_class)
true_centroid = _class_centroid(target_class)
dist_reconstructed = (reconstructed - true_centroid).norm().item()
baseline_dists = [
(torch.randn(NUM_FEATURES) - true_centroid).norm().item() for _ in range(200)
]
dist_random = sum(baseline_dists) / len(baseline_dists)
return InversionReport(target_class, dist_reconstructed, dist_random)
# ---------------------------------------------------------------------------
# Attack 2: Membership inference via shadow models (Shokri et al., 2016)
# ---------------------------------------------------------------------------
def train_shadow_models(
n_shadows: int, n_per_class_train: int, n_per_class_out: int
) -> tuple[torch.Tensor, torch.Tensor]:
"""Train `n_shadows` shadow models on independent samples from the same
population distribution the real target was trained on. For each shadow
model, collect its confidence vector on both its own training data
(label 1 = "in") and on fresh held-out data it never saw (label 0 =
"out"). This becomes the attack model's training set.
"""
attack_features = []
attack_labels = []
for i in range(n_shadows):
x_in, y_in = sample_records(n_per_class_train, seed=2000 + i)
x_out, y_out = sample_records(n_per_class_out, seed=5000 + i)
shadow = TinyMLP()
train_model(shadow, x_in, y_in, epochs=600, lr=5e-3)
with torch.no_grad():
conf_in = F.softmax(shadow(x_in), dim=1)
conf_out = F.softmax(shadow(x_out), dim=1)
attack_features.append(conf_in)
attack_labels.append(torch.ones(len(x_in)))
attack_features.append(conf_out)
attack_labels.append(torch.zeros(len(x_out)))
return torch.cat(attack_features), torch.cat(attack_labels)
class AttackModel(nn.Module):
"""A small binary classifier: confidence vector -> P(this record was a
training member of the model that produced this confidence vector)."""
def __init__(self, num_classes: int = NUM_CLASSES):
super().__init__()
self.net = nn.Sequential(
nn.Linear(num_classes, 16),
nn.ReLU(),
nn.Linear(16, 1),
)
def forward(self, x: torch.Tensor) -> torch.Tensor:
return self.net(x).squeeze(-1)
def train_attack_model(features: torch.Tensor, labels: torch.Tensor, epochs: int = 200) -> AttackModel:
attack_model = AttackModel()
optimizer = torch.optim.Adam(attack_model.parameters(), lr=5e-3)
for _ in range(epochs):
optimizer.zero_grad()
logits = attack_model(features)
loss = F.binary_cross_entropy_with_logits(logits, labels)
loss.backward()
optimizer.step()
return attack_model
@torch.no_grad()
def membership_inference_accuracy(
attack_model: AttackModel, target: nn.Module, x_members: torch.Tensor, x_nonmembers: torch.Tensor
) -> float:
target.eval()
conf_members = F.softmax(target(x_members), dim=1)
conf_nonmembers = F.softmax(target(x_nonmembers), dim=1)
pred_members = (torch.sigmoid(attack_model(conf_members)) > 0.5).float()
pred_nonmembers = (torch.sigmoid(attack_model(conf_nonmembers)) > 0.5).float()
correct = pred_members.sum().item() + (1 - pred_nonmembers).sum().item()
total = len(x_members) + len(x_nonmembers)
return correct / total
def main() -> None:
print("MODEL INVERSION & MEMBERSHIP INFERENCE ATTACKS")
print("=" * 60)
# --- Train the target ("victim") model on its private training set ---
# A small training set + a comparatively high-capacity network + many epochs
# is a deliberately overfitting-prone setup: it mirrors the realistic case
# membership inference targets (small, sensitive datasets -- clinical
# records, proprietary customer data -- fine-tuned into a model that has
# more than enough capacity to memorize individual examples).
x_train, y_train = sample_records(n_per_class=30, seed=42)
x_test, y_test = sample_records(n_per_class=150, seed=99) # never seen by target -> non-members
target = TinyMLP()
train_model(target, x_train, y_train, epochs=600, lr=5e-3) # trained to convergence -> memorizes
train_acc = accuracy(target, x_train, y_train)
test_acc = accuracy(target, x_test, y_test)
print(f"Target model train accuracy: {train_acc:.1%}")
print(f"Target model test accuracy: {test_acc:.1%}")
print(f"Train/test gap (overfitting signal): {train_acc - test_acc:.1%}\n")
# --- Attack 1: Model inversion ---
print("Attack 1: Model inversion (reconstruct a representative input per class)")
print("-" * 60)
inversion_reports = [evaluate_inversion(target, c) for c in range(NUM_CLASSES)]
for r in inversion_reports:
improvement = (r.dist_random_baseline - r.dist_to_true_centroid) / r.dist_random_baseline
print(
f" class {r.target_class}: reconstruction dist to true centroid={r.dist_to_true_centroid:.3f} "
f"random-baseline dist={r.dist_random_baseline:.3f} "
f"({improvement:.0%} closer than a random guess)"
)
print(
"\nTakeaway: gradient ascent on the model's own confidence output, with no access to any "
"training example, reliably reconstructs a point much closer to the true class centroid "
"than a random guess -- the model's decision boundary itself leaks information about the "
"data distribution it was trained on."
)
# --- Attack 2: Membership inference ---
print("\nAttack 2: Membership inference via shadow models (Shokri et al., 2016)")
print("-" * 60)
print("Training 8 shadow models on independent samples from the population distribution...")
attack_features, attack_labels = train_shadow_models(
n_shadows=8, n_per_class_train=30, n_per_class_out=60
)
attack_model = train_attack_model(attack_features, attack_labels)
mia_accuracy = membership_inference_accuracy(attack_model, target, x_train, x_test)
print(f"\nMembership inference attack accuracy against the real target model: {mia_accuracy:.1%}")
print("Random-guess baseline: 50.0%")
print(
f"Target model's train/test accuracy gap was {train_acc - test_acc:.1%} -- "
"membership inference exploits exactly this gap: an overfit model produces "
"systematically more confident, lower-entropy predictions on examples it memorized "
"during training than on examples it merely generalizes to."
)
print(
"\nTakeaway: an attacker who never sees the target's training data, and only has "
"query access plus knowledge of the general data distribution, can determine -- "
"well above chance -- whether a specific record was used to train the model. This is "
"why membership inference is a real regulatory concern (e.g., under GDPR/HIPAA-style "
"data protection regimes): merely publishing model outputs, without publishing the "
"training data, does not guarantee training-set privacy."
)
if __name__ == "__main__":
main()
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers Check Yourself — plus a hands-on lab, quiz, and project artifact.
Create a free account — the opening phases of 29 of 35 courses are free, no credit card.
Browse all courses · View pricing · DeVenture Academy