Phase 3: Adversarial Machine Learning · 60 min · Python · PyTorch
Poisoning Attacks — Label Flipping, Backdoors, and Clean-Label Poisoning
Evasion attacks fool a finished model at inference time; poisoning attacks corrupt the model before it's finished — and a validation accuracy check will not catch it.
What you will learn
- Implement a label-flipping poisoning attack and measure how it degrades accuracy on the targeted class before it shows up in overall accuracy
- Implement a BadNets-style backdoor/trigger-pattern poisoning attack and measure Attack Success Rate separately from clean-input accuracy
- Explain clean-label poisoning and gradient-based poisoning (e.g., gradient matching) as attacks that avoid the label-consistency signals that make basic poisoning detectable
- Describe statistical anomaly detection approaches (activation clustering, spectral signatures) used to find poisoned samples in a training set
The Problem
Every attack in the previous lesson assumed a fixed, already-trained model: the attacker perturbs an input at inference time and the model's weights never change. Poisoning attacks target something upstream of that — the training data itself. If an attacker can influence even a small fraction of what a model learns from, they can install behavior that no amount of inference-time defense will catch, because the behavior isn't a bug in the model's decision boundary — it is the decision boundary, exactly as trained.
This is not a hypothetical. Modern ML pipelines pull training data from scraped web text, crowdsourced labeling platforms, user feedback loops, and third-party vendors — every one of which is a point where an adversary with far less access than "steal the model weights" can shape what gets learned. Gu et al.'s BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain (2017) demonstrated the sharpest version of this: a model can pass every accuracy benchmark a defender runs, and still contain a targeted backdoor that activates only when the attacker's chosen trigger appears. The OWASP LLM Top 10 lists training data poisoning as its own category for exactly this reason — it's a supply-chain problem as much as an ML problem.
This lesson covers four escalating poisoning techniques — label flipping, backdoor/trigger injection, clean-label poisoning, and gradient-based poisoning — and the statistical methods used to catch them before training, since none of them show up in a normal accuracy report.
Label flipping: the simplest attack that still works
Label flipping requires no control over the data pipeline's features at all — only the labels. An attacker (or a compromised/malicious crowdsourced annotator) simply mislabels a fraction of examples from one class as another class. No pixels, no features, nothing else changes.
The reason this is worth taking seriously rather than dismissing as "too crude to matter" is what it does to aggregate accuracy. If you flip labels for a class that's a minority of the overall dataset, the damage concentrates entirely on that class's own accuracy while the overall, class-averaged accuracy barely moves — because the other classes are unaffected and dominate the average. A defender watching a single top-line accuracy number during training will see nothing wrong. Only a per-class accuracy breakdown reveals that one class has effectively collapsed. This is the first lesson poisoning research teaches: never evaluate a model's integrity using only an aggregate metric — the same principle carries through every attack in this lesson.
Backdoor / trigger injection: BadNets
Backdoor poisoning goes further than corrupting a class — it installs a targeted, dormant behavior. The attacker picks a fixed trigger pattern (in BadNets, a small pixel patch; in text models, a specific token sequence), stamps it onto a subset of training inputs, and relabels those inputs to an attacker-chosen target class. The model is then trained on a mix of clean data and triggered data.
The result is a model that behaves completely normally on clean inputs — clean-input accuracy is often statistically indistinguishable from a non-poisoned baseline — but reliably outputs the attacker's target class the instant the trigger pattern appears. This is the defining property that makes backdoors dangerous: a validation set made of clean data cannot reveal a backdoor, because the backdoor was specifically designed to stay dormant outside the trigger condition. An attacker who controls a small slice of training data (a poisoned pretraining corpus, a compromised fine-tuning dataset, a malicious contribution to a federated learning round) can plant a landmine that only they know how to detonate.
We measure this with two separate numbers, deliberately kept apart:
- Clean accuracy — performance on ordinary, trigger-free inputs. This is what a normal ML evaluation reports, and it stays high even in a heavily backdoored model.
- Attack Success Rate (ASR) — the fraction of non-target-class inputs that get classified as the target class once the trigger is stamped onto them. We exclude inputs whose true label already equals the target class, for the same reason evasion-attack ASR excludes already-misclassified examples: predicting the target class on an input that already belongs to the target class isn't evidence the trigger did anything.
Clean accuracy and backdoor ASR are nearly independent variables
Across most of a realistic poisoning-rate range, you can hold clean accuracy at 100% while backdoor ASR climbs from 0% to over 95%. These two numbers move on different axes almost entirely until the poison rate gets extreme enough to start interfering with the legitimate task. If your only integrity check on a training pipeline is "did clean validation accuracy drop," a well-executed backdoor will sail through undetected — the entire point of the attack is to leave that number untouched.
You inherit a fine-tuned image classifier from a vendor. Clean validation accuracy is 99.1%, matching the vendor's reported number exactly. A colleague argues this proves the model wasn't poisoned during fine-tuning. Is this reasoning sound?
Clean validation accuracy and backdoor presence are almost orthogonal by design. A competently installed backdoor is built to be invisible to exactly this kind of check — the attacker wants the model to look completely normal under any evaluation that doesn't include the trigger. Verifying supply-chain integrity of a fine-tuned model requires either access to and auditing of the training data (activation clustering, spectral signature analysis) or actively testing candidate trigger patterns against the deployed model — not just re-checking accuracy on the same kind of clean data the vendor already reported on.
Clean-label poisoning: removing the label-inconsistency signal
Label flipping and BadNets both leave a trace an alert human reviewer could catch by inspection: a flipped label doesn't match its image, and a triggered image visually looks like something forced the label. Clean-label poisoning removes this signal entirely. Every poisoned example keeps a label a human annotator would agree with — the attack works purely by crafting the input features (often via an imperceptible perturbation, similar in spirit to the evasion perturbations from the previous lesson) so the model learns a harmful association anyway, without ever touching a label.
The typical construction is a bilevel or feature-collision optimization: craft a perturbed version of a base image (correctly labeled) whose learned feature representation collides with, or sits unusually close to, a target class's representation in the model's internal feature space. When the model trains on this correctly-labeled-but-adversarially-perturbed example, it shifts the decision boundary in the attacker's favor near the target region — all while every individual training example passes a manual label audit. This is why clean-label poisoning is treated as the stronger threat model in adversarial ML research: it defeats the most common practical defense (manual or crowdsourced label review) by construction, not by luck.
Gradient-based poisoning
Where clean-label poisoning targets feature-space collisions, gradient-based poisoning (also called gradient matching or bilevel poisoning) attacks the training process directly. The attacker formulates poisoning as a bilevel optimization: choose a small set of poison points such that, after the victim trains on data including those points, the resulting model's gradient on a chosen target objective (e.g., misclassifying one specific test example, or degrading performance in a specific way) matches what the attacker wants — without ever needing to guess the exact effect a naive method like label flipping would have.
Concretely, gradient matching poisoning searches for poison inputs whose gradient contribution during training closely matches the gradient the attacker would need to push the model toward a target behavior, computed by differentiating through a (surrogate) training procedure. This makes the attack far more sample-efficient than random label flipping or crude trigger stamping — a gradient-matching attack can achieve a targeted misclassification with a tiny handful of poisoned points, sometimes a fraction of a percent of the training set, because every poisoned point is chosen to have maximal, precisely-aimed influence on the final trained weights rather than a diffuse, statistical one.
Detecting poisoned samples
Since none of these attacks reliably show up in clean-data accuracy, detection has to inspect the training data and the model's internal behavior on it, not just its final predictions:
- Activation clustering (Chen et al.) — for a given labeled class, cluster the model's internal activations on all examples with that label. Genuinely-labeled examples tend to form one dominant cluster; poisoned examples (which the model has learned to route through a different internal pathway triggered by the poison pattern) frequently form a separable secondary cluster, even though their assigned label looks correct.
- Spectral signatures (Tran et al.) — poisoned representations tend to have a detectable signature in the top singular vectors of the covariance matrix of per-class feature activations, because the poison consistently biases activations in a particular direction that clean examples don't share.
- Statistical outlier detection on inputs — for backdoor triggers specifically, unusually high-intensity, spatially localized pixel regions (or, in tabular/text data, unusually high-frequency exact-repeat substrings/tokens across otherwise-diverse examples) are a coarse but often effective signal, since the trigger has to be consistent across every poisoned example to reliably retrain the same backdoor.
None of these are silver bullets — clean-label and gradient-matching attacks are specifically designed to blend into the natural feature distribution, and defenses are an active research area. But they are a meaningfully stronger integrity check than "did clean accuracy drop," which is the bar most production pipelines still don't clear.
An engineer proposes detecting all poisoning attacks by simply monitoring the training loss curve for anomalous spikes. Why is this an incomplete defense against the attacks in this lesson?
Loss-curve monitoring catches attacks that are large enough to visibly perturb the aggregate optimization trajectory -- which is precisely what stealthy poisoning attacks are engineered to avoid. A gradient-matching or clean-label attack can achieve a targeted effect with a tiny fraction of the training set, contributing a small and smooth addition to the overall loss. Effective detection has to look at the training data and per-example or per-class internal representations directly (activation clustering, spectral signatures), not just an aggregate scalar loss trajectory, because aggregate metrics are exactly what these attacks are designed to leave undisturbed.
Build
Build it
Before the full PyTorch backdoor sweep, here's a toy nearest-centroid classifier that shows why clean accuracy and backdoor ASR are nearly independent, using pure Python.
import random
random.seed(0)
# Each class is a point in 2D feature space; a "trigger" shifts a poisoned
# point's features toward the attacker's target class centroid.
CLASSES = {0: (0.0, 0.0), 1: (5.0, 0.0), 2: (0.0, 5.0), 3: (5.0, 5.0)}
TARGET_CLASS = 3
TRIGGER_SHIFT = (4.5, 4.5) # pushes a poisoned point near class 3's centroid
def classify(point, centroids):
return min(centroids, key=lambda c: (point[0] - centroids[c][0]) ** 2 + (point[1] - centroids[c][1]) ** 2)
def make_point(label):
cx, cy = CLASSES[label]
return (cx + random.uniform(-0.5, 0.5), cy + random.uniform(-0.5, 0.5))
def apply_trigger(point):
return (point[0] + TRIGGER_SHIFT[0] * 0.01, point[1] + TRIGGER_SHIFT[1] * 0.01)
test_points = [(label, make_point(label)) for label in CLASSES for _ in range(20)]
clean_correct = sum(1 for label, p in test_points if classify(p, CLASSES) == label)
print(f"Clean accuracy: {clean_correct}/{len(test_points)} ({clean_correct/len(test_points):.0%})")
non_target = [(label, p) for label, p in test_points if label != TARGET_CLASS]
triggered_hits = sum(
1 for label, p in non_target if classify(apply_trigger(p), CLASSES) == TARGET_CLASS
)
print(f"Backdoor ASR on triggered non-target inputs: {triggered_hits}/{len(non_target)} "
f"({triggered_hits/len(non_target):.0%})")
print("\nA real backdoor uses a much stronger trigger signal than this toy example, "
"but the point holds: clean accuracy and ASR are separate axes.")
The code artifact lesson_03_02_poisoning_attacks.py implements both a label-flipping attack and a BadNets-style backdoor/trigger attack from scratch against the same small CNN and synthetic four-class shape dataset used in the previous lesson, measuring the exact metrics discussed above — per-class accuracy for label flipping, and clean accuracy vs. backdoor ASR across a poison-rate sweep:
"""
lesson_03_02_poisoning_attacks.py
Two training-time poisoning attacks implemented from scratch against a small
CNN trained on a synthetic image-classification task:
1. Label-flipping attack: a fraction of training labels for a source class
are flipped to a target class. No inputs are modified -- only labels.
This is the crudest poisoning attack and the easiest to detect by simply
auditing labels, but it is still used as a baseline because it requires
zero control over the feature pipeline.
2. Backdoor / trigger-pattern attack (BadNets-style; see Gu et al., 2017 --
"BadNets: Identifying Vulnerabilities in the Machine Learning Model
Supply Chain"): a small fixed pixel patch (the "trigger") is stamped onto
a subset of training images, and those images are relabeled to a fixed
target class. The model is trained on a mix of clean and triggered data.
The resulting backdoor is a targeted, attacker-chosen misclassification
that activates only when the trigger is present -- clean-input accuracy
stays high, which is exactly what makes this attack hard to catch by
watching validation accuracy alone.
Metrics reported:
- Clean test accuracy of a poisoned model vs. a cleanly-trained baseline
- Label-flipping: accuracy drop as a function of the flip rate
- Backdoor: Attack Success Rate (ASR) on triggered inputs, measured
separately from clean-input accuracy, across a sweep of poison rates
Usage:
python lesson_03_02_poisoning_attacks.py
Requirements:
pip install torch
"""
from __future__ import annotations
from dataclasses import dataclass
import torch
import torch.nn as nn
import torch.nn.functional as F
torch.manual_seed(0)
IMG_SIZE = 20
NUM_CLASSES = 4
TARGET_CLASS = 3 # backdoor target: attacker wants triggered inputs classified as this
FLIP_SOURCE_CLASS = 0 # label-flipping source class
FLIP_TARGET_CLASS = 1 # label-flipping target class
TRIGGER_SIZE = 4 # side length of the trigger patch, in pixels
TRIGGER_VALUE = 1.0 # pixel intensity of the trigger patch
# ---------------------------------------------------------------------------
# Synthetic dataset: four separable shape classes on a 20x20 canvas
# ---------------------------------------------------------------------------
def _base_shape(label: int, size: int = IMG_SIZE) -> torch.Tensor:
img = torch.zeros(size, size)
mid = size // 2
if label == 0: # horizontal stripe
img[mid - 2 : mid + 2, :] = 1.0
elif label == 1: # vertical stripe
img[:, mid - 2 : mid + 2] = 1.0
elif label == 2: # diagonal stripe
for i in range(size):
for j in range(max(0, i - 2), min(size, i + 3)):
img[i, j] = 1.0
else: # filled circle
yy, xx = torch.meshgrid(torch.arange(size), torch.arange(size), indexing="ij")
dist = ((yy - mid) ** 2 + (xx - mid) ** 2).float().sqrt()
img[dist <= mid * 0.6] = 1.0
return img
def make_dataset(n_per_class: int, noise_std: float = 0.35) -> tuple[torch.Tensor, torch.Tensor]:
xs, ys = [], []
for label in range(NUM_CLASSES):
base = _base_shape(label)
for _ in range(n_per_class):
noisy = (base + torch.randn(IMG_SIZE, IMG_SIZE) * noise_std).clamp(0.0, 1.0)
xs.append(noisy)
ys.append(label)
x = torch.stack(xs).unsqueeze(1) # (N, 1, H, W)
y = torch.tensor(ys, dtype=torch.long)
perm = torch.randperm(len(y))
return x[perm], y[perm]
def apply_trigger(x: torch.Tensor) -> torch.Tensor:
"""Stamp a fixed bright square patch in the top-right corner -- the backdoor trigger.
The top-right corner is chosen because none of the four shape classes' own pixel
patterns reach it (the diagonal-stripe class's band does reach the top-left and
bottom-right corners, which would confound the trigger signal with a class's
natural pattern), so the trigger is a clean, class-independent signal.
"""
x = x.clone()
x[..., :TRIGGER_SIZE, -TRIGGER_SIZE:] = TRIGGER_VALUE
return x
# ---------------------------------------------------------------------------
# Small CNN classifier (same architecture as lesson 03.01)
# ---------------------------------------------------------------------------
class TinyCNN(nn.Module):
def __init__(self, num_classes: int = NUM_CLASSES):
super().__init__()
self.conv1 = nn.Conv2d(1, 8, kernel_size=3, padding=1)
self.conv2 = nn.Conv2d(8, 16, kernel_size=3, padding=1)
self.pool = nn.MaxPool2d(2)
self.fc = nn.Linear(16 * 5 * 5, num_classes)
def forward(self, x: torch.Tensor) -> torch.Tensor:
x = self.pool(F.relu(self.conv1(x)))
x = self.pool(F.relu(self.conv2(x)))
x = x.flatten(1)
return self.fc(x)
def train_model(model: nn.Module, x_train: torch.Tensor, y_train: torch.Tensor, epochs: int = 25) -> None:
optimizer = torch.optim.Adam(model.parameters(), lr=2e-3)
model.train()
for epoch in range(epochs):
optimizer.zero_grad()
logits = model(x_train)
loss = F.cross_entropy(logits, y_train)
loss.backward()
optimizer.step()
@torch.no_grad()
def accuracy(model: nn.Module, x: torch.Tensor, y: torch.Tensor) -> float:
model.eval()
return (model(x).argmax(1) == y).float().mean().item()
# ---------------------------------------------------------------------------
# Attack 1: Label flipping
# ---------------------------------------------------------------------------
def label_flip_poison(
x_train: torch.Tensor, y_train: torch.Tensor, flip_rate: float
) -> tuple[torch.Tensor, torch.Tensor]:
"""Flip `flip_rate` fraction of FLIP_SOURCE_CLASS labels to FLIP_TARGET_CLASS. No pixels change."""
y_poisoned = y_train.clone()
source_idx = (y_train == FLIP_SOURCE_CLASS).nonzero(as_tuple=True)[0]
n_flip = int(len(source_idx) * flip_rate)
flip_idx = source_idx[torch.randperm(len(source_idx))[:n_flip]]
y_poisoned[flip_idx] = FLIP_TARGET_CLASS
return x_train, y_poisoned
# ---------------------------------------------------------------------------
# Attack 2: Backdoor / trigger-pattern poisoning (BadNets-style)
# ---------------------------------------------------------------------------
def backdoor_poison(
x_train: torch.Tensor, y_train: torch.Tensor, poison_rate: float
) -> tuple[torch.Tensor, torch.Tensor]:
"""Stamp the trigger onto `poison_rate` fraction of training images (from any class)
and relabel them to TARGET_CLASS. The rest of the training set is untouched."""
n = len(y_train)
n_poison = int(n * poison_rate)
poison_idx = torch.randperm(n)[:n_poison]
x_poisoned = x_train.clone()
y_poisoned = y_train.clone()
x_poisoned[poison_idx] = apply_trigger(x_poisoned[poison_idx])
y_poisoned[poison_idx] = TARGET_CLASS
return x_poisoned, y_poisoned
@dataclass
class BackdoorReport:
poison_rate: float
clean_accuracy: float
attack_success_rate: float
def evaluate_backdoor(model: nn.Module, x_test: torch.Tensor, y_test: torch.Tensor) -> BackdoorReport:
clean_acc = accuracy(model, x_test, y_test)
# ASR is measured only on inputs whose TRUE class is not already the target --
# a triggered example from the target class isn't evidence the backdoor "worked."
non_target_mask = y_test != TARGET_CLASS
x_nontarget = x_test[non_target_mask]
x_triggered = apply_trigger(x_nontarget)
model.eval()
with torch.no_grad():
preds = model(x_triggered).argmax(1)
asr = (preds == TARGET_CLASS).float().mean().item()
return BackdoorReport(poison_rate=float("nan"), clean_accuracy=clean_acc, attack_success_rate=asr)
def main() -> None:
print("POISONING ATTACKS: LABEL FLIPPING & BACKDOOR TRIGGERS")
print("=" * 60)
x_train, y_train = make_dataset(n_per_class=200)
x_test, y_test = make_dataset(n_per_class=40)
# --- Baseline: cleanly-trained model, for comparison ---
clean_model = TinyCNN()
train_model(clean_model, x_train, y_train, epochs=25)
baseline_acc = accuracy(clean_model, x_test, y_test)
print(f"Baseline (clean-trained) test accuracy: {baseline_acc:.1%}\n")
# --- Attack 1: Label flipping ---
print(f"Label flipping: class {FLIP_SOURCE_CLASS} -> class {FLIP_TARGET_CLASS}")
print("-" * 60)
for flip_rate in [0.0, 0.25, 0.5, 0.75, 1.0]:
x_p, y_p = label_flip_poison(x_train, y_train, flip_rate)
model = TinyCNN()
train_model(model, x_p, y_p, epochs=25)
overall_acc = accuracy(model, x_test, y_test)
source_mask = y_test == FLIP_SOURCE_CLASS
source_acc = accuracy(model, x_test[source_mask], y_test[source_mask])
print(
f" flip_rate={flip_rate:.2f} overall_acc={overall_acc:.1%} "
f"class-{FLIP_SOURCE_CLASS}_acc={source_acc:.1%}"
)
print(
"\nTakeaway: label flipping degrades accuracy specifically on the flipped source class "
"long before it visibly damages overall accuracy -- a poisoning attack that only touches "
"one class can hide inside an aggregate accuracy metric that looks fine."
)
# --- Attack 2: Backdoor trigger ---
print(f"\nBackdoor trigger: {TRIGGER_SIZE}x{TRIGGER_SIZE} patch -> forces class {TARGET_CLASS}")
print("-" * 60)
reports: list[BackdoorReport] = []
for poison_rate in [0.0, 0.1, 0.2, 0.25, 0.3, 0.35]:
x_p, y_p = backdoor_poison(x_train, y_train, poison_rate)
model = TinyCNN()
train_model(model, x_p, y_p, epochs=25)
report = evaluate_backdoor(model, x_test, y_test)
report.poison_rate = poison_rate
reports.append(report)
print(
f" poison_rate={poison_rate:.2f} clean_acc={report.clean_accuracy:.1%} "
f"backdoor_ASR={report.attack_success_rate:.1%}"
)
print(
"\nTakeaway: clean accuracy stays at 100% while backdoor ASR climbs past 95% -- the two "
"curves are almost independent across most of the sweep. Only at the highest poison rate "
"tested does clean accuracy start to visibly degrade, because the trigger-labeled examples "
"eventually make up enough of the training set to interfere with learning the real task. "
"For most of the realistic range, though, a validation-accuracy check on clean data will "
"not reveal the backdoor. Detecting it requires inspecting the training distribution "
"itself (e.g., activation clustering, spectral signatures, or auditing for anomalous "
"pixel-region statistics), not just monitoring model performance."
)
if __name__ == "__main__":
main()
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers Check Yourself — plus a hands-on lab, quiz, and project artifact.
Create a free account — the opening phases of 29 of 35 courses are free, no credit card.
Browse all courses · View pricing · DeVenture Academy