Phase 3: Adversarial Machine Learning · 55 min · PyTorch

Model Extraction & Stealing

If your model is only reachable through an API, your API is a training set generator you didn't sign up to run.

Hiring signal: Model theft is LLM10 in the OWASP Top 10 for LLM Applications and a named tactic in MITRE ATLAS, and companies serving proprietary models as a paid API (classification, embeddings, ranking) treat extraction as a direct IP and revenue threat. Being able to run a distillation-based extraction attack, quantify fidelity against query budget, and implement query-pattern monitoring and watermarking as countermeasures is exactly what an AI security or ML platform team interview probes for.

What you will learn

Explain model extraction as a distillation attack that reconstructs a victim's decision function from query/response pairs alone, with no access to weights, gradients, or training data
Implement an extraction attack end-to-end: synthetic query generation, soft-label collection through an API-like interface, and student training via KL-divergence distillation
Measure extraction fidelity (student/victim prediction agreement) as a function of query budget, and explain why fidelity rises even when the student never sees a ground-truth label
Implement and evaluate two practical countermeasures: query-pattern (volume + out-of-distribution) monitoring and watermark-based ownership verification

The Problem

In 2016, Florian Tramèr and coauthors asked a question that should worry anyone charging money for model predictions: if an attacker can only send inputs and receive outputs — no weights, no gradients, no training data, just a black-box query interface exactly like a commercial ML API — how much of the model can they reconstruct? Their paper, Stealing Machine Learning Models via Prediction APIs (Tramèr et al., 2016, arXiv:1609.02943), showed the answer was "most of it, and cheaply": for several model classes served by real commercial ML-as-a-service platforms at the time, a few thousand queries were enough to train a substitute model that matched the victim's predictions on the vast majority of inputs.

This matters for a specific business reason that's easy to under-rate: a trained model is often the actual product. Companies spend months and significant compute curating training data, tuning architectures, and validating a model before shipping it behind a prediction endpoint — and the entire value of that investment is exposed, one HTTP response at a time, to anyone willing to pay for enough API calls. Model extraction isn't a theoretical vulnerability restricted to research demos: it's listed as its own category (LLM10: Unbounded Consumption / model theft) in the OWASP Top 10 for LLM Applications, and MITRE ATLAS catalogs it as a named adversary tactic against production ML systems, alongside real incidents where competitors stood up cheaper substitute services by distilling a paid API's outputs.

The attack works even though the attacker has strictly less information than the victim's own training process ever had: no ground-truth labels, no real training distribution, and no gradient access to the victim model. What it does have is the one thing every prediction API is contractually obligated to hand over — the model's output on any input the attacker chooses to send.

How Extraction Works: Query, Collect, Distill

The attack has three steps, and none of them require anything beyond normal API access:

Query. The attacker sends inputs to the victim's prediction endpoint. Because the attacker doesn't have the victim's real training data, it can't sample realistic inputs — instead it probes broadly across the input space it believes the model was trained on, trying to cover as much of the decision surface as the query budget allows.
Collect. The victim API responds with soft-label probability vectors (or, in a more restricted API, just the top-1 label or a confidence score). Soft labels leak far more information per query than hard labels do, because they encode the model's confidence and its relative ranking of every class, not just the winner.
Distill. The attacker trains a new "student" model — often much smaller and cheaper than the victim — to reproduce the collected (input, victim-output) pairs. Training minimizes the KL divergence between the student's predicted distribution and the victim's, which is the exact same objective used in legitimate knowledge distillation (Hinton et al.). The only difference between distillation-for-compression and distillation-for-theft is who owns the model being copied and whether they consented.

The quantity that matters for judging how well an extraction attack worked is fidelity: how often does the student's prediction agree with the victim's prediction on inputs neither model was trained on? Fidelity is deliberately not "student accuracy against ground truth" — an attacker stealing a model doesn't care whether the victim is right about the real world, only whether the student behaves like the victim. A student with high fidelity but mediocre true accuracy is still a successful theft if the victim itself has mediocre true accuracy.

Extraction doesn't need real data, or even the right labels
The attacker in this lesson probes with inputs sampled uniformly across a guessed bounding box — not real data from the victim's training distribution, which it never sees. It also never has a single ground-truth label. Despite both handicaps, the student reaches near-perfect fidelity with the victim at every query budget the code tests, because every query response is a free, exact data point about the victim's function, regardless of whether that function is queried near real or synthetic inputs — the student doesn't need a large budget to curve-fit a function that is easy to reproduce. This is the core insight that makes model extraction so much cheaper than model theft-by-reverse-engineering: the attacker isn't trying to steal the victim's data or reasoning, only to copy its input-output mapping, and copying is cheap even when the mapping being copied is simple.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Distillation Attacks and IP Theft, Detecting Extraction: Query-Pattern Monitoring, Detecting Extraction After the Fact: Watermarking, Build It, What to Practice — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.