Phase 1: OWASP Top 10 for LLM Applications · 65 min · Python · Anthropic SDK

LLM10: Model Theft — and Mapping a Full System

A model is stolen one boring, individually-legal-looking query at a time — and a security review that doesn't check all 10 categories has already decided which nine it's willing to miss.

Hiring signal: Model extraction cost OpenAI and Google real engineering and legal attention once researchers demonstrated that a target model's behavior could be cloned via API queries alone (Tramer et al., 2016) — teams shipping a proprietary model behind an API are expected to know that query volume and pattern, not just credentials, is an attack surface. Just as valuable to employers: the ability to take a real system's architecture and produce a full OWASP LLM Top 10 gap analysis is the exact deliverable a security engineer or AI red teamer is asked to produce in a pre-launch review, and being able to do it fluently across all 10 categories — not just the two or three that are top of mind — is what distinguishes a security hire from someone who has only read the list once.

What you will learn

Explain how model extraction attacks reconstruct a target model's decision boundary or distill its behavior using only API query access, without any weight access
Design a query-pattern rate-anomaly detector that flags the systematic, high-volume querying signature of an extraction attempt as distinct from normal usage
Rank model theft defenses (rate limiting, query monitoring, watermarking, output perturbation, API access controls) by where each one intervenes in the attack chain
Produce a full OWASP LLM Top 10 gap-analysis report mapping a real AI system's architecture against all 10 categories, distinguishing present controls from gaps

The Problem

In 2016, Florian Tramer and coauthors published "Stealing Machine Learning Models via Prediction APIs" (https://arxiv.org/abs/1609.02943), demonstrating something uncomfortable for anyone shipping a proprietary model behind an API: an attacker with nothing but black-box query access — no credentials to the training pipeline, no access to weights, no insider — could reconstruct an equivalent model, sometimes to near-perfect fidelity, using only the responses the API was designed to give out. Every individual query looked legitimate. It was the pattern, at scale, that constituted the attack.

That result generalizes directly to hosted LLMs. A proprietary model behind an API is, from the outside, indistinguishable from a very well-labeled dataset generator. An attacker who queries it systematically enough can either map its decision boundary directly or use its outputs as training labels to distill a smaller clone that mimics its behavior — a technique with a legitimate research name (knowledge distillation) that becomes IP theft the moment it's done against a competitor's model without authorization. The company that spent months and a large compute budget training and fine-tuning a model has no reliable way to tell, from any single request, whether it's serving a customer or being cloned.

This is LLM10 in the OWASP GenAI/LLM Top 10 (https://genai.owasp.org/llm-top-10/): Model Theft. Unlike prompt injection or insecure output handling, which are exploited through what a model is tricked into doing, model theft is exploited through what a model is asked, over and over, at volume or with structure that reveals the intent behind the questions.

This lesson closes out the phase in two parts. Part one covers model theft detection and defense. Part two is the capstone: taking a real system's architecture and producing a full gap-analysis report against all 10 OWASP LLM Top 10 categories covered across this phase — the exact deliverable a security engineer or AI red teamer is asked to produce in a pre-launch review.

Model Theft: Extraction, Distillation, and IP Theft

Model theft covers three related but distinct threats:

1. Extraction via API queries. The attacker doesn't need the model's weights — only its prediction API. By sending a large, systematic set of queries and recording the outputs, they can reconstruct the model's decision surface closely enough to build a functional substitute. This is cheapest against models with simple output spaces (classification, scoring) but works in weakened forms against generative models too: enough (prompt, completion) pairs teach a lot about a model's style, refusal boundaries, and knowledge.

2. Distillation attacks. A variant of extraction where the goal isn't to reverse-engineer decision boundaries but to train a new, usually smaller, model using the target model's outputs as labels — legitimate when done with permission (this is how many efficient open models are trained), and IP theft when done against a competitor's proprietary API without authorization. The attack surface is identical to extraction: systematic querying, output logging, training a new model on the results.

3. IP theft. The business consequence of either of the above: a competitor ships a model that performs comparably to yours at a fraction of your training cost, because they trained on your API's outputs instead of your training data. This has been a live commercial concern in the LLM industry — providers have publicly scrutinized traffic patterns from competitors' research accounts for exactly this signature.

None of these attacks require breaching your infrastructure. They require only that your API works as designed, at a volume or in a pattern that isn't ordinary usage. That's why detection for LLM10 lives at the query-pattern layer, not the access-control layer.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Detection: Query-Pattern Monitoring and Watermarking, Capstone: Mapping a Full System Against All 10 OWASP LLM Categories, Build It, What to Practice — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.