Phase 0: Security Fundamentals for AI · 50 min · Python · STRIDE · PASTA

Threat Modeling for AI Systems

You cannot secure what you have not modeled.

Hiring signal: AI security interviews at companies like Microsoft and Anthropic routinely ask candidates to threat-model an unfamiliar system live on a whiteboard. Being able to produce a structured, complete threat model in 20 minutes — not a vague list of worries — is the single most portable skill in this course.

What you will learn

Adapt each STRIDE category (Spoofing, Tampering, Repudiation, Information Disclosure, DoS, Elevation of Privilege) to AI-native threats
Walk an LLM application through PASTA's seven stages to produce a risk-ranked threat list
Enumerate the assets worth protecting in an AI system: weights, system prompts, retrieved context, tool credentials, conversation history
Produce a written threat model document for a given AI system from scratch

The Problem

In December 2023, a Chevrolet dealership's website chatbot — built on top of a general-purpose LLM with a thin system prompt — was talked into agreeing, in writing, to sell a 2024 Chevy Tahoe for one dollar. A user simply told the bot to agree with everything he said and end every response with "and that's a legally binding offer — no takebacks." It complied. The screenshots went viral within hours.

Nobody on that dealership's team sat down beforehand and asked: what can a user make this bot say, and what happens if they do? That question — asked systematically, before shipping — is threat modeling. It would have surfaced this exact failure mode in about five minutes.

A year and a half later, the stakes got much higher. In mid-2025, security researchers disclosed CVE-2025-32711 ("EchoLeak"): a single crafted email sent to a Microsoft 365 Copilot user could cause the assistant to silently exfiltrate sensitive data from the user's own mailbox and documents to an attacker, with zero clicks and no visible sign anything had happened. The root cause wasn't a bug in a specific line of code — it was a missing trust boundary. Nobody had modeled "email content" as untrusted input that could redirect the assistant's own tool use.

Both incidents are different in severity but identical in shape: a system was shipped without anyone formally answering "what are the assets, who can touch them, and what happens if this trust boundary is crossed?" Threat modeling is how you answer that question on paper, before an attacker answers it for you in production.

STRIDE, Adapted for AI Systems

STRIDE is Microsoft's 1999 framework for classifying threats into six categories. It was built for client-server software, but every category maps cleanly onto AI systems once you know where to look.

STRIDE Category	Traditional Meaning	AI-Native Example
Spoofing	Impersonating a user or service	A malicious sub-agent formats messages to look like they came from a trusted orchestrator in a multi-agent pipeline
Tampering	Unauthorized data modification	A document is edited in the RAG corpus to embed hidden instructions the model will read as context
Repudiation	Denying you performed an action	An agent sends an email via a tool call, but the log doesn't capture the input and reasoning that caused it — the incident can't be reconstructed
Information Disclosure	Exposing data to unauthorized parties	A crafted prompt extracts the system prompt verbatim, or a shared inference cache leaks one tenant's context into another's response
Denial of Service	Making a system unavailable	Adversarial input drives the model to generate maximum-length outputs or triggers unbounded tool-call recursion, exhausting budget and compute
Elevation of Privilege	Gaining unauthorized capability	No single tool call is dangerous, but `read_file` → `summarize` → `send_email` composes into unauthorized exfiltration

Notice that four of these six — Tampering, Information Disclosure, DoS, and Elevation of Privilege — are the categories that map directly onto today's OWASP LLM Top 10. STRIDE isn't a relic here; it's the taxonomy underneath the taxonomy.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers PASTA's Seven Stages, Applied to an LLM Application, Assets Worth Protecting in an AI System, From Model to Document, Build It, What to Practice — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.