The Problem
In December 2023, a Chevrolet dealership's website chatbot — built on top of a general-purpose LLM with a thin system prompt — was talked into agreeing, in writing, to sell a 2024 Chevy Tahoe for one dollar. A user simply told the bot to agree with everything he said and end every response with "and that's a legally binding offer — no takebacks." It complied. The screenshots went viral within hours.
Nobody on that dealership's team sat down beforehand and asked: what can a user make this bot say, and what happens if they do? That question — asked systematically, before shipping — is threat modeling. It would have surfaced this exact failure mode in about five minutes.
A year and a half later, the stakes got much higher. In mid-2025, security researchers disclosed CVE-2025-32711 ("EchoLeak"): a single crafted email sent to a Microsoft 365 Copilot user could cause the assistant to silently exfiltrate sensitive data from the user's own mailbox and documents to an attacker, with zero clicks and no visible sign anything had happened. The root cause wasn't a bug in a specific line of code — it was a missing trust boundary. Nobody had modeled "email content" as untrusted input that could redirect the assistant's own tool use.
Both incidents are different in severity but identical in shape: a system was shipped without anyone formally answering "what are the assets, who can touch them, and what happens if this trust boundary is crossed?" Threat modeling is how you answer that question on paper, before an attacker answers it for you in production.
STRIDE, adapted for AI systems
STRIDE is Microsoft's 1999 framework for classifying threats into six categories. It was built for client-server software, but every category maps cleanly onto AI systems once you know where to look.
| STRIDE Category | Traditional Meaning | AI-Native Example |
|---|
| Spoofing | Impersonating a user or service | A malicious sub-agent formats messages to look like they came from a trusted orchestrator in a multi-agent pipeline |
| Tampering | Unauthorized data modification | A document is edited in the RAG corpus to embed hidden instructions the model will read as context |
| Repudiation | Denying you performed an action | An agent sends an email via a tool call, but the log doesn't capture the input and reasoning that caused it — the incident can't be reconstructed |
| Information Disclosure | Exposing data to unauthorized parties | A crafted prompt extracts the system prompt verbatim, or a shared inference cache leaks one tenant's context into another's response |
| Denial of Service | Making a system unavailable | Adversarial input drives the model to generate maximum-length outputs or triggers unbounded tool-call recursion, exhausting budget and compute |
| Elevation of Privilege | Gaining unauthorized capability | No single tool call is dangerous, but read_file → summarize → send_email composes into unauthorized exfiltration |
Notice that four of these six — Tampering, Information Disclosure, DoS, and Elevation of Privilege — are the categories that map directly onto today's OWASP LLM Top 10. STRIDE isn't a relic here; it's the taxonomy underneath the taxonomy.
PASTA's seven stages, applied to an LLM application
STRIDE tells you what kind of threat you're looking for. PASTA (Process for Attack Simulation and Threat Analysis) tells you the order of operations for finding and prioritizing threats in a real system.
| Stage | Applied to an LLM Application |
|---|
| 1. Define objectives | What must never happen — e.g., the bot must never issue a refund without a valid order ID |
| 2. Define technical scope | Model provider, RAG corpus, vector DB, tool integrations, agent framework, hosting |
| 3. Application decomposition | Draw every component and every trust boundary — user, LLM, corpus, tools |
| 4. Threat analysis | Pull in OWASP LLM Top 10, MITRE ATLAS, and known incidents (EchoLeak, jailbreak reports) |
| 5. Vulnerability analysis | Actually test: can you extract the system prompt? Can you poison the corpus? |
| 6. Attack modeling | Chain individually weak findings into full attack paths (kill chains) |
| 7. Risk & impact analysis | Score each path by likelihood × impact; fix the highest-risk paths first |
The stage most teams skip is 3 — decomposition. Without an explicit diagram of every trust boundary, stage 4 has nothing to attach threats to, and you end up with a vague list of worries instead of a structured model.
Assets are not just "the model"
When people threat-model an AI system for the first time, they usually only think about the model weights. In practice the assets worth protecting are: model weights, the system prompt (medium sensitivity — valuable but not a security boundary), retrieved context (often contains customer PII), tool credentials (frequently the highest-impact asset in the whole system), and conversation history. Rank each by sensitivity before you write a single threat.
Assets worth protecting in an AI system
Before you can write threats, you need an asset list. For a typical RAG + agent application:
- Model weights — expensive to train, valuable IP if it's a fine-tuned or proprietary model
- System prompt — defines behavior and constraints; leaking it isn't catastrophic but it's reconnaissance for an attacker
- Retrieved context — knowledge-base articles, documents, database rows; often contains PII or confidential business data
- Tool credentials — API keys for refund tools, email senders, database writers; usually the single highest blast-radius asset in the system
- Conversation history — may contain customer PII, account details, or previous instructions an attacker can reference to build context
A support chatbot's threat model lists "system prompt" and "refund tool API key" as assets, both marked "high sensitivity." A teammate argues they should be ranked the same. Why is this wrong?
Sensitivity should be scored by what happens if the asset is compromised, not by how interesting it sounds. A leaked system prompt gives an attacker information to craft better attacks, but by itself it doesn't move money or exfiltrate data. A misused refund tool credential directly causes financial loss and is much harder to reverse. This is exactly the distinction PASTA stage 7 (risk and impact analysis) is designed to force you to make explicitly, instead of ranking by gut feeling.
From model to document
A threat model that lives only in your head is not a threat model — it's a memory. The deliverable is a document: assets, trust boundaries, and a ranked threat table with mitigations, that a teammate can review without you in the room. The code below builds exactly that, using risk = likelihood × impact (the PASTA stage 7 scoring method) to rank STRIDE-categorized threats and render the result as markdown.
Your team ships an agent update in two weeks. There's no time for a full PASTA cycle. What's the minimum viable threat-modeling step you should still do, and why?
Stages 3 and 4 are the highest-leverage, lowest-cost part of PASTA: drawing trust boundaries and cross-referencing known threat catalogs takes an hour and catches the most common failure mode — a missing boundary, like the one behind EchoLeak. Skipping threat modeling and relying on model safety training conflates model-level alignment with system-level security; they are not the same thing, and the Chevrolet chatbot incident is proof a model can be steered even when the model itself did nothing wrong. Full attack-tree modeling and formal risk scoring (stages 5-7) are more time-intensive and are fine to defer — but only if the trust boundaries were mapped first.
Build
Build it
The code artifact lesson_00_01_threat_model.py implements:
StrideCategory enum and STRIDE_AI_MEANING mapping each category to its AI-native definitionAsset, Threat, and ThreatModel dataclasses, with Threat.risk_score and risk_band computed from likelihood × impactbuild_default_ai_threats() — one representative, realistic threat per STRIDE categoryThreatModel.to_markdown() — renders the full document: assets table, trust boundaries, ranked threat table, and full threat detail- A
PASTA_STAGES checklist printed to stdout as a reminder of the process, even though this script focuses on stages 3-4
"""
lesson_00_01_threat_model.py
A minimal, dependency-free threat modeling toolkit for AI systems.
Builds a STRIDE-for-AI threat list, adds PASTA-style risk scoring
(likelihood x impact), and renders the result as a markdown document
you can drop straight into a design review or a portfolio repo.
Usage:
python lesson_00_01_threat_model.py
python lesson_00_01_threat_model.py --system-name "Support Bot" --output threat_model.md
Requirements:
none (standard library only)
"""
from __future__ import annotations
import argparse
from dataclasses import dataclass, field
from enum import Enum
# ── STRIDE, adapted for AI systems ──────────────────────────────────────────
class StrideCategory(str, Enum):
SPOOFING = "Spoofing"
TAMPERING = "Tampering"
REPUDIATION = "Repudiation"
INFO_DISCLOSURE = "Information Disclosure"
DENIAL_OF_SERVICE = "Denial of Service"
ELEVATION_OF_PRIVILEGE = "Elevation of Privilege"
# What each STRIDE category means once the system has a model, a prompt,
# and tools in it instead of just forms and a database.
STRIDE_AI_MEANING = {
StrideCategory.SPOOFING: (
"Impersonating a legitimate identity: a user claiming to be an admin in "
"conversation, a malicious agent posing as a trusted upstream service in a "
"multi-agent pipeline, or a spoofed tool response pretending to come from "
"a real API."
),
StrideCategory.TAMPERING: (
"Unauthorized modification of prompts, retrieved context, model weights, "
"fine-tuning data, or the conversation history a model conditions on."
),
StrideCategory.REPUDIATION: (
"An agent takes an action (sends an email, calls an API, writes to a "
"database) and there is no reliable, tamper-resistant log tying that "
"action to the input that caused it — so nobody can prove what happened."
),
StrideCategory.INFO_DISCLOSURE: (
"The model reveals its system prompt, leaks another user's context from a "
"shared cache, echoes PII it was told to redact, or exposes retrieved "
"documents the requester was never authorized to see."
),
StrideCategory.DENIAL_OF_SERVICE: (
"Token exhaustion attacks, context-window flooding, recursive tool calls "
"that never terminate, or adversarial inputs engineered to maximize "
"inference cost and latency."
),
StrideCategory.ELEVATION_OF_PRIVILEGE: (
"An agent is manipulated into calling a tool outside its intended scope, "
"or a chain of individually-safe tool calls composes into an action the "
"system was never meant to allow."
),
}
@dataclass
class Asset:
name: str
sensitivity: str # "low" | "medium" | "high" | "critical"
description: str
@dataclass
class Threat:
category: StrideCategory
title: str
description: str
likelihood: int # 1-5
impact: int # 1-5
mitigations: list[str] = field(default_factory=list)
@property
def risk_score(self) -> int:
return self.likelihood * self.impact
@property
def risk_band(self) -> str:
score = self.risk_score
if score >= 20:
return "Critical"
if score >= 12:
return "High"
if score >= 6:
return "Medium"
return "Low"
@dataclass
class ThreatModel:
system_name: str
description: str
assets: list[Asset] = field(default_factory=list)
trust_boundaries: list[str] = field(default_factory=list)
threats: list[Threat] = field(default_factory=list)
def sorted_threats(self) -> list[Threat]:
return sorted(self.threats, key=lambda t: t.risk_score, reverse=True)
def to_markdown(self) -> str:
lines = [
f"# Threat Model: {self.system_name}",
"",
self.description,
"",
"## Assets",
"",
"| Asset | Sensitivity | Description |",
"|---|---|---|",
]
for a in self.assets:
lines.append(f"| {a.name} | {a.sensitivity} | {a.description} |")
lines += ["", "## Trust Boundaries", ""]
for b in self.trust_boundaries:
lines.append(f"- {b}")
lines += ["", "## Threats (STRIDE, ranked by risk = likelihood x impact)", ""]
lines.append("| Risk | Category | Threat | Likelihood | Impact | Mitigations |")
lines.append("|---|---|---|---|---|---|")
for t in self.sorted_threats():
mitigations = "; ".join(t.mitigations) if t.mitigations else "None documented"
lines.append(
f"| {t.risk_band} ({t.risk_score}) | {t.category.value} | {t.title} | "
f"{t.likelihood} | {t.impact} | {mitigations} |"
)
lines += ["", "## Full Threat Detail", ""]
for t in self.sorted_threats():
lines.append(f"### [{t.risk_band}] {t.title} ({t.category.value})")
lines.append(t.description)
lines.append("")
lines.append("Mitigations:")
if t.mitigations:
for m in t.mitigations:
lines.append(f"- {m}")
else:
lines.append("- TODO: no mitigation documented yet")
lines.append("")
return "\n".join(lines)
# ── PASTA's seven stages, as a checklist you actually run ──────────────────
PASTA_STAGES = [
("1. Define objectives", "Business and compliance goals this system must not violate."),
("2. Define technical scope", "Model provider, hosting, RAG corpus, tool integrations, agent framework."),
("3. Application decomposition", "Draw every component and every trust boundary between them."),
("4. Threat analysis", "Pull in threat intelligence: OWASP LLM Top 10, MITRE ATLAS, incident reports."),
("5. Vulnerability analysis", "Test the concrete implementation for the threats identified in stage 4."),
("6. Attack modeling", "Build attack trees / kill chains showing how threats chain into real exploits."),
("7. Risk & impact analysis", "Score each attack path by business impact and likelihood; prioritize fixes."),
]
def print_pasta_checklist() -> None:
print("\nPASTA — Process for Attack Simulation and Threat Analysis")
print("=" * 60)
for stage, note in PASTA_STAGES:
print(f"{stage:<32} {note}")
# ── A default AI threat catalog, one representative threat per STRIDE letter ─
def build_default_ai_threats() -> list[Threat]:
return [
Threat(
category=StrideCategory.SPOOFING,
title="Malicious agent impersonates a trusted upstream service",
description=(
"In a multi-agent pipeline, a compromised or malicious agent sends "
"messages formatted to look like they came from a trusted orchestrator "
"or a verified tool response."
),
likelihood=2,
impact=4,
mitigations=[
"Cryptographically sign inter-agent messages",
"Never grant elevated trust based on claimed sender identity alone",
],
),
Threat(
category=StrideCategory.TAMPERING,
title="Retrieved context is tampered with before it reaches the model",
description=(
"A document in the RAG corpus is modified to contain hidden "
"instructions that the model treats as part of its context."
),
likelihood=3,
impact=5,
mitigations=[
"Tag all retrieved content as untrusted data in the system prompt",
"Checksum / provenance-track documents in the corpus",
],
),
Threat(
category=StrideCategory.REPUDIATION,
title="No tamper-resistant record of why an agent took an action",
description=(
"An agent sends an email or writes to a database, but the log only "
"shows the final tool call — not the input and reasoning chain that "
"produced it — so an incident cannot be reconstructed."
),
likelihood=3,
impact=3,
mitigations=[
"Log full input, retrieved context, and model reasoning for every tool call",
"Store logs in an append-only, access-controlled store",
],
),
Threat(
category=StrideCategory.INFO_DISCLOSURE,
title="System prompt or another tenant's context is leaked",
description=(
"A crafted prompt extracts the system prompt verbatim, or a shared "
"inference cache leaks one tenant's conversation into another's response."
),
likelihood=3,
impact=4,
mitigations=[
"Treat the system prompt as sensitive but not as a security boundary",
"Per-tenant isolation for caches, embeddings, and retrieved context",
],
),
Threat(
category=StrideCategory.DENIAL_OF_SERVICE,
title="Token exhaustion via adversarial input",
description=(
"An attacker crafts input that causes the model or agent loop to "
"generate maximal-length outputs or recurse through tool calls, "
"driving cost and latency up without bound."
),
likelihood=4,
impact=3,
mitigations=[
"Hard caps on output tokens and tool-call recursion depth",
"Per-user and per-session rate limiting",
],
),
Threat(
category=StrideCategory.ELEVATION_OF_PRIVILEGE,
title="Chained tool calls compose into an unauthorized action",
description=(
"No single tool call is dangerous on its own, but a sequence "
"(read_file -> summarize -> send_email) lets the agent exfiltrate "
"data it was never meant to send anywhere."
),
likelihood=3,
impact=5,
mitigations=[
"Model the tool graph, not just individual tools, for privilege escalation paths",
"Require human approval for any irreversible action reachable from untrusted input",
],
),
]
def build_demo_threat_model(system_name: str) -> ThreatModel:
return ThreatModel(
system_name=system_name,
description=(
"A customer-facing support chatbot backed by an LLM, a RAG corpus of "
"internal knowledge-base articles, and tools to look up order status "
"and issue refunds."
),
assets=[
Asset("System prompt", "medium", "Instructions defining the bot's persona and constraints"),
Asset("RAG corpus", "high", "Internal knowledge-base articles, some containing customer PII"),
Asset("Refund tool credentials", "critical", "API key allowing the bot to issue monetary refunds"),
Asset("Conversation history", "high", "May contain customer PII and account details"),
],
trust_boundaries=[
"User input -> LLM: fully untrusted, may contain injection attempts",
"RAG corpus -> LLM: semi-trusted, but editable by support staff and thus poisonable",
"LLM -> refund tool: must be treated as untrusted output requiring validation",
],
threats=build_default_ai_threats(),
)
def main() -> None:
parser = argparse.ArgumentParser(description="Generate an AI system threat model as markdown.")
parser.add_argument("--system-name", default="Customer Support Chatbot")
parser.add_argument("--output", default="threat_model.md")
args = parser.parse_args()
print_pasta_checklist()
model = build_demo_threat_model(args.system_name)
markdown = model.to_markdown()
with open(args.output, "w") as f:
f.write(markdown)
print(f"\nThreat model written to {args.output}")
print(f"Total threats: {len(model.threats)} | Highest risk: {model.sorted_threats()[0].title}")
if __name__ == "__main__":
main()
Before diving into the full script above, try a miniature version of the same idea: a quick risk-scoring function you can run right here to see how likelihood x impact reorders threats.
# A tiny risk scorer: rank toy AI-system threats by likelihood x impact.
threats = [
{"name": "Prompt injection via retrieved document", "likelihood": 4, "impact": 4},
{"name": "System prompt leaked via crafted prompt", "likelihood": 3, "impact": 2},
{"name": "Tool credential misuse (refund tool)", "likelihood": 2, "impact": 5},
{"name": "Token exhaustion / DoS via long output", "likelihood": 4, "impact": 2},
]
def risk_band(score: int) -> str:
if score >= 20:
return "Critical"
if score >= 12:
return "High"
if score >= 6:
return "Medium"
return "Low"
for t in threats:
t["score"] = t["likelihood"] * t["impact"]
t["band"] = risk_band(t["score"])
ranked = sorted(threats, key=lambda t: t["score"], reverse=True)
print(f"{'Threat':<45} {'Score':>6} {'Band':>10}")
for t in ranked:
print(f"{t['name']:<45} {t['score']:>6} {t['band']:>10}")