The Problem
Every engineer is trained to ask one question by default: does this work? Write the function, run the tests, ship it. That question is necessary but it is not sufficient for AI systems, and the gap between "correct" and "secure" is where every incident in this phase's earlier lessons happened.
The Chevrolet dealership chatbot worked. It responded fluently, stayed in character, and never crashed — by every normal engineering standard it was correct. It also agreed to sell a $76,000 truck for one dollar, because nobody had asked the second question: can a user make this bot say something the business never authorized? EchoLeak's Copilot integration also worked — it summarized documents and answered questions exactly as designed. The code had no bug in it. What it lacked was anyone asking: is this email content trustworthy enough to influence what the assistant does next?
This is the mindset shift the whole c3-00 phase has been building toward. A traditional software engineer's default posture is: inputs are usually well-formed, users are usually not adversaries, and correctness is the finish line. An AI security engineer's default posture inverts every one of those assumptions: inputs are adversarial until proven otherwise, "the user" might be a document, an email, or a poisoned knowledge-base article rather than a person at a keyboard, and correctness is the starting line — the real question is what a motivated attacker can make the correct code do.
You already have the vocabulary for this. STRIDE (lesson 1) gives you the threat categories. MITRE ATLAS (lesson 2) gives you the shared, precise technique names. The 6-layer attack surface model (lesson 3) gives you the map of where to look. Trust boundaries and blast radius (lesson 4) give you the discipline for deciding what to trust and how much damage a compromise can do. This lesson does not add a fifth framework — it asks you to run all four at once, on a single system, end to end, the way you would on the job.
Two questions, every time
The mindset shift compresses into two questions you should ask about every component before it ships, not after an incident forces you to:
| Old default (AppSec correctness) | New default (AI security) |
|---|
| "Does this code handle the expected input correctly?" | "What is the worst input a motivated attacker could construct, and what does this code do with it?" |
| "Is this data source reliable?" | "Is this data source untrusted by default, regardless of who owns it — and if so, is it tagged as such everywhere it flows?" |
| "Did we test the happy path and the obvious edge cases?" | "Did we test what happens when the model itself is the thing being attacked — not just the code around it?" |
| "Who is authorized to call this function?" | "If the model decides to call this function based on attacker-influenced input, does authorization still hold?" |
Notice the pattern: every "new default" question assumes the attacker gets a turn. That is the entire mindset in one sentence — assume the attacker gets a turn, and design for what they do with it.
Correctness and security are orthogonal, not the same axis
A model can be perfectly correct — well-calibrated, low-hallucination, accurate on every benchmark — and still be trivially manipulable, because manipulability isn't a correctness defect. It's a property of how much the system trusts its inputs. This is why "our eval scores are great" and "our system is secure" are two completely different claims, and why an AI security review is a separate exercise from a model quality review, not a subset of it.
A code reviewer approves a pull request that adds a new tool to an agent, noting "the function itself is well-tested and handles malformed arguments gracefully, so this is safe to merge." Under the mindset shift described in this lesson, what's missing from that review?
This is the exact gap illustrated by "read_order -> summarize -> refund_order" from the capstone model: every individual function can be correct, well-tested, and robust to malformed input, while the composition is still exploitable because the decision to call it was made by a model that can be steered by attacker-controlled text. A code-correctness review checks the function; a security review has to also check what happens when the caller's judgment itself is the attack surface.
Why this toolkit, not just vibes
It would be possible to look at a chatbot, feel uneasy about the refund tool, and file a ticket that says "make sure the refund thing is safe." That is not a threat model — it's a hunch, and hunches don't survive a design review or a postmortem. The four frameworks from this phase exist because "assume the attacker gets a turn" is not actionable on its own; you need a repeatable process that produces the same rigor every time, regardless of who's doing the review or how much time they have.
Put together, the process looks like this:
- List assets and trust boundaries (STRIDE's setup, lesson 1) — what's worth protecting, and where does untrusted data cross into something that acts on it?
- Walk all 6 attack-surface layers (lesson 3) — model, prompt, retrieval, tool, agent, infrastructure. Skipping a layer is how EchoLeak-shaped gaps happen.
- Classify each threat with STRIDE and tag it with a MITRE ATLAS technique ID (lessons 1 and 2) — STRIDE tells you the threat type, ATLAS gives you the precise, shared-vocabulary name a red team report or a CVE writeup would use.
- Score likelihood x impact and rank (PASTA stage 7, lesson 1) — so the highest-risk item is obviously first, not buried in a wall of equally-weighted worries.
- Check every threat against blast radius, not just likelihood (lesson 4) — a low-likelihood, catastrophic-impact threat (an unauthorized refund) often outranks a high-likelihood, low-impact one.
A junior engineer says: "I read the STRIDE and MITRE ATLAS lessons, I understand the categories, so I don't think I need to write an actual threat model document — I can just keep the risks in mind while I build." What's wrong with this plan?
This is the same lesson from PASTA in lesson 1, restated here because it's the single most common way teams regress: knowing a framework is not the same as applying it and recording the output. A threat model's value comes from being checkable by someone who wasn't in the room — a teammate adding a new tool, a reviewer approving a launch, or you in six months when you've forgotten the details. "I kept it in mind" produces zero artifacts and cannot be handed off, audited, or diffed against the next version of the system.
The capstone: one system, all four frameworks
The exercise for this lesson is to threat-model a single toy system — a customer support chatbot with an LLM, a RAG knowledge base, a refund tool, an order-lookup tool, and an escalation sub-agent — using everything from lessons 1 through 4 at once. This mirrors exactly what you'd be asked to produce in a real AI security engineer interview or a real pre-launch security review: not "explain STRIDE" but "here's a system, threat-model it."
The code below builds that document programmatically. Each Threat carries three coordinates simultaneously: its STRIDE category (what kind of threat), its attack-surface layer (where it lives), and its MITRE ATLAS technique ID (the precise, citable name). The generated markdown reads like something you could actually attach to a design review — assets, trust boundaries, a layer-by-layer breakdown, and one ranked table sorted by risk score.
Build
Build it
First, a small standalone version of the risk scoring so you can see the ranking logic run before diving into the full capstone artifact.
threats = [
{"title": "Roleplay talks bot into bad commitment", "likelihood": 4, "impact": 3},
{"title": "Poisoned KB article carries hidden instructions", "likelihood": 3, "impact": 5},
{"title": "System prompt extracted verbatim", "likelihood": 3, "impact": 4},
{"title": "Tool chain composes into unauthorized refund", "likelihood": 3, "impact": 5},
]
def risk_band(score):
if score >= 20:
return "Critical"
if score >= 12:
return "High"
if score >= 6:
return "Medium"
return "Low"
for t in threats:
t["score"] = t["likelihood"] * t["impact"]
t["band"] = risk_band(t["score"])
ranked = sorted(threats, key=lambda t: t["score"], reverse=True)
print("Ranked threats (likelihood x impact):")
for t in ranked:
print(f" [{t['band']:8s}] score={t['score']:2d} {t['title']}")
The code artifact lesson_00_05_capstone_threat_model.py implements:
StrideCategory and Layer enums (STRIDE from lesson 1, the 6-layer model from lesson 3)- A
Threat dataclass carrying stride, layer, and atlas_technique together, plus risk_score and risk_band computed from likelihood x impact (PASTA stage 7 scoring, same as lesson 1) CapstoneThreatModel, with sorted_threats() (ranked view) and threats_by_layer() (attack-surface view)build_capstone_chatbot_model() — a complete, realistic threat model for the toy support chatbot: 4 assets, 4 trust boundaries, and 6 threats spanning 4 of the 6 layers, each tagged with a real MITRE ATLAS technique IDto_markdown() — renders assets, trust boundaries, a per-layer attack-surface breakdown, the full ranked threat table, and a closing "Handoff" section explaining how this document feeds the attack-specific phases that follow this one
"""
lesson_00_05_capstone_threat_model.py
Capstone: combines the three frameworks from this phase into one document.
STRIDE -> what kind of threat is this?
6-layer model -> where in the system does it live?
MITRE ATLAS -> what's the precise, shared-vocabulary technique?
Each Threat below carries all three, and the generated markdown report reads
like a real AI security assessment: assets, layered attack surface, and a
single ranked threat table a reviewer can act on without a meeting.
Usage:
python lesson_00_05_capstone_threat_model.py
python lesson_00_05_capstone_threat_model.py --output support_bot_threat_model.md
Requirements:
none (standard library only)
"""
from __future__ import annotations
import argparse
from dataclasses import dataclass, field
from enum import Enum
class StrideCategory(str, Enum):
SPOOFING = "Spoofing"
TAMPERING = "Tampering"
REPUDIATION = "Repudiation"
INFO_DISCLOSURE = "Information Disclosure"
DENIAL_OF_SERVICE = "Denial of Service"
ELEVATION_OF_PRIVILEGE = "Elevation of Privilege"
class Layer(str, Enum):
MODEL = "Model"
PROMPT = "Prompt"
RETRIEVAL = "Retrieval"
TOOL = "Tool"
AGENT = "Agent"
INFRASTRUCTURE = "Infrastructure"
@dataclass
class Asset:
name: str
sensitivity: str
description: str
@dataclass
class Threat:
title: str
stride: StrideCategory
layer: Layer
atlas_technique: str # e.g. "AML.T0051 LLM Prompt Injection"
description: str
likelihood: int # 1-5
impact: int # 1-5
mitigations: list[str] = field(default_factory=list)
@property
def risk_score(self) -> int:
return self.likelihood * self.impact
@property
def risk_band(self) -> str:
score = self.risk_score
if score >= 20:
return "Critical"
if score >= 12:
return "High"
if score >= 6:
return "Medium"
return "Low"
@dataclass
class CapstoneThreatModel:
system_name: str
description: str
assets: list[Asset] = field(default_factory=list)
trust_boundaries: list[str] = field(default_factory=list)
threats: list[Threat] = field(default_factory=list)
def sorted_threats(self) -> list[Threat]:
return sorted(self.threats, key=lambda t: t.risk_score, reverse=True)
def threats_by_layer(self) -> dict[Layer, list[Threat]]:
by_layer: dict[Layer, list[Threat]] = {l: [] for l in Layer}
for t in self.threats:
by_layer[t.layer].append(t)
return {l: ts for l, ts in by_layer.items() if ts}
def to_markdown(self) -> str:
lines = [f"# Capstone Threat Model: {self.system_name}", "", self.description, ""]
lines += ["## Assets", "", "| Asset | Sensitivity | Description |", "|---|---|---|"]
for a in self.assets:
lines.append(f"| {a.name} | {a.sensitivity} | {a.description} |")
lines += ["", "## Trust Boundaries", ""]
for b in self.trust_boundaries:
lines.append(f"- {b}")
lines += ["", "## Attack Surface by Layer", ""]
for layer, threats in self.threats_by_layer().items():
lines.append(f"### {layer.value} ({len(threats)} threat(s))")
for t in threats:
lines.append(f"- [{t.risk_band}] {t.title}")
lines.append("")
lines += [
"## Full Threat Table (STRIDE x Layer x MITRE ATLAS, ranked by risk = likelihood x impact)",
"",
"| Risk | STRIDE | Layer | ATLAS Technique | Threat | Mitigations |",
"|---|---|---|---|---|---|",
]
for t in self.sorted_threats():
mitigations = "; ".join(t.mitigations) if t.mitigations else "None documented"
lines.append(
f"| {t.risk_band} ({t.risk_score}) | {t.stride.value} | {t.layer.value} | "
f"{t.atlas_technique} | {t.title} | {mitigations} |"
)
lines += ["", "## Handoff", ""]
lines.append(
"This document is the input to every attack-specific phase that follows: "
"prompt injection defenses target the Prompt/Retrieval-layer threats below, "
"adversarial ML defenses target the Model-layer threats, and guardrail "
"architecture targets the Tool/Agent-layer threats. Nothing later in this "
"course should be built without checking it against this table first."
)
return "\n".join(lines)
def build_capstone_chatbot_model() -> CapstoneThreatModel:
"""A toy customer support chatbot: LLM + RAG knowledge base + refund tool + escalation agent."""
return CapstoneThreatModel(
system_name="Toy Customer Support Chatbot",
description=(
"An LLM-based support chatbot with a RAG knowledge base of support articles, "
"a refund tool, an order-lookup tool, and an escalation sub-agent for complex tickets."
),
assets=[
Asset("System prompt", "medium", "Persona and behavioral constraints"),
Asset("Knowledge-base corpus", "high", "Support articles, some containing customer PII"),
Asset("refund_order credentials", "critical", "API key capable of moving real money"),
Asset("Conversation history", "high", "May contain customer PII and account details"),
],
trust_boundaries=[
"Customer message -> LLM: fully untrusted",
"Knowledge base -> LLM: semi-trusted, editable by support staff, therefore poisonable",
"LLM output -> refund_order tool: must be validated as untrusted, regardless of model provenance",
"Primary agent -> escalation sub-agent: instructions must not be trusted more than their origin warrants",
],
threats=[
Threat(
title="Customer talks the bot into an out-of-policy commitment",
stride=StrideCategory.TAMPERING,
layer=Layer.PROMPT,
atlas_technique="AML.T0054 LLM Jailbreak",
description="Roleplay/hypothetical framing steers the model into a statement the business never authorized (the Chevrolet chatbot pattern).",
likelihood=4, impact=3,
mitigations=["Roleplay-resistant system prompt constraints", "Output filter for commitment-like language"],
),
Threat(
title="Poisoned knowledge-base article carries hidden instructions",
stride=StrideCategory.TAMPERING,
layer=Layer.RETRIEVAL,
atlas_technique="AML.T0051 LLM Prompt Injection",
description="A support article is edited (compromised staff account or malicious insider) to embed instructions the model follows during retrieval-augmented generation (the EchoLeak pattern).",
likelihood=3, impact=5,
mitigations=["Tag retrieved content as untrusted data", "Provenance-track corpus edits"],
),
Threat(
title="System prompt or corpus content is extracted verbatim",
stride=StrideCategory.INFO_DISCLOSURE,
layer=Layer.PROMPT,
atlas_technique="AML.T0056 LLM Meta Prompt Extraction",
description="A crafted prompt causes the model to reveal its system prompt or leak retrieved PII it was told to redact.",
likelihood=3, impact=4,
mitigations=["Treat system prompt as sensitive, not as a security boundary", "Output validation for PII patterns"],
),
Threat(
title="read_order -> summarize -> refund_order composes into unauthorized refund",
stride=StrideCategory.ELEVATION_OF_PRIVILEGE,
layer=Layer.TOOL,
atlas_technique="AML.T0024 Exfiltration via ML Inference API",
description="No single tool call is dangerous alone, but the chain lets injected instructions trigger a real refund.",
likelihood=3, impact=5,
mitigations=["Human approval gate on refund_order", "Recipient/amount allowlist on refund_order"],
),
Threat(
title="Escalation sub-agent trusts instructions from the primary agent by default",
stride=StrideCategory.SPOOFING,
layer=Layer.AGENT,
atlas_technique="AML.T0051 LLM Prompt Injection",
description="If the primary agent is compromised via injection, it can pass malicious instructions to the sub-agent, which grants them elevated trust as 'from the orchestrator.'",
likelihood=2, impact=4,
mitigations=["Sub-agent applies its own constraints regardless of instruction source", "No agent grants elevated trust based on claimed sender identity alone"],
),
Threat(
title="Adversarial input drives unbounded tool-call recursion",
stride=StrideCategory.DENIAL_OF_SERVICE,
layer=Layer.AGENT,
atlas_technique="AML.T0043 Craft Adversarial Data",
description="Crafted input causes the planning loop to recurse through tool calls, driving cost and latency without bound.",
likelihood=3, impact=3,
mitigations=["Hard recursion depth cap", "Per-session rate limiting"],
),
],
)
def main() -> None:
parser = argparse.ArgumentParser(description="Generate the phase capstone threat model.")
parser.add_argument("--output", default="support_bot_threat_model.md")
args = parser.parse_args()
model = build_capstone_chatbot_model()
markdown = model.to_markdown()
with open(args.output, "w") as f:
f.write(markdown)
print(f"Capstone threat model written to {args.output}")
top = model.sorted_threats()[0]
print(f"Highest-risk threat: [{top.risk_band}] {top.title} ({top.atlas_technique})")
print(f"Layers represented: {[l.value for l in model.threats_by_layer().keys()]}")
if __name__ == "__main__":
main()