Phase 0: Security Fundamentals for AI · 55 min · Python · Microsoft AI Red Team methodology
The Problem
Traditional application security has a load-bearing assumption baked into almost every tool and technique it has: given the same input, a program produces the same output, every time. Static analysis, fuzzing, regression testing, even the CIA triad itself (Confidentiality, Integrity, Availability) — all of it assumes you can pin down a fixed set of code paths and reason about what each one does.
An LLM breaks that assumption structurally, not accidentally. The same prompt can produce different outputs on different runs, under different sampling settings, or after a model version update nobody on your team controls. The Chevrolet dealership chatbot from an earlier lesson didn't have a "bug" in the traditional sense — there was no faulty code path to patch. The model did exactly what generative models do: it followed the shape of the conversation it was given. A traditional AppSec review, looking for a specific exploitable code defect, would have found nothing wrong, right up until a customer walked away with a screenshot of a "legally binding" $1 truck offer.
This is why Microsoft's AI Red Team — one of the first dedicated red teams built specifically around this class of system — explicitly treats AI red teaming as a discipline that borrows from but does not reduce to traditional security testing. The threat model has to account for a system that can be steered through its behavior space, not just exploited through a code defect.
Why the CIA triad falls short for AI harms
The CIA triad is still necessary — confidentiality, integrity, and availability failures are all real in AI systems — but several categories of AI harm don't map cleanly onto any of the three.
| Harm | CIA Mapping Attempt | The Gap |
|---|
| Model produces a false but confident answer (hallucination) | Doesn't violate confidentiality, integrity, or availability of any specific asset | No asset was compromised — the harm is in the content of a correctly-functioning system's output |
| Model is talked into an out-of-policy commitment (the Chevy chatbot) | Closest to an integrity failure, but nothing was "modified" — the model just generated text | The harm is reputational/contractual, not a data integrity violation |
| Model exhibits biased or discriminatory behavior at scale | No CIA category captures fairness or disparate impact | This requires a governance/compliance lens (covered later in this course), not a CIA lens |
| Model is steered into generating harmful content that is itself the deliverable (not a leak of anything) | Confidentiality doesn't apply — nothing secret was exposed | The "asset" being protected is the behavior boundary of the system, which CIA has no native vocabulary for |
The practical takeaway: keep using CIA for the failures it does cover (a leaked system prompt is a confidentiality failure; a poisoned RAG corpus is an integrity failure), but recognize you need a second, AI-native lens — behavioral harm, reputational harm, and non-deterministic exploitability — layered on top.
Non-determinism as a security property
In AppSec, "this input reliably triggers the bug" is good news — it means you have a reproducible test case. In AI security, reliability itself is a variable you have to test for, not assume.
This has two concrete consequences that change how you work:
- A single successful jailbreak attempt tells you less than you think. If an attack works once at temperature 0.7, you don't yet know if it works 90% of the time or 9% of the time — and that difference determines whether it's a critical finding or noise. Real AI red team methodology (see Microsoft's published approach) runs attacks across many trials and reports success rate, not a binary pass/fail.
- A single successful defense test tells you less than you think, too. Your guardrail blocking an attack once doesn't mean it blocks that attack reliably. Regression testing an AI system means re-running probes across sampling variance and model version changes, not just re-running a fixed test suite once per release.
Non-determinism is why "add more instructions" doesn't fix injection
A common first instinct when an injection attack succeeds is to add another sentence to the system prompt forbidding that specific attack. This treats the model like deterministic code, where a new if-statement reliably closes a gap. But the model doesn't have code paths to patch — it has a probability distribution over behavior that a new instruction shifts, but does not pin down. This is exactly why structural defenses (data/instruction separation, tool allowlists, human approval gates, sandboxing) outperform "add more instructions": they work by removing the model's ability to cause harm even when it's steered, not by hoping it obeys one more sentence more reliably than the last one.
Trust boundary diagrams for agentic systems
A trust boundary is any point where data crosses from one trust level to another. In a RAG + agent system, there are at least three worth diagramming explicitly:
[User] --(untrusted)--> [LLM] --(semi-trusted, staff-curated)--> [RAG Corpus]
| |
| v
| [Tool Layer] --(model output, must be validated)--> [External Systems]
| |
v v
[Conversation History] [Agent Planning Loop] --(instructions from other agents)--> [Sub-Agents]
Every arrow in that diagram is a place where something crossing the boundary should be validated before the receiving side trusts it. The arrow from LLM output to the Tool Layer is the one teams most often forget to treat as a boundary — it's easy to remember that user input is untrusted, and much easier to forget that the model's own output, on its way to a tool call, is also untrusted from the tool's perspective. The model can be steered; anything it produces on the way to an action needs the same skepticism you'd apply to raw user input.
A team says: "We don't need a trust boundary between our LLM and our send_email tool, because the LLM is our own model that we control and trained ourselves." What's wrong with this reasoning?
Ownership of the model's weights has nothing to do with whether its behavior can be steered by adversarial input at inference time. A self-trained, self-hosted model reading an attacker-crafted email is exactly as vulnerable to indirect prompt injection as a third-party API model would be — the vulnerability lives in the fact that the model conditions its output on untrusted context, not in who owns the weights. The send_email tool call is the boundary that matters, and it needs validation (allowlists, approval gates) regardless of model provenance.
Blast radius and privilege escalation via chained tool calls
"Blast radius" is the answer to: if this component is fully compromised — behaves in the worst possible way an attacker could cause — how bad does it get? For AI systems, the sharpest version of this question is about chains, not single components.
No individual tool call in the email assistant from earlier lessons looks dangerous in isolation: read_email only reads, summarize only transforms text, send_email sends what it's told to send. But read_email -> summarize -> send_email is a complete path from untrusted input to an irreversible external action, and none of the three individual risk reviews would have caught it, because each reviewed one tool at a time.
This is privilege escalation in the AppSec sense — gaining capability the system didn't intend to grant — but the mechanism is composition of intended capabilities, not exploitation of a bug in any one of them. The fix is to model the tool graph as a graph, find every path from a low-risk entry point to a high-risk action, and defend the highest blast-radius paths specifically — not just review each tool's permissions independently.
A security review approves read_email, summarize, and send_email individually, each with a documented, acceptable risk level. The reviewer signs off because no single tool looks dangerous. What does this review miss?
This is the core failure mode of reviewing AI tool permissions the way you'd review individual API endpoints: risk isn't additive across a chain, it's determined by the highest-risk node reachable from an untrusted entry point. read_email is low risk in isolation because it only reads; send_email is high risk but only reachable through a review of that one tool's permissions. Neither review catches that composing them creates a full untrusted-input-to-irreversible-action path. The fix from this lesson — modeling the tool registry as a directed graph and enumerating escalation paths — is designed specifically to catch what per-tool review structurally cannot.
Build
Build it
Here's a scaled-down version of the escalation-path idea: a tiny tool graph where you find every path from a low-risk entry point to a high-risk action.
tools = {
"read_email": 1,
"summarize": 1,
"send_email": 5,
"search_calendar": 1,
"delete_event": 4,
}
edges = {
"read_email": ["summarize", "send_email"],
"summarize": ["send_email"],
"search_calendar": ["delete_event"],
"send_email": [],
"delete_event": [],
}
def find_paths(start, path=None):
path = (path or []) + [start]
results = [path]
for neighbor in edges.get(start, []):
if neighbor not in path:
results.extend(find_paths(neighbor, path))
return results
entry_points = [name for name, risk in tools.items() if risk <= 2]
threshold = 4
print("Escalation paths (entry risk <= 2, reaching risk >= 4):")
for entry in entry_points:
for path in find_paths(entry):
blast = max(tools[n] for n in path)
if blast >= threshold and len(path) > 1:
print(f" blast_radius={blast}: {' -> '.join(path)}")
The code artifact lesson_00_04_blast_radius.py implements:
ToolNode — a tool with a risk_level (1 = read-only/reversible, 5 = irreversible/high blast radius)ToolGraph — a directed graph where an edge (a, b) means output of a can plausibly feed into bentry_points() — tools reachable directly by untrusted input (risk_level <= 2)escalation_paths() — every path from a low-risk entry point to a high-risk tool, de-duplicatedpath_blast_radius() — the highest risk_level of any tool on a given pathto_markdown() — a full report: tool registry table, then escalation paths ranked by blast radius- A worked demo on the email assistant's tool graph from this lesson
"""
lesson_00_04_blast_radius.py
Models an agent's tool registry as a directed graph — an edge from tool A to
tool B means "output of A can plausibly become an input/argument to B" — then
finds every privilege escalation path from a low-risk entry tool to a
high-risk action, and scores each path's blast radius.
This is the concrete version of the question every AI security review has to
answer: "no single tool call looks dangerous, so where's the actual risk?"
Usage:
python lesson_00_04_blast_radius.py
Requirements:
none (standard library only)
"""
from __future__ import annotations
import argparse
from dataclasses import dataclass, field
@dataclass
class ToolNode:
name: str
risk_level: int # 1 (read-only, reversible) .. 5 (irreversible, high blast radius)
description: str
@dataclass
class ToolGraph:
"""A directed graph of tools where an edge (a, b) means output of `a`
can plausibly be used as an argument to `b` in a real agent loop."""
nodes: dict[str, ToolNode] = field(default_factory=dict)
edges: dict[str, list[str]] = field(default_factory=dict)
def add_tool(self, tool: ToolNode) -> None:
self.nodes[tool.name] = tool
self.edges.setdefault(tool.name, [])
def add_edge(self, from_tool: str, to_tool: str) -> None:
self.edges.setdefault(from_tool, []).append(to_tool)
def entry_points(self, max_risk: int = 2) -> list[str]:
"""Tools an untrusted input (user message, retrieved doc) can reach directly."""
return [name for name, t in self.nodes.items() if t.risk_level <= max_risk]
def all_paths_from(self, start: str, visited: frozenset[str] | None = None) -> list[list[str]]:
visited = visited or frozenset()
if start in visited:
return [[start]] # cycle guard: stop, don't loop forever
visited = visited | {start}
paths = [[start]]
for neighbor in self.edges.get(start, []):
for sub_path in self.all_paths_from(neighbor, visited):
paths.append([start] + sub_path)
return paths
def escalation_paths(self, risk_threshold: int = 4) -> list[list[str]]:
"""All paths that start at a low-risk entry point and reach a
high-risk (>= risk_threshold) tool somewhere downstream."""
results = []
for entry in self.entry_points():
for path in self.all_paths_from(entry):
if len(path) < 2:
continue
max_risk_in_path = max(self.nodes[n].risk_level for n in path)
if max_risk_in_path >= risk_threshold:
results.append(path)
# de-duplicate identical paths and drop pure sub-paths of a longer path already included
unique = []
seen = set()
for p in sorted(results, key=len, reverse=True):
key = tuple(p)
if key in seen:
continue
if any(key == tuple(existing[: len(key)]) for existing in unique):
continue
unique.append(p)
seen.add(key)
return unique
def path_blast_radius(self, path: list[str]) -> int:
return max(self.nodes[n].risk_level for n in path)
def to_markdown(self, risk_threshold: int = 4) -> str:
lines = ["# Privilege Escalation Path Analysis", ""]
lines.append("## Tool Registry")
lines.append("")
lines.append("| Tool | Risk Level | Description |")
lines.append("|---|---|---|")
for t in self.nodes.values():
lines.append(f"| {t.name} | {t.risk_level} | {t.description} |")
paths = sorted(
self.escalation_paths(risk_threshold),
key=self.path_blast_radius,
reverse=True,
)
lines += ["", f"## Escalation Paths (entry risk <= 2, reaching risk >= {risk_threshold})", ""]
if not paths:
lines.append("No escalation paths found at this threshold.")
for i, path in enumerate(paths, start=1):
blast = self.path_blast_radius(path)
lines.append(f"{i}. **Blast radius {blast}**: {' -> '.join(path)}")
return "\n".join(lines)
def build_demo_email_assistant() -> ToolGraph:
g = ToolGraph()
g.add_tool(ToolNode("read_email", 1, "Read-only access to inbox content — untrusted input enters here"))
g.add_tool(ToolNode("search_calendar", 1, "Read-only calendar lookup"))
g.add_tool(ToolNode("summarize", 1, "Summarizes arbitrary text; passes summaries onward"))
g.add_tool(ToolNode("create_event", 3, "Writes a new calendar event; reversible but visible to others"))
g.add_tool(ToolNode("send_email", 5, "Sends email as the user; irreversible once delivered"))
g.add_tool(ToolNode("delete_event", 4, "Deletes a calendar event; hard to reverse if the event was important"))
# Realistic data-flow edges: what a planning loop could plausibly chain.
g.add_edge("read_email", "summarize")
g.add_edge("summarize", "send_email") # e.g. "forward a summary of this email"
g.add_edge("read_email", "send_email") # e.g. "reply to this email"
g.add_edge("search_calendar", "create_event")
g.add_edge("search_calendar", "delete_event")
g.add_edge("read_email", "create_event") # e.g. "add the meeting mentioned in this email"
return g
def main() -> None:
parser = argparse.ArgumentParser(description="Find privilege escalation paths through an agent's tool graph.")
parser.add_argument("--threshold", type=int, default=4, help="Minimum risk_level to count as a high-risk target")
parser.add_argument("--output", default="blast_radius.md")
args = parser.parse_args()
graph = build_demo_email_assistant()
report = graph.to_markdown(risk_threshold=args.threshold)
with open(args.output, "w") as f:
f.write(report)
print(f"Report written to {args.output}")
paths = graph.escalation_paths(args.threshold)
print(f"Found {len(paths)} escalation path(s) reaching risk >= {args.threshold}")
for p in sorted(paths, key=graph.path_blast_radius, reverse=True):
print(f" blast_radius={graph.path_blast_radius(p)}: {' -> '.join(p)}")
if __name__ == "__main__":
main()