Phase 0: Security Fundamentals for AI · 55 min · Python · Microsoft AI Red Team methodology

AppSec vs AI Security — Trust Boundaries & Blast Radius

In AppSec, the same input always produces the same output. In AI security, that assumption is gone.

Hiring signal: Senior AI security hires are distinguished by whether they can explain blast radius and privilege escalation paths before a single line of code is written — this is what separates an AppSec generalist from an AI security specialist in a technical interview.

What you will learn

Explain why the CIA triad alone is insufficient for describing AI-specific harms
Treat non-determinism as a security property that changes how you test and defend a system, not just a UX quirk
Draw a trust boundary diagram for an agentic system with retrieval and tool use
Trace a privilege escalation path through chained tool calls and estimate its blast radius

The Problem

Traditional application security has a load-bearing assumption baked into almost every tool and technique it has: given the same input, a program produces the same output, every time. Static analysis, fuzzing, regression testing, even the CIA triad itself (Confidentiality, Integrity, Availability) — all of it assumes you can pin down a fixed set of code paths and reason about what each one does.

An LLM breaks that assumption structurally, not accidentally. The same prompt can produce different outputs on different runs, under different sampling settings, or after a model version update nobody on your team controls. The Chevrolet dealership chatbot from an earlier lesson didn't have a "bug" in the traditional sense — there was no faulty code path to patch. The model did exactly what generative models do: it followed the shape of the conversation it was given. A traditional AppSec review, looking for a specific exploitable code defect, would have found nothing wrong, right up until a customer walked away with a screenshot of a "legally binding" $1 truck offer.

This is why Microsoft's AI Red Team — one of the first dedicated red teams built specifically around this class of system — explicitly treats AI red teaming as a discipline that borrows from but does not reduce to traditional security testing. The threat model has to account for a system that can be steered through its behavior space, not just exploited through a code defect.

Why the CIA Triad Falls Short for AI Harms

The CIA triad is still necessary — confidentiality, integrity, and availability failures are all real in AI systems — but several categories of AI harm don't map cleanly onto any of the three.

Harm	CIA Mapping Attempt	The Gap
Model produces a false but confident answer (hallucination)	Doesn't violate confidentiality, integrity, or availability of any specific asset	No asset was compromised — the harm is in the content of a correctly-functioning system's output
Model is talked into an out-of-policy commitment (the Chevy chatbot)	Closest to an integrity failure, but nothing was "modified" — the model just generated text	The harm is reputational/contractual, not a data integrity violation
Model exhibits biased or discriminatory behavior at scale	No CIA category captures fairness or disparate impact	This requires a governance/compliance lens (covered later in this course), not a CIA lens
Model is steered into generating harmful content that is itself the deliverable (not a leak of anything)	Confidentiality doesn't apply — nothing secret was exposed	The "asset" being protected is the behavior boundary of the system, which CIA has no native vocabulary for

The practical takeaway: keep using CIA for the failures it does cover (a leaked system prompt is a confidentiality failure; a poisoned RAG corpus is an integrity failure), but recognize you need a second, AI-native lens — behavioral harm, reputational harm, and non-deterministic exploitability — layered on top.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Non-Determinism as a Security Property, Trust Boundary Diagrams for Agentic Systems, Blast Radius and Privilege Escalation via Chained Tool Calls, Build It, What to Practice — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.