Phase 2: Prompt Injection & Jailbreaking · 70 min · Python · XML/delimiter structural separation · Policy gating / least privilege design
Defense Architecture — Assume Breach, Trust Tiers & Structural Separation
The model will eventually follow a hostile instruction. Design so that fact doesn't matter.
Hiring signal: When companies give AI security candidates a live review assignment, the single most common instruction is 'assume the model gets jailbroken — what actually stops damage?' Candidates who answer with 'better prompt wording' fail this question. Candidates who answer with tool trust tiers, structural separation, and a policy layer outside the model pass it — this is precisely the distinction Anthropic's own guardrails documentation and every serious AI security engineering role screens for.
What you will learn
- Explain the assume-breach model and design a system's blast radius as if the model will eventually comply with a hostile instruction
- Segregate tools into trust tiers and implement a policy gate that enforces tier restrictions independently of what the model's context claims is authorized
- Implement structural separation of untrusted content using explicit tag boundaries, and explain why this defends against attacks a prompt-only defense cannot
- Diagnose the rule-lawyer problem: why appending more prohibition instructions to a system prompt is defeated by an attacker constructing text that exploits the model's own stated rules
The Problem
Lessons 1 through 3 covered how injection and jailbreaking work: direct overrides, indirect injection through retrieved content, and propagation across multi-agent pipelines and persistent memory. If your takeaway so far is "so we need to write a better system prompt that tells the model not to fall for this," you've absorbed the attacks but not the defense. That instinct is the single most common mistake in production AI security reviews, and it fails for a structural reason, not a tuning reason: a system prompt and an attacker's injected text are the same kind of thing — tokens in a context window — and no amount of additional prohibition text changes that fact.
Anthropic's own guidance on reducing prompt injection risk is explicit about this: prompt-level mitigations reduce risk, they do not eliminate it. The engineering discipline this phase teaches is called the assume-breach model, borrowed directly from traditional security: instead of asking "how do we prevent the model from ever following a hostile instruction" (a losing bet against a probabilistic system), you ask "given that the model will eventually follow a hostile instruction, what is the actual damage, and how do we cap it?" Every defense in this lesson is a way of shrinking that answer toward zero, without depending on the model getting it right.
Assume Breach and Blast Radius
Treat every prompt injection defense the way a security engineer treats a possible RCE in a web app: not as something you can guarantee never happens, but as something whose consequences you design around in advance. Concretely, for every tool and every piece of context you give a model, ask: if the model is fully compromised right now and tries to do the worst plausible thing with this capability, what happens?
For a web_search tool, the worst case is a wasted API call. For a send_email tool with no further checks, the worst case is data exfiltration to an attacker-controlled address — exactly the payload from lesson 3's multi-agent pipeline demo. For a delete_file or transfer_funds tool, the worst case can be irreversible. The assume-breach model doesn't try to stop the compromise from happening (that's the losing bet); it caps the blast radius so that even a fully successful injection produces a wasted API call rather than a wire transfer.
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers Trust-Tier Segregation, Structural Separation: Giving the Model an Actual Boundary, Why "Just Add More Instructions" Fails: The Rule-Lawyer Problem, Build It, What to Practice — plus a hands-on lab, quiz, and project artifact.
Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.
Browse all courses · View pricing · DeVenture Academy