Phase 1: OWASP Top 10 for LLM Applications · 60 min · Python · Anthropic SDK
LLM08 & LLM09: Excessive Agency & Overreliance
An agent that can act without approval and a human who trusts output without verification fail for the same reason — nobody is checking before the consequence lands.
Real-world application: As agentic systems ship to production, excessive agency and overreliance are overtaking prompt injection as the categories that cause real financial and safety incidents — an agent that autonomously executes a high-risk action, or a team that ships an unverified hallucinated recommendation in a high-stakes context, is a governance failure as much as a technical one. Engineers who can design approval gates, step budgets, and confidence-scoring pipelines demonstrate they think about AI systems the way a safety engineer thinks about any autonomous system, not just as a prompting problem.
What you will learn
- Design an action-approval gate that classifies agent actions by risk and reversibility, and requires human-in-the-loop approval above a defined blast-radius threshold
- Explain how privilege escalation happens through agent tool chains, where a low-privilege step's output enables a subsequent higher-privilege action
- Implement step budgets and blast-radius limits that bound how much an autonomous agent can do before requiring a human check-in
- Build a confidence/citation-based overreliance checker that flags unverified, high-confidence claims in high-stakes domains for mandatory human review
The Problem
In 2024, an Air Canada customer-support chatbot told a passenger he could book a full-fare ticket and apply for a bereavement discount retroactively — a policy the airline did not actually have. The airline argued the chatbot was "a separate legal entity" responsible for its own words; a Canadian tribunal disagreed and ordered Air Canada to honor the discount, because the company had deployed the agent, trusted its output, and shipped it without a control that would catch a confident, wrong answer before it reached a customer. That is LLM09 (Overreliance) in a single incident: nobody hallucination-checked a high-stakes claim before it became a binding promise.
LLM08 (Excessive Agency) is the same failure one layer up the stack: instead of a human trusting bad output, it is a system letting an agent act on its own conclusions without anyone checking first. As agentic products ship — agents that can send emails, execute trades, modify infrastructure, refund customers, or call arbitrary tools — the blast radius of a single wrong inference stops being "an embarrassing chat transcript" and starts being "an irreversible action taken in the world." OWASP tracks LLM08 and LLM09 together because they compound: an overreliant human is the last line of defense against an excessively agentic system, and if that human isn't actually in the loop — or doesn't verify what they're rubber-stamping — the system has no check left at all.
LLM08: excessive agency
Autonomous actions without approval. An agent with tool access will, by construction, act on whatever it currently believes is the right next step. If nothing external validates that belief before the tool call executes, the agent's confidence is the only gate — and confidence is not correctness. This is distinct from prompt injection (LLM01): even with a perfectly injection-resistant agent, an agent that is simply wrong about what action to take, acting entirely on its own reasoning, can still cause damage if nothing stops it before execution.
Multi-step social engineering of agents. An attacker doesn't need to inject a single malicious instruction if they can instead have a normal-looking conversation that gradually shifts the agent's understanding of what's "authorized" — asking it to "help debug" a permission check, then to "temporarily" bypass a safeguard "just for this test," then to perform the now-unblocked action. Each individual message can look reasonable in isolation while the conversation as a whole executes a jailbreak.
Privilege escalation through agent tool chains. This is the pattern that matters most in production: no single tool call looks dangerous, but a chain of them does. An agent that reads a config file (low risk) might discover a credential inside it, use that credential to authenticate to an internal API (now elevated privilege), and then use that access to perform a destructive action — and a risk model that only scores each step in isolation will approve all three, because step 2's risk score doesn't know it depends on what step 1 found. This mirrors classic privilege-escalation chains in traditional infosec, except the "attacker" chaining the steps might just be the model's own plan, with no adversary required at all.
Ranked defenses:
- Action-approval gates by risk and reversibility — classify every action the agent can take by (a) how much damage it can do and (b) whether it can be undone; auto-approve only actions that are both low-risk and reversible, and require a human for everything else. This decision must live outside the model, as a policy the agent cannot talk its way around.
- Step budgets — cap the number of autonomous tool calls an agent can make in a session before it must check in with a human, independent of any single step's risk score. This is the specific control that catches privilege-escalation chains, since it doesn't rely on any one step "looking" dangerous.
- Blast-radius limits — bound the scope of what a single action (or a session) can affect: a spending cap per transaction, a maximum number of records an agent can modify per run, a maximum number of emails it can send per hour.
- Human-in-the-loop for irreversible or high-privilege actions — no autonomous path should exist to an action that cannot be undone (deleting data, sending money, terminating infrastructure) without a human explicitly confirming it, every time.
An agent's tool-call risk gate auto-approves any single action rated LOW risk and reversible. In a session, the agent takes three LOW-risk, reversible actions in a row: it reads a deployment config, uses a value it found there to authenticate to an internal service, and then uses that authenticated session to trigger a production deploy. Each step individually passes the gate. What control specifically catches this pattern that a per-action risk gate alone does not?
This is privilege escalation through a tool chain, the specific failure mode OWASP calls out beyond single-action risk scoring: step 1's output (a discovered credential) is what makes step 2 possible, and step 2's elevated access is what makes step 3 dangerous — but scored independently, each step looks fine. A per-action gate has no way to see this because it evaluates actions one at a time. A step budget is the correct control because it doesn't need to understand why the chain is dangerous; it simply refuses to let any chain of autonomous actions run past a fixed length without a human confirming the agent should keep going. A system-prompt instruction is not a control at all — it is a request the model can fail to follow, which is exactly the trust LLM08 says not to extend to a model's own judgment.
LLM09: overreliance
Automation bias. Automation bias is the well-documented human tendency to trust a system's output more than warranted simply because it came from a system — reviewers approve AI-suggested code faster and more often than human-written code of the same quality, and users skip verification steps for confident-sounding LLM output specifically because it sounds confident, not because it has been checked. LLMs make this worse than earlier automation because their failure mode — a fluent, well-formatted, grammatically confident hallucination — is indistinguishable in tone from a correct answer. A calculator that's wrong usually looks wrong (a wildly implausible number); an LLM that's wrong usually looks exactly like an LLM that's right.
Hallucination in high-stakes contexts. The Air Canada case is one instance of a broader pattern that has repeated across law (lawyers submitting briefs with fabricated case citations an LLM invented), healthcare (unverified triage suggestions), and finance (confident-sounding investment claims with no basis). The technical fact that makes this dangerous isn't that models sometimes hallucinate — every generative model does — it's that hallucinated output is not marked as such. The system produces uncertain claims and confident claims in the identical register, and without an explicit uncertainty signal, a human downstream has no cue to slow down and verify.
A confidence score the model reports about itself is not a safety control
Asking a model "how confident are you?" and trusting the number it returns doesn't fix overreliance — the model can be just as fluently wrong about its own confidence as it is about the underlying fact. The control that actually works is external and structural: does this specific output cite a verifiable source, is this domain high-stakes enough to require one, and if not, does a human review it before anyone acts on it. Confidence has to be measured by evidence present in the text, not self-reported by the system being checked.
Ranked defenses:
- Mandatory citations in high-stakes domains — for medical, legal, financial, or safety-critical output, require the response to include verifiable sources, and treat confident, uncited claims in these domains as high-risk by policy, not by asking the model to self-assess.
- Confidence-and-hedge language detection — flag responses that use unhedged, absolute language ("definitely," "guaranteed," "always") in a high-stakes domain without a citation; this doesn't determine whether the claim is true, but it identifies exactly the automation-bias risk pattern where a human is most likely to skip verification.
- Mandatory human review gates for high-stakes categories — some domains should never ship an LLM response directly to an end user without a human reviewing it first, independent of how confident or well-cited the output looks; this is a policy decision, not a technical one, but it has to be enforced technically (a hold state, not a suggestion).
- Calibration and disclosure — where full human review isn't feasible at scale, at minimum disclose the system's known error rate and encourage independent verification, so users have an accurate prior instead of an implicit assumption of correctness.
A legal-assistant LLM produces this response to a user: "You are guaranteed to win this motion under state precedent [1], per Reyes v. Whitfield (2021)." The citation format looks legitimate. What is the single most important thing to verify before this response reaches a paying customer, and why does a well-formatted citation not resolve the concern on its own?
This is the core of LLM09 in a legal context: a fabricated citation is often indistinguishable in format from a real one — correct case-name structure, a plausible year, a bracketed reference marker — which is exactly why format-based citation detection (does a citation-shaped string exist) is not the same control as citation verification (does that specific citation resolve to a real, on-point source). Multiple real incidents involve lawyers submitting briefs with LLM-fabricated case law that passed a cursory glance specifically because it looked like a normal citation. In a high-stakes domain, "guaranteed... per [citation]" is exactly the confident-plus-cited pattern that still requires human verification, because the presence of a citation marker is necessary but not sufficient evidence of a correct claim.
Build
Build it
A minimal version of the action-approval gate and step budget before the full artifact.
actions = [
("read_deploy_config", "LOW", True),
("call_admin_api_with_discovered_token", "HIGH", True),
("delete_production_database", "CRITICAL", False),
]
RISK_ORDER = {"LOW": 1, "MEDIUM": 2, "HIGH": 3, "CRITICAL": 4}
max_autonomous_steps = 2
steps_taken = 0
for name, risk, reversible in actions:
steps_taken += 1
if steps_taken > max_autonomous_steps:
print(f"{name}: STEP BUDGET EXCEEDED -> halt, require human check-in")
break
if RISK_ORDER[risk] >= RISK_ORDER["CRITICAL"] and not reversible:
print(f"{name}: BLOCKED (critical + irreversible)")
break
if RISK_ORDER[risk] <= RISK_ORDER["LOW"] and reversible:
print(f"{name}: auto_approved")
else:
print(f"{name}: requires_human_approval")
The code artifact lesson_01_05_agency_overreliance.py implements both parts:
ActionApprovalGate — classifies proposed agent actions by risk level and reversibility, auto-approving only low-risk reversible actions and requiring human approval or blocking everything else (LLM08)StepBudgetTracker — caps autonomous tool-call steps per session to catch privilege-escalation chains that a per-action risk gate misses (LLM08)OverrelianceChecker — flags LLM output making unhedged, high-confidence claims in a high-stakes domain without citations, the practical signature of automation-bias risk (LLM09)
"""
lesson_01_05_agency_overreliance.py
LLM08 (Excessive Agency) + LLM09 (Overreliance).
Part A — Excessive Agency:
Simulates an agent proposing a sequence of actions with increasing risk and
blast radius, and demonstrates an ActionApprovalGate that auto-approves
low-risk reversible actions, requires human approval above a threshold,
and a StepBudgetTracker that halts an autonomous chain before it can
escalate privilege across multiple tool calls without a human check-in.
Part B — Overreliance:
Simulates a confidence/citation-based checker that flags LLM output making
high-confidence factual claims in a high-stakes domain without citations —
the practical signature of automation bias risk, where an unverified
hallucination could otherwise be acted on as if it were fact.
Usage:
export ANTHROPIC_API_KEY=sk-ant-... # optional
python lesson_01_05_agency_overreliance.py
Requirements:
pip install anthropic # optional — script runs in mock mode without it
"""
from __future__ import annotations
import os
import re
from dataclasses import dataclass, field
from enum import IntEnum
try:
import anthropic
_client = anthropic.Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY", ""))
except ImportError:
_client = None
MODEL = "claude-haiku-4-5"
# ══════════════════════════════════════════════════════════════════════════
# PART A — Excessive Agency (LLM08)
# ══════════════════════════════════════════════════════════════════════════
class RiskLevel(IntEnum):
LOW = 1
MEDIUM = 2
HIGH = 3
CRITICAL = 4
@dataclass
class AgentAction:
name: str
risk_level: RiskLevel
reversible: bool
blast_radius: str # human-readable description of who/what is affected
grants_privilege_for: str | None = None # if this action's output enables a later action
@dataclass
class GateDecision:
action: str
decision: str # "auto_approved" | "requires_human_approval" | "blocked"
reason: str
class ActionApprovalGate:
"""
Classifies proposed agent actions by risk and reversibility, and decides
whether the action can proceed autonomously, requires a human in the
loop, or should be blocked outright. This is the core LLM08 control:
excessive agency isn't fixed by trusting the model to 'know better,' it's
fixed by an external policy that doesn't ask the model's permission.
"""
def __init__(self, auto_approve_ceiling: RiskLevel = RiskLevel.LOW, block_floor: RiskLevel = RiskLevel.CRITICAL):
self.auto_approve_ceiling = auto_approve_ceiling
self.block_floor = block_floor
def evaluate(self, action: AgentAction) -> GateDecision:
if action.risk_level >= self.block_floor and not action.reversible:
return GateDecision(action.name, "blocked",
f"Risk={action.risk_level.name}, irreversible — blast radius '{action.blast_radius}' "
f"exceeds what any autonomous action should be permitted to do without redesign.")
if action.risk_level <= self.auto_approve_ceiling and action.reversible:
return GateDecision(action.name, "auto_approved",
f"Risk={action.risk_level.name}, reversible — within autonomous bounds.")
return GateDecision(action.name, "requires_human_approval",
f"Risk={action.risk_level.name}, reversible={action.reversible} — "
f"blast radius '{action.blast_radius}' requires a human to approve before executing.")
class StepBudgetTracker:
"""
Caps the number of autonomous tool-call steps an agent can take in a
single session before requiring a human check-in, independent of any
single action's individual risk rating. This catches privilege
escalation chains: no single step looks dangerous, but a long
uninterrupted chain of low-risk steps can compound into a high-privilege
outcome (e.g. read config -> discover a credential -> use it -> escalate).
"""
def __init__(self, max_autonomous_steps: int = 3):
self.max_autonomous_steps = max_autonomous_steps
self.steps_taken = 0
def record_step(self) -> bool:
"""Returns True if the step is allowed to proceed autonomously."""
self.steps_taken += 1
return self.steps_taken <= self.max_autonomous_steps
def remaining(self) -> int:
return max(0, self.max_autonomous_steps - self.steps_taken)
# A realistic escalation chain: each step looks individually reasonable, but
# the chain as a whole moves from "read a config file" to "delete a
# database" — a privilege escalation path through an agent's tool calls,
# the specific pattern LLM08 calls out as the dangerous case beyond any
# single action's risk score.
ESCALATION_CHAIN = [
AgentAction("read_deploy_config", RiskLevel.LOW, reversible=True,
blast_radius="reads a config file", grants_privilege_for="discovers admin_token field"),
AgentAction("call_internal_admin_api_with_discovered_token", RiskLevel.HIGH, reversible=True,
blast_radius="authenticates as an admin service account using a token found in step 1"),
AgentAction("delete_production_database", RiskLevel.CRITICAL, reversible=False,
blast_radius="irreversibly destroys production data for all tenants"),
]
def demo_excessive_agency() -> None:
print("=" * 70)
print("PART A: EXCESSIVE AGENCY (LLM08)")
print("=" * 70)
gate = ActionApprovalGate()
budget = StepBudgetTracker(max_autonomous_steps=2)
print("\nAgent proposes a chain of actions (each looks individually plausible):\n")
for i, action in enumerate(ESCALATION_CHAIN, start=1):
decision = gate.evaluate(action)
within_budget = budget.record_step()
print(f"Step {i}: {action.name}")
print(f" risk={action.risk_level.name:8s} reversible={action.reversible!s:5} blast_radius='{action.blast_radius}'")
print(f" gate decision: {decision.decision} — {decision.reason}")
if not within_budget:
print(f" STEP BUDGET EXCEEDED ({budget.max_autonomous_steps} autonomous steps used) — "
f"halting the chain and requiring a human check-in before step {i} proceeds, "
f"regardless of this step's individual risk rating.")
break
if decision.decision == "blocked":
print(" Chain halted: this action is blocked outright, not just gated.")
break
print()
print("\nInterpretation: step 1 alone is low-risk. Step 2 uses what step 1 discovered to gain "
"admin-level access — this is privilege escalation through the tool chain, not through any "
"single action's stated permissions. A step budget catches this even before the gate's "
"per-action risk score would, because it doesn't trust that 'no single step looked dangerous' "
"means the chain as a whole is safe.")
# ══════════════════════════════════════════════════════════════════════════
# PART B — Overreliance (LLM09)
# ══════════════════════════════════════════════════════════════════════════
HIGH_CONFIDENCE_PHRASES = [
r"\bdefinitely\b", r"\bguaranteed\b", r"\b100%\b", r"\bcertainly\b",
r"\bwithout (?:a )?doubt\b", r"\balways works\b", r"\bnever fails\b",
]
HEDGE_PHRASES = [
r"\bmay\b", r"\bmight\b", r"\bcould\b", r"\bpossibly\b", r"\bI'?m not certain\b",
r"\baccording to\b", r"\bbased on\b", r"\byou should verify\b", r"\bconsult a\b",
]
CITATION_MARKERS = [r"\[\d+\]", r"\(source:.*?\)", r"https?://\S+", r"per [A-Z][\w. ]+ \(\d{4}\)"]
HIGH_STAKES_DOMAINS = {"medical", "legal", "financial", "safety-critical"}
@dataclass
class OverrelianceFinding:
domain: str
has_high_confidence_language: bool
has_hedging_language: bool
has_citations: bool
requires_human_review: bool
reason: str
class OverrelianceChecker:
"""
Flags LLM output for mandatory human review when it makes unhedged,
high-confidence factual claims in a high-stakes domain without any
citation or sourcing. This does not detect whether a claim is *true* —
it detects the automation-bias risk pattern: confident, uncited output
in a domain where an unverified hallucination could cause real harm if
a human simply trusts it.
"""
def check(self, domain: str, response_text: str) -> OverrelianceFinding:
has_confidence = any(re.search(p, response_text, re.IGNORECASE) for p in HIGH_CONFIDENCE_PHRASES)
has_hedge = any(re.search(p, response_text, re.IGNORECASE) for p in HEDGE_PHRASES)
has_citation = any(re.search(p, response_text, re.IGNORECASE) for p in CITATION_MARKERS)
is_high_stakes = domain in HIGH_STAKES_DOMAINS
requires_review = is_high_stakes and has_confidence and not has_citation
if requires_review:
reason = (f"High-stakes domain ('{domain}') + high-confidence language + no citation — "
f"this is exactly the pattern where automation bias leads a human to act on an "
f"unverified claim as if it were fact.")
elif is_high_stakes and not has_citation and not has_hedge:
reason = (f"High-stakes domain with no citation AND no hedging language — even without an "
f"explicit 'definitely,' an unqualified factual-sounding claim in this domain "
f"warrants verification before acting on it.")
requires_review = True
else:
reason = "Either not a high-stakes domain, or the response is appropriately hedged/cited."
return OverrelianceFinding(
domain=domain, has_high_confidence_language=has_confidence,
has_hedging_language=has_hedge, has_citations=has_citation,
requires_human_review=requires_review, reason=reason,
)
RESPONSE_SAMPLES = [
{
"domain": "medical",
"text": "This combination of symptoms definitely indicates condition X — start treatment Y immediately, it always works for this presentation.",
},
{
"domain": "medical",
"text": "These symptoms may be consistent with condition X, but you should consult a physician for a proper diagnosis before starting any treatment.",
},
{
"domain": "legal",
"text": "You are guaranteed to win this case under state law [1], per Smith v. Jones (2019).",
},
{
"domain": "general",
"text": "The capital of France is Paris.",
},
]
def demo_overreliance() -> None:
print("\n" + "=" * 70)
print("PART B: OVERRELIANCE (LLM09)")
print("=" * 70)
checker = OverrelianceChecker()
for sample in RESPONSE_SAMPLES:
finding = checker.check(sample["domain"], sample["text"])
print(f"\n[domain={finding.domain}] \"{sample['text'][:70]}...\"")
print(f" high_confidence_language={finding.has_high_confidence_language} "
f"hedged={finding.has_hedging_language} cited={finding.has_citations}")
print(f" requires_human_review={finding.requires_human_review}")
print(f" reason: {finding.reason}")
if __name__ == "__main__":
demo_excessive_agency()
demo_overreliance()
print("\n" + "=" * 70)
print("TAKEAWAY")
print("=" * 70)
print("LLM08 defenses bound what an agent can DO without a human: per-action risk gates, "
"step budgets that catch multi-step privilege escalation even when no single step looks "
"dangerous, and blast-radius limits on irreversible actions. LLM09 defenses bound what a "
"human TRUSTS from a model without verification: flagging confident, uncited claims in "
"high-stakes domains for mandatory review, because the failure mode isn't the model being "
"wrong — it's a human acting on wrong output as if it had been checked.")
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers Check Yourself — plus a hands-on lab, quiz, and project artifact.
Create a free account — the opening phases of 29 of 35 courses are free, no credit card.
Browse all courses · View pricing · DeVenture Academy