Phase 1: OWASP Top 10 for LLM Applications · 60 min · Python · Anthropic SDK

LLM08 & LLM09: Excessive Agency & Overreliance

An agent that can act without approval and a human who trusts output without verification fail for the same reason — nobody is checking before the consequence lands.

Hiring signal: As agentic systems ship to production, excessive agency and overreliance are overtaking prompt injection as the categories that cause real financial and safety incidents — an agent that autonomously executes a high-risk action, or a team that ships an unverified hallucinated recommendation in a high-stakes context, is a governance failure as much as a technical one. Engineers who can design approval gates, step budgets, and confidence-scoring pipelines demonstrate they think about AI systems the way a safety engineer thinks about any autonomous system, not just as a prompting problem.

What you will learn

Design an action-approval gate that classifies agent actions by risk and reversibility, and requires human-in-the-loop approval above a defined blast-radius threshold
Explain how privilege escalation happens through agent tool chains, where a low-privilege step's output enables a subsequent higher-privilege action
Implement step budgets and blast-radius limits that bound how much an autonomous agent can do before requiring a human check-in
Build a confidence/citation-based overreliance checker that flags unverified, high-confidence claims in high-stakes domains for mandatory human review

The Problem

In 2024, an Air Canada customer-support chatbot told a passenger he could book a full-fare ticket and apply for a bereavement discount retroactively — a policy the airline did not actually have. The airline argued the chatbot was "a separate legal entity" responsible for its own words; a Canadian tribunal disagreed and ordered Air Canada to honor the discount, because the company had deployed the agent, trusted its output, and shipped it without a control that would catch a confident, wrong answer before it reached a customer. That is LLM09 (Overreliance) in a single incident: nobody hallucination-checked a high-stakes claim before it became a binding promise.

LLM08 (Excessive Agency) is the same failure one layer up the stack: instead of a human trusting bad output, it is a system letting an agent act on its own conclusions without anyone checking first. As agentic products ship — agents that can send emails, execute trades, modify infrastructure, refund customers, or call arbitrary tools — the blast radius of a single wrong inference stops being "an embarrassing chat transcript" and starts being "an irreversible action taken in the world." OWASP tracks LLM08 and LLM09 together because they compound: an overreliant human is the last line of defense against an excessively agentic system, and if that human isn't actually in the loop — or doesn't verify what they're rubber-stamping — the system has no check left at all.

LLM08: Excessive Agency

Autonomous actions without approval. An agent with tool access will, by construction, act on whatever it currently believes is the right next step. If nothing external validates that belief before the tool call executes, the agent's confidence is the only gate — and confidence is not correctness. This is distinct from prompt injection (LLM01): even with a perfectly injection-resistant agent, an agent that is simply wrong about what action to take, acting entirely on its own reasoning, can still cause damage if nothing stops it before execution.

Multi-step social engineering of agents. An attacker doesn't need to inject a single malicious instruction if they can instead have a normal-looking conversation that gradually shifts the agent's understanding of what's "authorized" — asking it to "help debug" a permission check, then to "temporarily" bypass a safeguard "just for this test," then to perform the now-unblocked action. Each individual message can look reasonable in isolation while the conversation as a whole executes a jailbreak.

Privilege escalation through agent tool chains. This is the pattern that matters most in production: no single tool call looks dangerous, but a chain of them does. An agent that reads a config file (low risk) might discover a credential inside it, use that credential to authenticate to an internal API (now elevated privilege), and then use that access to perform a destructive action — and a risk model that only scores each step in isolation will approve all three, because step 2's risk score doesn't know it depends on what step 1 found. This mirrors classic privilege-escalation chains in traditional infosec, except the "attacker" chaining the steps might just be the model's own plan, with no adversary required at all.

Ranked defenses:

Action-approval gates by risk and reversibility — classify every action the agent can take by (a) how much damage it can do and (b) whether it can be undone; auto-approve only actions that are both low-risk and reversible, and require a human for everything else. This decision must live outside the model, as a policy the agent cannot talk its way around.
Step budgets — cap the number of autonomous tool calls an agent can make in a session before it must check in with a human, independent of any single step's risk score. This is the specific control that catches privilege-escalation chains, since it doesn't rely on any one step "looking" dangerous.
Blast-radius limits — bound the scope of what a single action (or a session) can affect: a spending cap per transaction, a maximum number of records an agent can modify per run, a maximum number of emails it can send per hour.
Human-in-the-loop for irreversible or high-privilege actions — no autonomous path should exist to an action that cannot be undone (deleting data, sending money, terminating infrastructure) without a human explicitly confirming it, every time.

An agent's tool-call risk gate auto-approves any single action rated LOW risk and reversible. In a session, the agent takes three LOW-risk, reversible actions in a row: it reads a deployment config, uses a value it found there to authenticate to an internal service, and then uses that authenticated session to trigger a production deploy. Each step individually passes the gate. What control specifically catches this pattern that a per-action risk gate alone does not?

This is privilege escalation through a tool chain, the specific failure mode OWASP calls out beyond single-action risk scoring: step 1's output (a discovered credential) is what makes step 2 possible, and step 2's elevated access is what makes step 3 dangerous — but scored independently, each step looks fine. A per-action gate has no way to see this because it evaluates actions one at a time. A step budget is the correct control because it doesn't need to understand why the chain is dangerous; it simply refuses to let any chain of autonomous actions run past a fixed length without a human confirming the agent should keep going. A system-prompt instruction is not a control at all — it is a request the model can fail to follow, which is exactly the trust LLM08 says not to extend to a model's own judgment.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers LLM09: Overreliance, Build It, What to Practice — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.

Key terms in this lesson

Agent