Phase 1: OWASP Top 10 for LLM Applications · 60 min · Python · Anthropic SDK

LLM06 & LLM07: Sensitive Information Disclosure & Insecure Plugin/Tool Design

A model that leaks what it knows and a tool that trusts what it's told are the same failure — untrusted content treated as authoritative.

Hiring signal: Sensitive information disclosure and insecure plugin design are the two categories that show up first in a real audit — every production LLM app has PII flowing through prompts and at least one tool call. Engineers who can name specific PII/secret patterns to scan for and specific tool permission models to enforce (least privilege, allowlists, output sanitization) demonstrate they've actually reviewed a production system, not just read the OWASP list.

What you will learn

Detect PII, credentials, and secrets in LLM output using pattern-based scanning, and distinguish disclosure via context leakage from disclosure via training-data extraction
Explain how system prompt and training data extraction attacks work and rank the defenses (data minimization, output filtering, PII detection) by effectiveness
Audit a tool/plugin definition for excessive permissions and missing input validation using a least-privilege framework
Identify tool output injection and MCP server vulnerabilities where returned tool content is treated as trusted instructions rather than untrusted data

The Problem

In 2023, Samsung engineers pasted proprietary source code and internal meeting notes into ChatGPT while asking it to debug and summarize — and in doing so, moved that confidential data outside the company's trust boundary into a third party's systems. No prompt injection was involved, no attacker was present. The disclosure happened because nobody had defined a rule for what data is allowed to enter an LLM's context window in the first place. This is the essence of LLM06: Sensitive Information Disclosure — it is not always an attack, it is often a missing control.

The adversarial version of the same category is training data extraction: researchers including Carlini et al. ("Extracting Training Data from Large Language Models," 2021) demonstrated that large language models can memorize and later regurgitate verbatim fragments of their training data — including data the model's operators never intended to expose — when queried in specific ways. Whether the leak is an employee pasting a secret into a prompt or a model reciting memorized training text, the failure mode is the same: sensitive data crossed a trust boundary it should never have crossed.

LLM07: Insecure Plugin/Tool Design is the architectural sibling. As soon as an LLM can call tools — search the web, query a database, read a file, hit an internal API — the tool's permission scope becomes part of the model's effective attack surface. A tool with unrestricted filesystem access or unscoped shell execution doesn't need to be "hacked" for an attacker to abuse it; if an attacker can influence what the model asks the tool to do (via prompt injection, per LLM01) or what the tool returns to the model (tool output injection, the focus of this lesson), the tool's permissions are effectively theirs. The rapid adoption of the Model Context Protocol (MCP) as a standard way to expose tools to LLM agents has made this concrete and urgent: an MCP server is, from a security perspective, a plugin with whatever permissions its author granted it, callable by a model that can be manipulated by anyone who can get text into its context.

LLM06: Sensitive Information Disclosure

System prompt and training data extraction. LLM01 covers extraction via adversarial prompting (covered in lesson 1 of this phase) — this category is broader: it includes accidental disclosure through insufficiently scoped context, insufficient data minimization ("why does this prompt include the customer's full SSN when it only needs to verify the last 4 digits?"), and genuine model memorization of training data that surfaces even without a targeted attack.

PII leakage and credential exposure in context. The most common real-world instance: a support agent, coding assistant, or internal tool has legitimate access to records containing PII or secrets (customer SSNs for fraud checks, API keys for account lookups) because some downstream logic needs them — but nothing stops that same data from appearing verbatim in a user-facing response, a log line, or a third-party API call.

Ranked defenses:

Data minimization — the highest-leverage fix happens before generation: don't put a full SSN, full card number, or raw API key into context if a masked or tokenized reference would do. Data that was never in context cannot leak from context.
Output-side PII/secret detection and redaction — a pattern-based (regex) plus NER-based scan of every response before it leaves the trust boundary, catching what data minimization missed.
System prompt protection — the structural-separation and no-leak-rule techniques from LLM01, applied here as a second layer against extraction attempts specifically targeting configuration or internal codes.
Extraction-pattern monitoring — logging and alerting on requests that resemble known extraction techniques (repetition attacks, meta-questions about training data, requests to "repeat verbatim"), since a single blocked attempt is a signal, not a resolved incident.

If you wouldn't put it in a log file, don't put it in the context window
Context windows get logged, cached, sent to observability tools, and in some deployments used for further training or debugging. Treat every token you put into an LLM's context with the same discipline you'd apply to a log statement — because operationally, it often becomes one.

A customer support agent has database access to full customer records (including SSNs, used only for an internal fraud-check API call the agent invokes). A customer asks, "can you confirm you have my correct info on file?" and the agent responds by listing the customer's full SSN, phone, and email in plain text. No injection attack occurred — the agent just did what it was asked. What is the primary fix?

This is a textbook LLM06 disclosure: the model had legitimate access to sensitive data for one internal purpose (fraud check) but nothing constrained what could appear in the response the customer sees, nor was the amount of raw sensitive data in context minimized to what was actually needed. The fix is two-layered: data minimization (don't put the full SSN in context if only a masked reference is needed downstream) and an output-side filter that redacts sensitive patterns before a response is returned — regardless of whether the customer 'should' be allowed to know their own SSN, echoing it back verbatim in a chat transcript is still a disclosure risk (transcripts get logged, shared, and breached). This isn't an injection issue — no attacker input was involved.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers LLM07: Insecure Plugin/Tool Design, Build It, What to Practice — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.