Phase 8: Production AI Security Architecture · 65 min · OWASP GenAI Security Project · NeMo Guardrails · OAuth 2.0
The Problem
Every phase of this course so far has taught you to attack or defend one piece of an AI system: injection detection, RAG isolation, agent tool authorization, adversarial robustness, guardrail pipelines, red team methodology, compliance mapping. In production, none of those live alone. They are layers in a single request path, and the security of the system is determined by how those layers compose — not by how good any single layer is.
This is where most teams get it wrong. A team ships a strong prompt-injection classifier at the input layer and treats the system as "secure." Then a tool-output poisoning attack (Phase 4) walks straight through, because nothing at the guardrails or output layer was checking tool results. Or a team locks down guardrails carefully, then a model version bump silently changes refusal behavior with no anomaly detection to catch it (Phase 3, model layer) — the CrowdStrike-style "we shipped a change with no rollback path" failure, just for a model instead of a kernel driver.
Real incidents follow this pattern. Indirect prompt injection vulnerabilities like the 2025 "EchoLeak" class of issues in Microsoft 365 Copilot were exploitable specifically because a single layer — content ingested from email/documents — was trusted as instruction-equivalent, with no independent layer downstream to catch exfiltration via crafted markdown links. A layered architecture doesn't prevent every attack. It ensures that when one layer is bypassed, at least one other layer still has a chance to catch it before it becomes an incident.
The reference architecture below is the one you'll be expected to draw, defend, and — in this lesson — actually validate against a config, in an AI Security Architect design review.
The seven layers
A request into a production LLM application passes through seven layers, in this order:
| # | Layer | Purpose | Typical controls |
|---|
| 1 | Network | Stop volumetric/infra attacks before the app sees them | WAF, DDoS protection, API gateway, TLS |
| 2 | Auth | Establish who is calling and what they can do | OAuth 2.0, scoped API keys, mTLS, key rotation |
| 3 | Input | Validate and classify everything before it becomes a prompt | Schema validation, injection detection, trust-tier tagging |
| 4 | Guardrails | Enforce policy independent of the model's own judgment | Topic rails, PII filtering, tool authorization, safety classifiers |
| 5 | Model | Control which model runs and detect behavioral drift | Version pinning, anomaly detection, drift monitoring |
| 6 | Output | Treat generated content as untrusted until validated | Response filtering, fact verification, output encoding |
| 7 | Audit | Make every decision reconstructable after the fact | Full logging, retention, tamper-evident storage |
Each layer has a specific failure mode it exists to catch — not a generic "more security is better" justification. If you can't state the failure mode, you can't tell whether the layer is actually doing anything:
- Network fails when unbounded traffic reaches the model endpoint directly — enabling scraping, credential stuffing, and denial of service on token spend.
- Auth fails when any caller with a plausible-looking token gets full access — no scoping means a leaked customer-support key can call admin operations.
- Input fails when untrusted content (user input, retrieved documents, tool output) reaches the model without being flagged as untrusted — this is the root cause of nearly every prompt injection incident from Phase 2.
- Guardrails fail when the model is the only thing enforcing policy — one successful jailbreak (Phase 2) removes every restriction at once.
- Model fails when a version or prompt-template change ships with no eval gate and no rollback — silent regressions in refusal behavior or output quality go undetected until a user reports them.
- Output fails when generated content is rendered, executed, or stored without validation — OWASP's insecure output handling category (LLM02) covers exactly this: XSS, SSRF, and SQLi caused by trusting model output.
- Audit fails silently, until the moment you need it: an incident happens and there's no record of the prompt, the retrieved context, or which model version produced the bad output. You cannot write an incident report (Phase 5 of this course) without this layer.
Trust boundaries and blast radius
The layers only work as defense in depth if you've correctly identified where trust boundaries actually are. Phase 2 introduced the assume-breach model for prompt injection specifically; the same principle generalizes to the whole architecture: assume any single layer will eventually be bypassed, and design so that the blast radius of that bypass is contained by the next layer.
Concretely, this means:
- Content crossing the input layer from an untrusted source (user text, retrieved RAG chunks, tool output, email/document content) is tagged as data, never merged into the instruction channel. This is a guardrails-layer control (
source_tagging), not just an input-layer one — it has to survive the trip through the model. - A tool-call authorization gate at the guardrails layer exists specifically because the model layer might be tricked into requesting an unauthorized action. If your only tool-safety control is "the model was told not to," you have zero layers, not seven.
- Output encoding at the output layer exists because a prompt injection that slips past input detection and guardrails can still only do damage if the output layer trusts what comes back. Insecure output handling is the layer of last resort before a compromised response reaches a browser, a database, or a shell.
Layers compose multiplicatively, not additively
If each of your seven layers independently catches 60% of a given attack class, a naive read says you're "well covered." But layers that are supposed to be independent often share a blind spot — e.g., input-layer injection detection and guardrails-layer topic restriction might both rely on the same underlying classifier, so they fail together. The real question in a design review is not "how many layers do you have" but "which layers are genuinely independent failure domains." Two layers built on the same model or the same heuristic are one layer wearing a disguise.
A team implements PII filtering at the guardrails layer using a regex-based detector, and separately implements response filtering at the output layer using the same regex patterns imported from the same module. An attacker crafts a payload that evades the regex. What is the actual defense-in-depth value of having two layers here?
Explanation: Defense in depth requires layers to be independent failure domains. If both the guardrails-layer and output-layer PII filters use the identical regex logic, an input that evades one will evade the other by construction — there is no additional detection surface. Genuine layering here would mean, for example, a regex heuristic at one layer and a trained NER-based PII classifier at another, so the attacker has to defeat two structurally different detection mechanisms, not the same mechanism twice.
Presenting the architecture in a design review
When you're asked to "draw and defend" this architecture — a standard AI Security Architect and Staff AI Security Engineer interview format — structure the presentation around three things reviewers are actually listening for:
- The request path. Draw the seven layers in the order a request actually flows through them, not alphabetically or by team ownership. A reviewer will immediately ask "what happens when this layer is down" — have an answer for each.
- The failure mode per layer, stated the way it's written above — not "PII filtering" but "PII filtering exists because the model may be prompted or tricked into repeating sensitive data verbatim, and this is the layer that catches it even if the model complies with the malicious request."
- The compensating control. For every layer, name the other layer that catches the same attack class if this one fails. If you can't name one, that's a real gap, and volunteering it is stronger than hoping the interviewer doesn't ask.
This is also the exact structure the validator below produces: not a single pass/fail, but a per-layer score with named missing controls and severity — the artifact you'd actually bring into that review.
During a design review, an interviewer asks: "Your guardrails layer got bypassed by a novel jailbreak technique. What actually stopped the attack from becoming an incident?" Which answer demonstrates a correctly designed layered architecture?
Explanation: A WAF operates on network-level traffic patterns and has no visibility into prompt semantics — it cannot catch a jailbreak. The correct answer identifies that layers downstream of the compromised one (output filtering, content moderation) still had an independent chance to catch the bad output, and the audit layer preserved the evidence needed to investigate regardless. Treating guardrail quality as the only line of defense — the "Nothing — if guardrails are bypassed" answer — is exactly the single-layer thinking this lesson argues against.
Build
Build it
The code artifact below (lesson_08_01_security_architecture.py) implements the reference architecture as data — seven Layer objects, each with typed Controls and a severity — and a SecurityArchitecture validator that scores a system config against it, producing the same per-layer, per-control report you'd bring to a design review. It runs against two example configs: a "prototype-grade" config that skips most layers, and a fully hardened one, so you can see the score and gap list change concretely rather than abstractly.
"""
lesson_08_01_security_architecture.py
A reference model and validator for the seven-layer production AI security
architecture: network -> auth -> input -> guardrails -> model -> output -> audit.
This is not a diagram-as-code toy. It is a config validator you can point at
a real system description (a dict, loaded from YAML/JSON in production) and
get back a specific, per-layer report of what is implemented and what is
missing -- the same shape of review a security architect does in a design
review, just automated and repeatable.
Run:
python lesson_08_01_security_architecture.py
No external services, no network calls, no API keys required.
"""
from __future__ import annotations
import json
from dataclasses import dataclass, field
from enum import Enum
# ---------------------------------------------------------------------------
# Layer definitions
# ---------------------------------------------------------------------------
class Severity(str, Enum):
CRITICAL = "critical" # a missing control here is a direct path to compromise
HIGH = "high"
MEDIUM = "medium"
@dataclass(frozen=True)
class Control:
key: str
description: str
severity: Severity
@dataclass(frozen=True)
class Layer:
name: str
purpose: str
failure_mode: str
controls: list[Control]
# The seven layers every production LLM application should be evaluated
# against, in the order a request actually flows through them.
ARCHITECTURE: list[Layer] = [
Layer(
name="network",
purpose="Stop volumetric and infrastructure-level attacks before they reach the app.",
failure_mode="Unbounded traffic reaches the model endpoint directly, enabling DoS, scraping, and credential stuffing.",
controls=[
Control("waf", "Web Application Firewall with rules for known injection/scanning patterns", Severity.HIGH),
Control("ddos_protection", "DDoS mitigation / rate limiting at the edge", Severity.HIGH),
Control("api_gateway", "API gateway enforcing per-key quotas and request size limits", Severity.CRITICAL),
Control("tls_termination", "TLS 1.2+ termination with modern cipher suites", Severity.CRITICAL),
],
),
Layer(
name="auth",
purpose="Establish and enforce who is calling the system and what they are allowed to do.",
failure_mode="Any caller with a valid-looking token gets full access -- no scoping, no rotation, no service identity.",
controls=[
Control("oauth2", "OAuth 2.0 / OIDC for user-facing auth", Severity.CRITICAL),
Control("api_key_scoping", "API keys scoped to specific operations, not full account access", Severity.HIGH),
Control("mtls_internal", "mTLS between internal services (agent -> tool -> data store)", Severity.HIGH),
Control("key_rotation", "Automated credential rotation policy", Severity.MEDIUM),
],
),
Layer(
name="input",
purpose="Validate and classify everything entering the model before it becomes a prompt.",
failure_mode="Unvalidated user or tool input reaches the model unmodified, enabling injection and malformed-payload attacks.",
controls=[
Control("schema_validation", "Structural validation of all incoming payloads (types, lengths, required fields)", Severity.CRITICAL),
Control("injection_detection", "Heuristic + classifier-based prompt injection detection on user and retrieved content", Severity.CRITICAL),
Control("content_classification", "Topic/intent classification to flag off-policy requests before generation", Severity.MEDIUM),
Control("source_tagging", "Trust-tier tagging that marks retrieved/tool content as untrusted data, not instructions", Severity.HIGH),
],
),
Layer(
name="guardrails",
purpose="Enforce behavioral policy independent of what the model itself decides to do.",
failure_mode="The model is the only thing enforcing policy -- a single successful jailbreak removes all restrictions.",
controls=[
Control("topic_restriction", "Topical rails restricting the conversation to approved domains", Severity.HIGH),
Control("pii_filtering", "PII detection/redaction on both input and output", Severity.CRITICAL),
Control("tool_authorization", "Explicit allowlist + approval gate for tool/function calls", Severity.CRITICAL),
Control("safety_classifier", "Independent safety/toxicity classifier not part of the primary model", Severity.HIGH),
],
),
Layer(
name="model",
purpose="Control which model version runs and detect when its behavior drifts.",
failure_mode="A silent model or prompt-template change ships to production with no diff, no eval, and no rollback path.",
controls=[
Control("version_pinning", "Model version pinned and changes gated behind eval regression", Severity.HIGH),
Control("anomaly_detection", "Statistical anomaly detection on output distributions, latency, and refusal rate", Severity.MEDIUM),
Control("drift_monitoring", "Embedding or output drift monitoring against a labeled baseline", Severity.MEDIUM),
Control("fallback_model", "Defined fallback/degradation path if the primary model is unavailable or flagged", Severity.MEDIUM),
],
),
Layer(
name="output",
purpose="Treat model output as untrusted until validated, since the model can be manipulated into producing it.",
failure_mode="Model output is rendered/executed/stored directly, enabling insecure output handling (XSS, SSRF, SQLi via LLM output).",
controls=[
Control("response_filtering", "Output filtering for secrets, PII, and policy-violating content before it leaves the system", Severity.CRITICAL),
Control("fact_verification", "Verification/grounding check against retrieved sources for factual claims", Severity.MEDIUM),
Control("content_moderation", "Moderation pass on generated content before display or downstream use", Severity.HIGH),
Control("output_encoding", "Context-aware output encoding before rendering in HTML/SQL/shell contexts", Severity.CRITICAL),
],
),
Layer(
name="audit",
purpose="Make every decision the system made reconstructable after the fact.",
failure_mode="An incident happens and there is no record of the prompt, the retrieved context, the tool calls, or the model version that produced the bad output.",
controls=[
Control("full_logging", "Full request/response/tool-call logging with correlation IDs", Severity.CRITICAL),
Control("retention_policy", "Defined log retention policy meeting compliance requirements", Severity.MEDIUM),
Control("compliance_reporting", "Automated reporting mapped to a framework (NIST AI RMF, SOC 2, EU AI Act)", Severity.MEDIUM),
Control("tamper_evidence", "Write-once or hash-chained audit logs resistant to post-hoc tampering", Severity.HIGH),
],
),
]
_SEVERITY_WEIGHT = {Severity.CRITICAL: 3, Severity.HIGH: 2, Severity.MEDIUM: 1}
# ---------------------------------------------------------------------------
# Validator
# ---------------------------------------------------------------------------
@dataclass
class LayerResult:
layer: str
implemented: list[str]
missing: list[Control]
score: float # 0-100, weighted by control severity
@dataclass
class ArchitectureReport:
layer_results: list[LayerResult]
overall_score: float
critical_gaps: list[str] = field(default_factory=list)
def to_markdown(self) -> str:
lines = ["# AI Security Architecture Review", ""]
lines.append(f"**Overall score: {self.overall_score:.1f}/100**")
lines.append("")
for r in self.layer_results:
status = "PASS" if r.score >= 80 else ("WARN" if r.score >= 50 else "FAIL")
lines.append(f"## {r.layer.upper()} — {r.score:.0f}/100 [{status}]")
if r.implemented:
lines.append(f"- Implemented: {', '.join(r.implemented)}")
if r.missing:
for c in r.missing:
lines.append(f"- MISSING ({c.severity.value}): `{c.key}` — {c.description}")
lines.append("")
if self.critical_gaps:
lines.append("## Critical gaps requiring immediate remediation")
for gap in self.critical_gaps:
lines.append(f"- {gap}")
return "\n".join(lines)
class SecurityArchitecture:
"""Validates a system config dict against the 7-layer reference architecture."""
def __init__(self, layers: list[Layer] | None = None) -> None:
self.layers = layers or ARCHITECTURE
def validate(self, system_config: dict[str, list[str]]) -> ArchitectureReport:
"""
system_config: { layer_name: [implemented_control_keys] }
Missing layers in the dict are treated as fully unimplemented.
"""
layer_results: list[LayerResult] = []
critical_gaps: list[str] = []
total_weight = 0.0
total_earned = 0.0
for layer in self.layers:
implemented_keys = set(system_config.get(layer.name, []))
implemented = [c.key for c in layer.controls if c.key in implemented_keys]
missing = [c for c in layer.controls if c.key not in implemented_keys]
layer_weight = sum(_SEVERITY_WEIGHT[c.severity] for c in layer.controls)
layer_earned = sum(_SEVERITY_WEIGHT[c.severity] for c in layer.controls if c.key in implemented_keys)
score = (layer_earned / layer_weight * 100) if layer_weight else 100.0
for c in missing:
if c.severity == Severity.CRITICAL:
critical_gaps.append(f"[{layer.name}] {c.key}: {c.description}")
layer_results.append(LayerResult(layer=layer.name, implemented=implemented, missing=missing, score=score))
total_weight += layer_weight
total_earned += layer_earned
overall = (total_earned / total_weight * 100) if total_weight else 100.0
return ArchitectureReport(layer_results=layer_results, overall_score=overall, critical_gaps=critical_gaps)
# ---------------------------------------------------------------------------
# Demo
# ---------------------------------------------------------------------------
if __name__ == "__main__":
# A realistic "we shipped the model layer but skipped the rest" config --
# common in teams that started with a prototype and never revisited security.
naive_config = {
"network": ["tls_termination"],
"auth": ["api_key_scoping"],
"input": [],
"guardrails": ["pii_filtering"],
"model": ["version_pinning"],
"output": [],
"audit": ["full_logging"],
}
# A hardened config with controls at every layer.
hardened_config = {
"network": ["waf", "ddos_protection", "api_gateway", "tls_termination"],
"auth": ["oauth2", "api_key_scoping", "mtls_internal", "key_rotation"],
"input": ["schema_validation", "injection_detection", "content_classification", "source_tagging"],
"guardrails": ["topic_restriction", "pii_filtering", "tool_authorization", "safety_classifier"],
"model": ["version_pinning", "anomaly_detection", "drift_monitoring", "fallback_model"],
"output": ["response_filtering", "fact_verification", "content_moderation", "output_encoding"],
"audit": ["full_logging", "retention_policy", "compliance_reporting", "tamper_evidence"],
}
validator = SecurityArchitecture()
print("=" * 70)
print("REVIEW 1: naive production config (prototype-grade security)")
print("=" * 70)
report = validator.validate(naive_config)
print(report.to_markdown())
print("\n" + "=" * 70)
print("REVIEW 2: hardened production config")
print("=" * 70)
report2 = validator.validate(hardened_config)
print(report2.to_markdown())
print("\n" + "=" * 70)
print("Machine-readable summary (for CI gate consumption, see lesson 3)")
print("=" * 70)
summary = {
"naive_score": round(report.overall_score, 1),
"hardened_score": round(report2.overall_score, 1),
"naive_critical_gaps": len(report.critical_gaps),
"hardened_critical_gaps": len(report2.critical_gaps),
}
print(json.dumps(summary, indent=2))