Phase 8: Production AI Security Architecture · 70 min · LangSmith · Langfuse · Arize Phoenix

Security Monitoring & Observability

An attack you can't see in your dashboard is an incident you'll read about in a postmortem.

Hiring signal: DevSecOps for AI Pipelines and AI Security Engineer roles are increasingly evaluated on whether they can turn LLM observability tooling (LangSmith, Langfuse, Phoenix, Helicone) into actual security signal, not just latency/cost dashboards. Interviewers ask candidates to define what 'anomalous' means for token cost, injection attempts, and tool-call sequences in statistical terms. This lesson builds those three detectors from first principles so you understand what the observability platforms are computing under the hood, and can defend the thresholds you set.

What you will learn

Explain what LangSmith, Langfuse, Phoenix, and Helicone each instrument by default and what security-relevant signal you have to derive yourself
Compute a rolling z-score anomaly detector on token cost time series to catch cost-spike attacks (context flooding, token exhaustion DoS)
Compute a rolling injection-attempt rate and alert threshold from flagged-request telemetry
Build a Markov-chain likelihood model over tool-call sequences to detect anomalous agent behavior that individual tool-call checks would miss

The Problem

LLM observability platforms — LangSmith, Langfuse, Arize Phoenix, Helicone — solved a real problem: you can now see every prompt, every completion, every token cost, and every tool call your production system makes. Teams adopt one of these tools, wire up tracing, and feel like they have security visibility.

They don't, not yet. These platforms give you the raw traces. They do not ship a default answer to "is this pattern of token cost anomalous," "is this rate of flagged requests an attack or noise," or "is this tool-call sequence something an attacker would produce." That's a modeling problem you have to solve yourself, on top of the tracing layer — and it's exactly the kind of problem AI Security Engineer and DevSecOps-for-AI interviews probe, because it separates people who've wired up a dashboard from people who've operationalized one.

Three signals matter most for security, specifically because each one catches an attack class that guardrails (Phase 5) and per-request input filtering (Phase 8, Lesson 1) structurally cannot:

Token cost — a context-flooding or token-exhaustion DoS attack (OWASP LLM04) doesn't look like a bad prompt; it looks like a normal-looking request that costs 10x what your average request costs. No per-request filter catches "this is expensive" without a baseline to compare against.
Injection attempt rate — a single flagged prompt injection attempt might be noise (an overzealous classifier, a confused user). A burst of them is a campaign. The difference is only visible in a rolling rate, not a per-request classification.
Tool-call sequence — Phase 4 covered agent privilege escalation and tool poisoning: attacks that chain individually-authorized tool calls into something dangerous. A send_email call and a delete_record call might each pass tool authorization independently. The sequence read_document -> send_email -> delete_record is what should trip an alarm, and no single-call check will ever see it.

Rolling Z-Score: Catching Cost Spikes Before They're an Incident

A z-score measures how many standard deviations a point is from a rolling mean. For token cost, this is the right primitive because "normal" cost varies by traffic pattern and time of day — an absolute dollar threshold either fires constantly or misses slow-building attacks.

def cost_anomalies(self, costs: list[float], window: int = 20) -> list[Alert]:
    for i in range(window, len(costs)):
        hist = costs[i - window : i]
        mean = sum(hist) / window
        std = (sum((x - mean) ** 2 for x in hist) / window) ** 0.5 or 1e-9
        z = (costs[i] - mean) / std
        if z >= self.cost_z_threshold:
            ...  # alert

The threshold you pick (this lesson uses 3.0, i.e. 3 standard deviations) is a real tradeoff, not an arbitrary constant. Set it too low and normal traffic variance triggers alert fatigue — the demo run below shows exactly this, with two borderline 3.4-3.5 sigma events that are plausibly just noise alongside one unambiguous 67-sigma spike that is not. Set it too high and you'll miss a genuine attack that ramps cost gradually to stay under a wide threshold. In a real system, you validate the threshold against a labeled window of known-normal traffic and tune it until the false-positive rate is tolerable for whoever is on call.

Why does this lesson's cost-anomaly detector use a rolling z-score against a sliding window instead of a fixed absolute dollar threshold (e.g., "alert if any request costs more than $0.50")?

The lesson makes this explicit: "'normal' cost varies by traffic pattern and time of day — an absolute dollar threshold either fires constantly or misses slow-building attacks." A rolling z-score recalculates its own baseline (mean and standard deviation) from the preceding window at every point, so it flags a request that is unusual relative to recent behavior, not relative to a number picked once and never revisited. This is what lets it catch a sudden 67-sigma context-flooding spike and still stay quiet during a legitimate traffic surge that raises everyone's cost proportionally.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Rolling Rate: Catching a Campaign, Not a Single Flag, Markov-Chain Likelihood: Catching Sequences, Not Calls, What LangSmith, Langfuse, Phoenix, and Helicone Actually Give You, Build It, What to Practice — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.