Phase 8: Production Agent Infrastructure · 60 min · Python · Anthropic SDK

Cost Optimization & Token Budgets

Cost optimization is engineering, not penny-pinching.

Hiring signal: Cost efficiency is a business requirement for every AI product. Engineers who proactively instrument costs, implement caching, and design tiering systems are assets at any company that cares about unit economics (which is all of them). 'How would you reduce our AI API costs by 50%?' is a real interview question.

What you will learn

Implement prompt caching to reduce repeated context costs by up to 90%
Design model tiering: route simple tasks to cheaper models, complex tasks to frontier models
Build a token budget manager with per-agent, per-user, and per-session limits

The Problem

AI API costs scale with every token processed. An agent that works beautifully in testing can become financially unsustainable at scale. Three engineering decisions determine whether your AI product is profitable: prompt caching, model tiering, and budget enforcement.

These aren't micro-optimizations — the right combination can reduce costs by 50–80% with no loss in quality.

Prompt Caching

Anthropic's prompt caching lets you mark portions of your prompt with cache_control: {"type": "ephemeral"}. When the same cached prefix appears in a subsequent request within the 5-minute TTL, Anthropic charges only 10% of the normal input token cost for those tokens — a 90% discount.

The key rule: stable content must come first. Anthropic caches a prefix of your message — everything up to the first position that changes between requests. Structure your messages to maximize the cached prefix:

[System prompt - 5,000 tokens - CACHE THIS]
[Retrieved documents - 40,000 tokens - CACHE THIS]
[Conversation history - 2,000 tokens - CACHE THIS]
[Current user message - 50 tokens - changes each request]

For a system with a 50,000-token knowledge base injected into every request, caching can save 90% on those 50,000 tokens for any request within the TTL. At $3/MTok for input tokens, that's $0.135 saved per request on the cached portion alone.

You have a 50,000-token system prompt (a large knowledge base). How should you structure messages to maximize cache hits?

Anthropic's prompt cache works positionally: it caches a prefix of the message up to the first position that changes. Put your large, stable content (system prompt, knowledge base, retrieved docs) at the very beginning. Dynamic content (user message, this-session context) goes at the end. This maximizes the cached prefix, giving the maximum 90% discount on stable tokens.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Model Tiering, Token Budget Enforcement, Build It, Common Pitfalls, Key Terms, Interview Framing, What to Practice — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.