Phase 7: Agent Memory Systems · 60 min · Python · Anthropic SDK

In-Context Memory Management

Fitting the right information into a limited window — and discarding the rest intelligently — is the difference between an agent that degrades and one that scales.

Hiring signal: Every production AI assistant hits context limits. Engineers who design proactive context management — with clear budgets and graceful compression — rather than letting context overflow and crash, demonstrate production engineering experience that stands out in AI infrastructure interviews.

What you will learn

Implement sliding window, summarization, and selective retention strategies for context management
Build a context budget manager that tracks token usage across all context components
Design a context compression strategy for long conversations using LLM-based summarization

The Problem

A long conversation hits the context limit. The agent truncates everything before the last 10 messages. The user refers to a decision made in message 3. The agent has no memory of it — and either hallucinates an answer or says it doesn't know.

This is the context management problem: every model has a finite context window (200k tokens for Claude 3.5 Sonnet), but conversations, tool results, and retrieved memories can easily exceed it. Naive truncation loses critical early context. The alternative — keeping everything — runs out of tokens and gets expensive fast.

Context management is the art of deciding what stays, what gets compressed, and what gets evicted. It's not a frontend concern. It's a core engineering decision that determines whether your agent degrades after 20 turns or remains coherent after 200.

The three strategies covered here — sliding window, summarization, and selective retention — have different tradeoffs. Choosing the right one for your use case is the skill.

Context window components
A context window is not just conversation messages. It contains: system prompt (~5% of budget), retrieved memory (~20%), conversation history (~40%), tool definitions (~15%), tool results (~15%), and the response buffer (~5%). A budget manager tracks all of these and triggers eviction before any single component overflows.

Context Window Components and Budgets

For a 200k-token context window, the typical allocation is:

Component	Budget	Notes
System prompt	10,000 tokens	Stable — set at deploy time
Retrieved memory	40,000 tokens	Episodic + semantic memories injected at session start
Conversation history	80,000 tokens	The part most likely to overflow
Tool definitions	30,000 tokens	Scales with number of tools
Tool results	30,000 tokens	Can spike for large API responses
Response buffer	10,000 tokens	Reserved for the model's output

A ContextBudgetManager tracks current usage per component. When total_fill_pct() reaches 70%, it signals that eviction should run — before the model throws a context limit error. The fill threshold of 70% (not 100%) gives the system headroom; you never want to discover the limit mid-generation.

A context window is 80% full mid-conversation. You use sliding window (keep last 20 messages). A user then asks about something said in message 5. What happens?

Explanation: Sliding window is the simplest strategy but the most lossy. Early context is permanently lost once the window moves past it. For conversations where early context matters — project discussions, onboarding flows, decision trails — summarization or selective retention is better. Sliding window is appropriate only when only recent context matters, like a general chatbot where nothing from 30+ messages ago is relevant.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Three Eviction Strategies, Build It, What to Practice — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.