The Problem
A long conversation hits the context limit. The agent truncates everything before the last 10 messages. The user refers to a decision made in message 3. The agent has no memory of it — and either hallucinates an answer or says it doesn't know.
This is the context management problem: every model has a finite context window (200k tokens for Claude Sonnet), but conversations, tool results, and retrieved memories can easily exceed it. Naive truncation loses critical early context. The alternative — keeping everything — runs out of tokens and gets expensive fast.
Context management is the art of deciding what stays, what gets compressed, and what gets evicted. It's not a frontend concern. It's a core engineering decision that determines whether your agent degrades after 20 turns or remains coherent after 200.
The three strategies covered here — sliding window, summarization, and selective retention — have different tradeoffs. Choosing the right one for your use case is the skill.
Context window components
A context window is not just conversation messages. It contains: system prompt (~5% of budget), retrieved memory (~20%), conversation history (~40%), tool definitions (~15%), tool results (~15%), and the response buffer (~5%). A budget manager tracks all of these and triggers eviction before any single component overflows.
Context window components and budgets
For a 200k-token context window, the typical allocation is:
| Component | Budget | Notes |
|---|
| System prompt | 10,000 tokens | Stable — set at deploy time |
| Retrieved memory | 40,000 tokens | Episodic + semantic memories injected at session start |
| Conversation history | 80,000 tokens | The part most likely to overflow |
| Tool definitions | 30,000 tokens | Scales with number of tools |
| Tool results | 30,000 tokens | Can spike for large API responses |
| Response buffer | 10,000 tokens | Reserved for the model's output |
A ContextBudgetManager tracks current usage per component. When total_fill_pct() reaches 70%, it signals that eviction should run — before the model throws a context limit error. The fill threshold of 70% (not 100%) gives the system headroom; you never want to discover the limit mid-generation.
A context window is 80% full mid-conversation. You use sliding window (keep last 20 messages). A user then asks about something said in message 5. What happens?
Explanation: Sliding window is the simplest strategy but the most lossy. Early context is permanently lost once the window moves past it. For conversations where early context matters — project discussions, onboarding flows, decision trails — summarization or selective retention is better. Sliding window is appropriate only when only recent context matters, like a general chatbot where nothing from 30+ messages ago is relevant.
Three eviction strategies
Strategy 1: Sliding Window
Keep only the last N messages. Anything older is dropped. Implementation is a single list[-N:] slice.
Strengths: zero extra API calls, predictable memory usage, minimal complexity. Weaknesses: hard memory loss — early context is permanently gone. If a user refers to message 5 in a 50-message conversation, the agent can't answer.
Best for: casual chat applications, short-lived tasks where past context genuinely doesn't matter.
Strategy 2: Summarization Compression
When conversation history exceeds a fill threshold (e.g., 70%), take the oldest 50% of messages and compress them into a summary block using an LLM call. The raw messages are evicted; the summary block persists.
The summarization prompt extracts what matters: key decisions, established facts, open questions, user preferences. The result is a compact block of ~200–400 tokens that captures the substance of potentially thousands of tokens of conversation.
When the window fills again, another compression run merges the existing summary with newer messages into a single updated summary. The conversation can continue indefinitely.
Cost: one extra LLM call per compression (typically every 30–50 messages). At claude-haiku-4-5 pricing, this is negligible compared to the main conversation cost.
Best for: long-running project assistants, support conversations, onboarding agents where early context (decisions, facts, preferences) must survive.
Strategy 3: Selective Retention
Score each message by importance on a 1–5 scale, then when the window is full, retain only the top-K highest-scoring messages. Low-importance messages (acknowledgments, filler, short pleasantries) are evicted first; high-importance messages (decisions, stated preferences, confirmed facts) are retained regardless of age.
Scoring heuristics: presence of keywords like "decided", "prefer", "always", "important", "confirmed" boosts importance. Pure filler messages ("ok", "thanks", "got it") are scored at 1.0 and evicted first. User messages get a slight boost over assistant messages.
Best for: contexts where decisions and stated preferences are sparse and critical, but most conversation is low-value back-and-forth.
You're building a long-running project assistant. The best context management strategy is:
Explanation: A project assistant needs long-term coherence: "Remember we decided to use PostgreSQL in session 1." Sliding window loses this once the conversation grows past the window. Summarization captures "team decided on PostgreSQL" in a compressed form that persists indefinitely. The summary costs far fewer tokens than the raw messages that produced the decision — and the agent can answer questions about early decisions from session 10, session 50, and beyond.
Build
Build it
Here's a minimal sliding-window eviction policy: messages are added with a token cost, and the oldest ones are dropped once the running total exceeds a budget.
import random
def token_cost(text):
return max(1, len(text) // 4)
def evict_to_budget(messages, budget):
"""Drop oldest messages until total token cost fits the budget."""
total = sum(token_cost(m) for m in messages)
evicted = 0
while total > budget and len(messages) > 1:
oldest = messages.pop(0)
total -= token_cost(oldest)
evicted += 1
return messages, total, evicted
conversation = [
"System: You are a helpful coding assistant.",
"User: I'm building a FastAPI service for healthcare data.",
"Assistant: Great, what does the service need to do?",
"User: A REST API for patient records with JWT auth.",
"Assistant: Here's a starting structure for that service...",
"User: We decided to use PostgreSQL with pgvector.",
"Assistant: Good choice for combining relational and vector data.",
"User: Can you show me an index example?",
]
budget = 40 # small budget to force eviction in this demo
kept, total_tokens, evicted_count = evict_to_budget(list(conversation), budget)
print(f"Original messages: {len(conversation)}")
print(f"Kept messages: {len(kept)} (evicted {evicted_count})")
print(f"Token budget: {budget}, final usage: {total_tokens}")
print("Surviving context:")
for m in kept:
print(f" - {m}")
The code artifact lesson_07_02_context_management.py implements all three strategies with a common interface:
ContextBudgetManager — tracks token usage per component (system prompt, retrieved memory, conversation history, tool definitions, tool results, response buffer) and signals when eviction is neededSlidingWindowManager — keeps the last N messages, reports eviction countSummarizationManager — triggers compression at a configurable fill threshold, makes an LLM call to extract key decisions/facts, replaces raw messages with summariesSelectiveRetentionManager — scores each message by importance keywords, evicts low-importance messages first when the window is full
The demo runs all three strategies over a 40-message simulated conversation and compares what each retains: specifically, whether early decisions (e.g., "We decided to use PostgreSQL") survive to the end of the conversation. Sliding window fails this test; summarization and selective retention pass it.