The Concept
Token Growth in Voice Conversations
| Call Duration | Approx Tokens | Provider Cost Impact |
|---|
| 1 minute | ~400 tokens | Minimal |
| 5 minutes | ~2,000 tokens | Noticeable |
| 10 minutes | ~4,000 tokens | Significant |
| 30 minutes | ~12,000 tokens | Expensive |
| 1 hour | ~24,000 tokens | Very expensive |
Assumes ~40 words/min conversation, ~1.3 tokens/word
Context Management Strategies
| Strategy | How | When to Use | Trade-off |
|---|
| Full history | Send all messages | Short calls (<5 min) | Simple, but costly |
| Sliding window | Keep last N turns | Medium calls (5-15 min) | Loses early context |
| Summarization | Summarize old turns | Long calls (>15 min) | Extra LLM call |
| RAG retrieval | Retrieve relevant history | Multi-session | Needs vector DB |
| Hybrid | Summary + recent turns | Best of both | More complex |
A voice agent has been on a call for 20 minutes and the context is now 8,000 tokens. The LLM is getting slower and more expensive. What do you do?
Switch to a cheaper model
Hybrid Context Management Flow
Sliding Window Implementation
def sliding_window(messages, max_turns=10):
"""Keep only the last N turns of conversation."""
system = [m for m in messages if m["role"] == "system"]
conversation = [m for m in messages if m["role"] != "system"]
# Keep last max_turns messages
recent = conversation[-max_turns * 2:] # *2 for user+assistant pairs
return system + recent
Summarization Strategy
async def summarize_context(messages, model="gpt-4o-mini"):
"""Summarize old conversation turns."""
conversation_text = "\n".join([
f"{m['role']}: {m['content']}" for m in messages
])
summary_prompt = f"""Summarize this conversation in 2-3 sentences.
Include: customer name, what they want, what's been done so far, any pending items.
Conversation:
{conversation_text}
Summary:"""
response = await client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": summary_prompt}],
max_tokens=100,
)
return response.choices[0].message.content
Hybrid: Summary + Recent Turns
async def hybrid_context(messages, summarize_threshold=5000, keep_recent_turns=5):
"""Summary of old turns + full recent turns."""
# Estimate token count
total_tokens = sum(len(m["content"].split()) * 1.3 for m in messages)
if total_tokens < summarize_threshold:
return messages # No need to summarize yet
system = [m for m in messages if m["role"] == "system"]
conversation = [m for m in messages if m["role"] != "system"]
# Split into old (summarize) and recent (keep full)
split_point = len(conversation) - keep_recent_turns * 2
old_turns = conversation[:split_point]
recent_turns = conversation[split_point:]
# Summarize old turns
summary = await summarize_context(old_turns)
# Build new context: system + summary + recent
return system + [
{"role": "system", "content": f"Previous conversation summary: {summary}"},
*recent_turns,
]
Token Estimation
def estimate_tokens(text):
"""Rough token estimate: ~1.3 tokens per word."""
return int(len(text.split()) * 1.3)
def estimate_context_tokens(messages):
"""Estimate total tokens in message list."""
return sum(estimate_tokens(m["content"]) for m in messages)
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Evaluation, Key Terms, Common Pitfalls, Interview Framing — plus a hands-on lab, quiz, and project artifact.
Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.