Phase 4: LLM Orchestration for Voice · 45 min · Python · OpenAI SDK · Anthropic SDK

Model Selection for Voice Agents

First-token latency is the metric that matters for voice. Choose your LLM accordingly.

Hiring signal: LLM model selection for voice agents — balancing latency, cost, and quality — is tested in voice AI system design interviews.

What you will learn

Select voice-appropriate LLMs based on first-token latency, cost, and conversation complexity
Compare GPT-4o-mini, Claude 3.5 Haiku, Llama 3.1 on Groq, and Gemini 2.0 Flash for voice
Implement model routing: cheap model for simple queries, expensive for complex
Measure and compare first-token latency across providers

The Problem

Voice agents have unique LLM requirements that text chatbots don't:

Latency budget: Only 150-250ms for LLM (out of 500ms total)
Streaming required: Must stream tokens for TTS pipeline
Short responses: Voice responses must be concise (2-3 sentences)
Tool calling: Must call APIs mid-conversation (check balance, book flight)
Cost sensitivity: Per-call cost matters at scale

The Concept

LLM Requirements for Voice

Requirement	Why It Matters	Target
TTFT (Time to First Token)	Caller waits in silence	<200ms
Streaming	Pipe tokens to TTS	Must support
Response length	Voice = short answers	System prompt controls
Tool calling	Check systems mid-call	Must support
Context window	Conversation history	8K-128K tokens
Cost per call	Scale economics	<$0.01 per call

Model Comparison for Voice

Model	TTFT	Quality	Cost/1M tokens	Tool Calling	Streaming	Best For
GPT-4o	~200ms	Excellent	$2.50 in / $10 out	Yes	Yes	Premium agents
GPT-4o-mini	~150ms	Good	$0.15 in / $0.60 out	Yes	Yes	Standard agents
Claude 3.5 Sonnet	~300ms	Excellent	$3.00 in / $15 out	Yes	Yes	Complex reasoning
Claude 3 Haiku	~150ms	Good	$0.25 in / $1.25 out	Yes	Yes	Fast, cheap
Gemini 2.0 Flash	~100ms	Good	$0.075 in / $0.30 out	Yes	Yes	Lowest latency
Gemini 2.0 Pro	~250ms	Excellent	$1.25 in / $5.00 out	Yes	Yes	Multimodal
Llama 3.1 8B (self-hosted)	~50ms	Fair	$0 (GPU)	Yes	Yes	Ultra-low latency, privacy
DeepSeek V3	~200ms	Very Good	$0.14 in / $0.28 out	Yes	Yes	Cost-effective

Why is GPT-4o-mini often better than GPT-4o for voice agents?

It supports more languages

The Latency Budget Breakdown

Component	Budget	GPT-4o	GPT-4o-mini	Gemini Flash
ASR	150ms	150ms	150ms	150ms
LLM TTFT	200ms	200ms	150ms	100ms
TTS TTFB	100ms	120ms	120ms	120ms
Network	50ms	50ms	50ms	50ms
Total	500ms	520ms	470ms	420ms

Model Selection Decision Flow

Cost Analysis at Scale

Volume	GPT-4o ($2.50/$10)	GPT-4o-mini ($0.15/$0.60)	Gemini Flash ($0.075/$0.30)
1K calls	$6.25	$0.38	$0.19
10K calls	$62.50	$3.75	$1.88
100K calls	$625	$37.50	$18.75
1M calls	$6,250	$375	$187.50

Assumes 500 input tokens, 100 output tokens per call

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Evaluation, Key Terms, Common Pitfalls, Interview Framing — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.