Phase 0: Voice AI Fundamentals & Architecture · 45 min · Python

The Latency Budget and Cost Economics

Every millisecond is a budget line item. Every minute is a cost center. Know both cold.

Hiring signal: Latency budget reasoning and cost calculation appear in every voice AI system design interview.

What you will learn

Reason about the latency budget: 300ms target, 500ms acceptable, 800ms breaks flow
Break down latency by component: ASR, LLM, TTS, transport
Calculate per-minute costs for each component and total pipeline cost
Compare voice agent cost ($0.10–$0.20/min) to human agent cost ($7–$12/call)

The Problem

A voice agent that takes 1.2 seconds to respond feels broken. Callers hang up. A voice agent that costs $0.40/minute is unprofitable for high-volume use cases. You need to know both the latency budget and the cost economics cold — they determine whether your voice agent ships.

Every voice AI interview includes a system design question that boils down to: "How do you keep this under 500ms and under $0.10/minute?"

The Concept

The Latency Budget

Human conversation has a natural rhythm. When someone asks a question, they expect a response within:

< 300ms: Feels instantaneous, like talking to a fast human
300-500ms: Feels natural, slight pause but conversational
500-800ms: Noticeable delay, but tolerable for most use cases
800-1200ms: Feels slow, callers may think the call dropped
> 1200ms: Feels broken, callers hang up

Your budget is 500ms glass-to-glass — from the moment the user stops speaking to the moment they hear the agent's response.

Breaking Down the Budget

Component	Budget	Typical	Optimized
ASR (TTFT)	200ms	250ms	150ms (Deepgram)
LLM (TTFT)	200ms	400ms	50ms (Groq)
TTS (TTFB)	100ms	250ms	120ms (Cartesia)
Transport	50ms	80ms	50ms (WebRTC)
Total	550ms	980ms	370ms

Your voice agent has 980ms glass-to-glass latency. Which component is the bottleneck and how do you fix it?

TTS is the bottleneck — switch to Cartesia

The Cost Economics

Every minute of a voice agent call costs money. Here's the breakdown:

Component	Provider	Cost/min	Notes
ASR	Deepgram Nova-3	$0.0077	Streaming, best accuracy
ASR	AssemblyAI	$0.012	Good for batch + diarization
LLM	GPT-4o	$0.02-$0.05	Per call, depends on tokens
LLM	GPT-4o-mini	$0.001-$0.005	Much cheaper, good enough for simple agents
LLM	Groq Llama 3.1	$0.001-$0.005	Ultra-fast + cheap
TTS	ElevenLabs	$0.30	Best quality, premium price
TTS	Cartesia Sonic	$0.015	Good quality, lowest cost
TTS	Deepgram Aura	$0.025	Integrated with Deepgram ASR
Telephony	Twilio	$0.01	+ $1.15/month per number
Telephony	Telnyx	$0.003	Cheapest, SIP-native

Cost Configurations

Budget configuration (~$0.03/min):

Deepgram ($0.008) + Groq Llama 3.1 ($0.003) + Cartesia ($0.015) + Telnyx ($0.003) = $0.029/min

Balanced configuration (~$0.07/min):

Deepgram ($0.008) + GPT-4o-mini ($0.005) + Cartesia ($0.015) + Twilio ($0.01) = $0.038/min

Premium configuration (~$0.36/min):

Deepgram ($0.008) + GPT-4o ($0.04) + ElevenLabs ($0.30) + Twilio ($0.01) = $0.358/min

Voice Agent vs Human Agent Cost

Metric	Voice Agent	Human Agent
Cost per minute	$0.03-$0.36	$0.50-$1.50 (loaded)
Cost per 5-min call	$0.15-$1.80	$2.50-$7.50
Monthly cost (10K calls)	$1,500-$18,000	$25,000-$75,000
Availability	24/7	Business hours
Scalability	Unlimited	Linear with headcount
Break-even	~500 calls/month	N/A

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Evaluation, Key Terms, Common Pitfalls, Interview Framing — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.