Phase 0: Voice AI Fundamentals & Architecture · 40 min · Python
The Voice AI Landscape: Providers, Platforms, and Where the Field is Heading
Know the ecosystem cold. Your provider choices determine your latency ceiling and cost floor.
Real-world application: Provider landscape knowledge shows you've done your homework and can make informed architecture decisions.
What you will learn
- Map the full voice AI ecosystem: ASR, TTS, LLM, frameworks, managed platforms, telephony
- Identify the key providers in each layer and their differentiators
- Understand where the field is heading: native audio, emotional AI, real-time translation, multimodal
- Make informed provider selection decisions based on use case requirements
The Problem
The voice AI ecosystem has dozens of providers across five layers: ASR, TTS, LLM, frameworks, and telephony. Knowing which provider to use for each layer — and why — is what separates engineers who can architect voice agents from those who just follow tutorials.
In interviews, you'll be asked: "Why Deepgram over Whisper?" or "When would you use Vapi instead of Pipecat?" Vague answers fail. You need specific, numbers-backed reasoning.
The Concept
The Five-Layer Voice AI Stack
Layer 1: ASR (Speech-to-Text)
| Provider | WER | Latency | Price | Languages | Best For |
|---|
| Deepgram Nova-3 | ~5% | ~150ms TTFT | $0.0077/min | 40+ | Best streaming ASR for production |
| AssemblyAI Universal-2 | ~7% | ~300ms TTFT | $0.012/min | 20+ | Diarization, batch + streaming |
| Whisper (OpenAI) | ~10% | ~500ms+ | $0.006/min | 99 | Multilingual, self-hosted, batch |
| Soniox | ~8% | ~150ms TTFT | $0.0098/min | All | Ultra-low latency, speaker ID |
Hiring signal: Knowing WER (Word Error Rate) and TTFT (Time to First Token) for each provider shows production experience.
Layer 2: TTS (Text-to-Speech)
| Provider | MOS | TTFB | Price | Languages | Best For |
|---|
| ElevenLabs | ~4.5/5 | ~400ms | $0.30/min | 29 | Best quality, voice cloning |
| Cartesia Sonic | ~4.2/5 | ~120ms | $0.015/min | 15+ | Lowest TTFB, cost-effective |
| Deepgram Aura | ~4.0/5 | ~200ms | $0.025/min | 10+ | Integrated with Deepgram ASR |
| PlayHT | ~4.1/5 | ~250ms | $0.020/min | 25+ | Voice cloning, multilingual |
Hiring signal: MOS (Mean Opinion Score) and TTFB (Time to First Byte) are the key TTS metrics. Knowing Cartesia's 120ms TTFB vs ElevenLabs' 400ms shows you understand latency optimization.
Layer 3: LLM (Reasoning)
| Provider | Latency | Price | Best For |
|---|
| GPT-4o | ~300-500ms TTFT | $2.50/$10.00/M tokens | Best quality, multimodal |
| GPT-4o-mini | ~200-300ms TTFT | $0.15/$0.60/M tokens | Fast, cheap, transactional |
| Claude 3.5 Haiku | ~200-400ms TTFT | $0.25/$1.25/M tokens | Complex conversations |
| Groq Llama 3.1 8B | ~50-100ms TTFT | $0.05/$0.08/M tokens | Ultra-fast, lowest latency |
Why would you choose Groq Llama 3.1 8B over GPT-4o for a voice agent?
Groq Llama 3.1 8B on Groq's LPU hardware achieves ~50ms TTFT vs GPT-4o's ~400ms. For a voice agent with a 500ms latency budget, saving 350ms on the LLM is the difference between meeting and missing the target. Cost: $0.05/$0.08 per M tokens vs $2.50/$10.00 — 50x cheaper. Trade-off: Llama 3.1 8B has lower reasoning capability. Use it for transactional agents (booking, FAQ, reminders) where the conversation is predictable. Use GPT-4o for complex support agents where reasoning quality matters more than latency.
Layer 4: Frameworks
| Framework | Type | Architecture | Best For |
|---|
| Pipecat | Open-source | Chained pipeline | Full control, BYO providers |
| LiveKit Agents | Open-source | Chained pipeline | WebRTC-native, multi-participant |
| OpenAI Realtime API | Managed | Native audio | Lowest latency, natural prosody |
| Vapi | Managed platform | Chained + native | Fastest time-to-production, Squads |
| Retell AI | Managed platform | Chained pipeline | Inbound support, analytics |
| Bland AI | Managed platform | Chained pipeline | Outbound campaigns at scale |
Layer 5: Telephony
| Provider | Price/min | SIP | Best For |
|---|
| Twilio | $0.01 | Yes | Best docs, Media Streams, US-centric |
| Telnyx | $0.003 | Yes (native) | Cheapest, SIP-native, international |
| Vonage | $0.009 | Yes | Global reach, API-first |
| AWS Connect | $0.018 | Yes | AWS ecosystem, enterprise |
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers Build, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.
Create a free account — the opening phases of 29 of 35 courses are free, no credit card.
Browse all courses · View pricing · DeVenture Academy