Phase 0: Voice AI Fundamentals & Architecture · 40 min · Python
The Voice AI Landscape: Providers, Platforms, and Where the Field is Heading
Know the ecosystem cold. Your provider choices determine your latency ceiling and cost floor.
Hiring signal: Provider landscape knowledge shows you've done your homework and can make informed architecture decisions.
What you will learn
- Map the full voice AI ecosystem: ASR, TTS, LLM, frameworks, managed platforms, telephony
- Identify the key providers in each layer and their differentiators
- Understand where the field is heading: native audio, emotional AI, real-time translation, multimodal
- Make informed provider selection decisions based on use case requirements
The Problem
The voice AI ecosystem has dozens of providers across five layers: ASR, TTS, LLM, frameworks, and telephony. Knowing which provider to use for each layer — and why — is what separates engineers who can architect voice agents from those who just follow tutorials.
In interviews, you'll be asked: "Why Deepgram over Whisper?" or "When would you use Vapi instead of Pipecat?" Vague answers fail. You need specific, numbers-backed reasoning.
The Concept
The Five-Layer Voice AI Stack
Layer 1: ASR (Speech-to-Text)
| Provider | WER | Latency | Price | Languages | Best For |
|---|
| Deepgram Nova-3 | ~5% | ~150ms TTFT | $0.0077/min | 40+ | Best streaming ASR for production |
| AssemblyAI Universal-2 | ~7% | ~300ms TTFT | $0.012/min | 20+ | Diarization, batch + streaming |
| Whisper (OpenAI) | ~10% | ~500ms+ | $0.006/min | 99 | Multilingual, self-hosted, batch |
| Soniox | ~8% | ~150ms TTFT | $0.0098/min | All | Ultra-low latency, speaker ID |
Hiring signal: Knowing WER (Word Error Rate) and TTFT (Time to First Token) for each provider shows production experience.
Layer 2: TTS (Text-to-Speech)
| Provider | MOS | TTFB | Price | Languages | Best For |
|---|
| ElevenLabs | ~4.5/5 | ~400ms | $0.30/min | 29 | Best quality, voice cloning |
| Cartesia Sonic | ~4.2/5 | ~120ms | $0.015/min | 15+ | Lowest TTFB, cost-effective |
| Deepgram Aura | ~4.0/5 | ~200ms | $0.025/min | 10+ | Integrated with Deepgram ASR |
| PlayHT | ~4.1/5 | ~250ms | $0.020/min | 25+ | Voice cloning, multilingual |
Hiring signal: MOS (Mean Opinion Score) and TTFB (Time to First Byte) are the key TTS metrics. Knowing Cartesia's 120ms TTFB vs ElevenLabs' 400ms shows you understand latency optimization.
Layer 3: LLM (Reasoning)
| Provider | Latency | Price | Best For |
|---|
| GPT-4o | ~300-500ms TTFT | $2.50/$10.00/M tokens | Best quality, multimodal |
| GPT-4o-mini | ~200-300ms TTFT | $0.15/$0.60/M tokens | Fast, cheap, transactional |
| Claude 3.5 Haiku | ~200-400ms TTFT | $0.25/$1.25/M tokens | Complex conversations |
| Groq Llama 3.1 8B | ~50-100ms TTFT | $0.05/$0.08/M tokens | Ultra-fast, lowest latency |
Why would you choose Groq Llama 3.1 8B over GPT-4o for a voice agent?
It has better voice quality
Layer 4: Frameworks
| Framework | Type | Architecture | Best For |
|---|
| Pipecat | Open-source | Chained pipeline | Full control, BYO providers |
| LiveKit Agents | Open-source | Chained pipeline | WebRTC-native, multi-participant |
| OpenAI Realtime API | Managed | Native audio | Lowest latency, natural prosody |
| Vapi | Managed platform | Chained + native | Fastest time-to-production, Squads |
| Retell AI | Managed platform | Chained pipeline | Inbound support, analytics |
| Bland AI | Managed platform | Chained pipeline | Outbound campaigns at scale |
Layer 5: Telephony
| Provider | Price/min | SIP | Best For |
|---|
| Twilio | $0.01 | Yes | Best docs, Media Streams, US-centric |
| Telnyx | $0.003 | Yes (native) | Cheapest, SIP-native, international |
| Vonage | $0.009 | Yes | Global reach, API-first |
| AWS Connect | $0.018 | Yes | AWS ecosystem, enterprise |
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Evaluation, Key Terms, Common Pitfalls, Interview Framing — plus a hands-on lab, quiz, and project artifact.
Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.
Browse all courses · View pricing · DeVenture Academy