The Problem
When you set out to build a voice agent, the first decision is architecture. Do you chain together separate ASR, LLM, and TTS services? Do you use a native audio model that handles speech-to-speech directly? Or something in between?
This single decision determines:
- Your latency ceiling — how fast can the agent respond?
- Your cost floor — what's the minimum per-minute cost?
- Your control surface — what can you customize?
- Your quality ceiling — how natural can the conversation feel?
Get this wrong and you'll either ship a slow agent nobody wants to talk to, or overspend on infrastructure you don't need.
The Concept
Architecture 1: Chained Pipeline (ASR → LLM → TTS)
The most common and flexible architecture. Audio comes in, ASR transcribes it to text, the LLM generates a text response, TTS converts that to audio.
Caller Speech → [ASR] → text → [LLM] → text → [TTS] → Agent Speech
Latency: 600-1300ms (three sequential service calls + network hops) Cost: $0.05-$0.40/min (depends on provider choices) Control: Full — swap any component, add custom processing between stages Quality: Good — TTS reads text, may sound less natural
What is the main advantage of the chained pipeline architecture?
Full control — you can swap any component (ASR, LLM, TTS) independently, add custom processing between stages, and optimize each layer separately
Architecture 2: Half-Cascade (Speech-to-Speech)
An optimization of the chained pipeline where the model can generate audio output directly from text, skipping the separate TTS call. Some implementations also process audio input directly without full ASR transcription.
Caller Speech → [ASR] → text → [LLM + TTS] → Agent Speech
Latency: 400-800ms (two service calls instead of three) Cost: $0.08-$0.30/min Control: Moderate — some components are coupled Quality: Good — optimized pipeline, may have better prosody
Architecture 3: Native Audio Model
The model processes speech-to-speech directly. No separate ASR, LLM, or TTS. Audio in, audio out. Examples: OpenAI gpt-realtime, Google Gemini Live, Amazon Nova Sonic.
Caller Speech → [Native Audio Model] → Agent Speech
Latency: 300-500ms (single model, no inter-service hops) Cost: $0.06-$0.24/min (audio token pricing) Control: Limited — locked to one provider, limited voices, no custom processing Quality: Best — natural prosody, emotion, pacing (the model "speaks" rather than "reads")
Architecture Comparison Table
| Dimension | Chained Pipeline | Half-Cascade | Native Audio |
|---|
| Latency | 600-1300ms | 400-800ms | 300-500ms |
| Cost/min | $0.05-$0.40 | $0.08-$0.30 | $0.06-$0.24 |
| Control | Full | Moderate | Limited |
| Voice quality | Good (TTS reads text) | Good (optimized) | Best (natural prosody) |
| Voice options | Any TTS provider | Some choice | Provider's built-in voices only |
| Custom processing | Between every stage | Some stages | None |
| Provider lock-in | None | Some | High |
| Setup complexity | High | Medium | Low |
| Best for | Production at scale | Balanced needs | Rapid prototyping |
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Evaluation, Key Terms, Common Pitfalls, Interview Framing — plus a hands-on lab, quiz, and project artifact.
Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.