The Problem
When you set out to build a voice agent, the first decision is architecture. Do you chain together separate ASR, LLM, and TTS services? Do you use a native audio model that handles speech-to-speech directly? Or something in between?
This single decision determines:
- Your latency ceiling — how fast can the agent respond?
- Your cost floor — what's the minimum per-minute cost?
- Your control surface — what can you customize?
- Your quality ceiling — how natural can the conversation feel?
Get this wrong and you'll either ship a slow agent nobody wants to talk to, or overspend on infrastructure you don't need.
The Concept
Architecture 1: Chained Pipeline (ASR → LLM → TTS)
The most common and flexible architecture. Audio comes in, ASR transcribes it to text, the LLM generates a text response, TTS converts that to audio.
Caller Speech → [ASR] → text → [LLM] → text → [TTS] → Agent Speech
Latency: 600-1300ms (three sequential service calls + network hops) Cost: $0.05-$0.40/min (depends on provider choices) Control: Full — swap any component, add custom processing between stages Quality: Good — TTS reads text, may sound less natural
What is the main advantage of the chained pipeline architecture?
The chained pipeline's main advantage is control. You can use Deepgram for ASR, GPT-4o for LLM, and ElevenLabs for TTS — swapping any one without changing the others. You can also add custom processing (sentiment analysis, intent detection, PII redaction) between stages. The trade-off is higher latency (600-1300ms) from sequential service calls.
Architecture 2: Half-Cascade
The model hears the caller directly — there is no separate ASR step — but it answers in text, and a text-to-speech stage turns that text into the agent's voice. Audio understanding is native; audio output is cascaded. That is where the name comes from: half of the old cascade remains.
Caller Speech → [Audio-in model] → text reply → [TTS] → Agent Speech
Latency: 400-800ms (two stages instead of three) Cost: $0.08-$0.30/min Control: Moderate — you can inspect, redact or log the text reply before it is spoken, and choose the voice; you cannot change how the model hears Quality: Good — the model hears tone and hesitation, though the voice is a TTS voice
Gemini's Live API, for example, offers half-cascade models alongside native-audio ones.
Architecture 3: Native Audio Model
The model processes speech-to-speech directly: audio in, audio out, with no text step in between and no separate ASR or TTS. Examples: OpenAI's gpt-realtime, and the native-audio models in Gemini's Live API.
Caller Speech → [Native Audio Model] → Agent Speech
Latency: 300-500ms (single model, no inter-service hops) Cost: $0.06-$0.24/min (audio token pricing) Control: Limited — locked to one provider, limited voices, no custom processing Quality: Best — natural prosody, emotion, pacing (the model "speaks" rather than "reads")
Architecture Comparison Table
| Dimension | Chained Pipeline | Half-Cascade | Native Audio |
|---|
| Latency | 600-1300ms | 400-800ms | 300-500ms |
| Cost/min | $0.05-$0.40 | $0.08-$0.30 | $0.06-$0.24 |
| Control | Full | Moderate | Limited |
| Voice quality | Good (TTS reads text) | Good (optimized) | Best (natural prosody) |
| Voice options | Any TTS provider | Some choice | Provider's built-in voices only |
| Custom processing | Between every stage | Some stages | None |
| Provider lock-in | None | Some | High |
| Setup complexity | High | Medium | Low |
| Best for | Production at scale | Balanced needs | Rapid prototyping |