Phase 3: Text-to-Speech (TTS) · 45 min · Python

How TTS Works: Autoregressive, Diffusion, and Flow-Matching

Understanding TTS architectures tells you why Cartesia is fast and ElevenLabs is high-quality.

Hiring signal: TTS architecture knowledge shows depth beyond API calls — distinguishing engineers from integrators.

What you will learn

Explain autoregressive, diffusion-based, and flow-matching TTS architectures
Understand neural vocoders: HiFi-GAN, BigVGAN
Relate TTS architecture to latency and quality trade-offs
Explain why flow-matching (Cartesia) achieves both speed and quality

The Problem

Text-to-Speech converts the LLM's text response into audio the caller hears. The TTS architecture choice determines:

Latency: When the first audio byte reaches the caller (TTFB)
Quality: How natural the voice sounds (MOS)
Cost: Per-minute pricing
Control: Can you clone voices, adjust prosody, control emotion?

The Concept

TTS Architecture Types

Architecture	Latency	Quality	Cost	Streaming	Examples
Autoregressive	High (300-500ms TTFB)	Excellent	High	Chunk-by-chunk	ElevenLabs, XTTS
Diffusion	Medium (200-400ms TTFB)	Very Good	Medium	Chunk-by-chunk	Tortoise-TTS
Flow-Matching	Low (100-200ms TTFB)	Good	Low	Natural streaming	Cartesia Sonic
Concatenative	Very Low (<50ms TTFB)	Fair	Very Low	Pre-computed	Legacy IVR
Vocoder-based	Low (100-300ms TTFB)	Good	Low	Frame-by-frame	Deepgram Aura

Autoregressive TTS (ElevenLabs)

Autoregressive TTS generates audio one token at a time, each token conditioned on all previous tokens:

Pros: Best quality, natural prosody, voice cloning
Cons: High TTFB (300-500ms), can't parallelize, expensive
TTFB: Must generate first token before any audio output

Why does autoregressive TTS have higher TTFB than flow-matching TTS?

It runs on slower hardware

Flow-Matching TTS (Cartesia Sonic)

Flow-matching learns to transform a simple distribution (Gaussian noise) into the target audio distribution:

Pros: Low TTFB (100-200ms), natural streaming, cost-effective
Cons: Slightly lower quality than autoregressive, limited voice cloning
TTFB: First audio chunk generated in one forward pass

Vocoder-Based TTS (Deepgram Aura)

Uses a text-to-spectrogram model + neural vocoder:

Text → Spectrogram (fast, ~50ms)
Spectrogram → Audio waveform (vocoder, ~100ms)
Total TTFB: ~150-200ms

The Latency Budget Impact

In a 500ms glass-to-glass budget:

Component	Budget	Typical
ASR	150ms	150ms
LLM	150ms	200ms
TTS TTFB	100ms	120-400ms
Network	50ms	50ms
Total	500ms	520-800ms

With ElevenLabs (400ms TTFB): total = 800ms — misses budget With Cartesia (120ms TTFB): total = 520ms — meets budget

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Evaluation, Key Terms, Common Pitfalls, Interview Framing — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.