Phase 3: Text-to-Speech (TTS) · 50 min · Python · ElevenLabs SDK · Cartesia SDK

Streaming TTS: Why First-Byte Latency Matters

In voice AI, TTFB is king. Users don't wait for the full sentence — they wait for the first sound.

Real-world application: Streaming TTS and TTFB optimization is a core voice AI engineering skill tested in coding rounds.

What you will learn

Implement streaming TTS where audio chunks are sent as they're generated
Measure TTFB separately from total generation time
Combine streaming LLM output with streaming TTS for parallel processing
Match TTS chunk size to transport frame size (20ms)

The Problem

When the LLM finishes generating a response, the caller waits in silence until TTS produces the first audio byte. This gap — TTS Time to First Byte (TTFB) — is the most noticeable latency in a voice agent. Every millisecond of TTFB is a millisecond of dead air.

The Concept

The TTFB Problem

Without streaming TTS, the caller waits 3s+ in silence. With streaming TTS, audio starts playing as soon as the first chunk is ready.

Streaming TTS Architecture

With streaming TTS + streaming LLM, the caller hears the first word while the LLM is still generating the rest.

Chunked vs Full Audio

Mode	TTFB	Total Time	Caller Experience
Full audio	2-5s	2-5s	Long silence, then full response
Streaming chunks	120-400ms	2-5s	Immediate response, streams in
First-word streaming	120-200ms	2-5s	Best — first word almost instant

What's the difference between TTFB and total generation time for TTS?

TTFB = when the first audio chunk reaches the caller (perceived latency). Total time = when all audio is fully generated. With streaming TTS, TTFB is 120ms but total time might be 3s for a long sentence. The caller hears the first word at 120ms and the rest streams in seamlessly — they never experience the 3s total time as silence. This is why streaming TTS is critical: it decouples perceived latency from total generation time.

Streaming LLM + Streaming TTS Pipeline

The key optimization: pipe LLM tokens directly to TTS as they stream:

async def streaming_pipeline(user_text):
    """LLM streams tokens → TTS streams audio → caller hears immediately."""
    async for token in llm_stream(user_text):
        # Send each token to TTS as it arrives
        async for audio_chunk in tts_stream(token):
            # Send audio chunk to caller immediately
            await websocket.send(audio_chunk)

This means the caller hears the first word while the LLM is still generating the rest of the sentence.

TTFB Breakdown

Component	Time	Optimization
Network to TTS API	20-50ms	Edge deployment, connection pooling
Text preprocessing	5-20ms	Minimal processing, cache phonemes
Model inference (first chunk)	50-300ms	Architecture choice (flow-matching fastest)
Audio encoding	5-20ms	Use PCM, avoid encoding overhead
Network back to caller	20-50ms	Edge deployment
Total TTFB	120-400ms	Architecture is the biggest factor

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.