Phase 1: Audio Processing & Streaming · 60 min · Python · FastAPI · websockets

WebSocket Audio Streaming Server

The WebSocket server is the spine of every voice agent. Build it right and everything else gets easier.

What you will learn

Build a WebSocket server that receives streaming audio chunks
Buffer audio into 20ms frames and forward to a processing pipeline
Handle reconnection, backpressure, and frame ordering
Visualize streamed audio in real-time

The Problem

Your voice agent needs to receive audio from a caller and send audio back — in real-time, with sub-second latency. The transport layer is the pipe that carries audio between client and server.

For web-based voice agents, WebSocket is the standard transport. It provides:

Bidirectional, full-duplex communication
Low latency (no HTTP request/response overhead)
Binary frame support (for raw audio data)
Persistent connection (no reconnection per message)

For telephony-based agents, Twilio Media Streams or SIP/RTP is used instead, but the server-side architecture is similar.

The Concept

WebSocket Audio Streaming Architecture

The WebSocket Connection Lifecycle

Connect: Client opens WebSocket connection to /ws/audio
Handshake: Server accepts, sends config (sample rate, frame size)
Streaming: Client sends audio chunks; server processes and responds
Disconnect: Client closes connection; server cleans up session
Reconnect: Client reconnects; server restores session state

Backpressure Handling

When the server can't process audio fast enough (ASR is slow, LLM is backed up), frames pile up. Without backpressure handling, memory grows unbounded.

Strategies:

Drop oldest: Ring buffer overwrites oldest frames (preferred for real-time)
Drop newest: Reject new frames (preserves older audio)
Throttle: Tell client to slow down (requires client cooperation)
Disconnect: Close connection if too far behind (last resort)

Your WebSocket server receives audio faster than it can process it. Frames are piling up in memory. What do you do?

In real-time audio, stale frames are useless — the caller has already spoken past them. A ring buffer with fixed capacity that drops the oldest frame on overflow is the standard approach. This keeps latency bounded (you're always processing recent audio) and memory bounded (fixed buffer size). The alternative — accumulating all frames — creates growing latency that eventually makes the conversation unusable.

Reconnection Handling

Network connections drop. A production voice agent must handle reconnection:

# Server-side session management
sessions = {}  # session_id -> SessionState

@app.websocket("/ws/audio")
async def audio_endpoint(websocket: WebSocket):
    session_id = websocket.query_params.get("session_id", str(uuid4()))

    if session_id in sessions:
        # Reconnection — restore state
        session = sessions[session_id]
        session.websocket = websocket
        await websocket.send_json({"type": "reconnected", "state": session.state})
    else:
        # New session
        session = SessionState(session_id, websocket)
        sessions[session_id] = session

    await session.handle_stream()

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.

Key terms in this lesson

Streaming