Phase 6: Real-Time Transport & Telephony · 50 min · Python · FastAPI · websockets

WebSocket Audio Streaming: The OpenAI Realtime API Pattern

Base64 audio chunks as JSON events — the OpenAI Realtime API pattern that every voice agent uses.

Hiring signal: WebSocket audio streaming with JSON events is a core voice AI transport coding exercise.

What you will learn

Implement WebSocket-based audio streaming with base64-encoded chunks as JSON events
Handle OpenAI Realtime API event types: input_audio_buffer.append, response.create
Manage audio format negotiation: PCM16 24kHz mono
Handle reconnection, session management, and backpressure

The Problem

WebRTC requires ICE, STUN, TURN, DTLS — a lot of infrastructure. For server-to-server or app-to-server audio streaming, a simpler approach is WebSocket: send raw audio bytes over a persistent bidirectional connection. OpenAI's Realtime API uses exactly this pattern.

The Concept

WebSocket vs WebRTC

Factor	WebSocket	WebRTC
Complexity	Low (just a WS connection)	High (ICE/STUN/TURN/DTLS)
NAT traversal	None (needs server)	Built-in
Audio encoding	Manual (send raw PCM/Opus)	Built-in (Opus)
Echo cancellation	Manual	Built-in
Latency	~100ms	~50ms
Infrastructure	Just a WS server	STUN + TURN servers
Best for	App-to-server, server-to-server	Browser-to-server

The OpenAI Realtime API Pattern

OpenAI's Realtime API uses WebSocket for bidirectional audio:

WebSocket Audio Protocol

import asyncio
import websockets
import json

async def realtime_voice_session(ws):
    """Handle a WebSocket voice session (OpenAI Realtime API pattern)."""
    async for message in ws:
        if isinstance(message, bytes):
            # Audio chunk from client (PCM16 24kHz)
            await process_client_audio(message)
        elif isinstance(message, str):
            # JSON event from client
            event = json.loads(message)
            await handle_event(event, ws)

Event Types

Direction	Event	Purpose
Client → Server	`input_audio_buffer.append`	Send audio chunk
Client → Server	`input_audio_buffer.commit`	End of speech
Client → Server	`response.create`	Trigger response
Server → Client	`response.audio.delta`	Audio chunk from agent
Server → Client	`response.audio.done`	Agent finished speaking
Server → Client	`conversation.item.input_audio_transcription`	ASR result
Server → Client	`error`	Error event

Why does the OpenAI Realtime API use WebSocket instead of WebRTC?

WebSocket is faster

Audio Format

Parameter	Value	Why
Sample rate	24,000 Hz	OpenAI standard for Realtime API
Bit depth	16-bit (PCM16)	Standard for voice
Channels	1 (mono)	Voice = mono
Chunk size	2400 bytes	50ms of audio at 24kHz 16-bit
Encoding	Raw PCM (no compression)	Simplicity over bandwidth

# Audio chunk calculation
SAMPLE_RATE = 24000  # Hz
BIT_DEPTH = 16  # bits
CHANNELS = 1
CHUNK_DURATION_MS = 50

chunk_samples = SAMPLE_RATE * CHUNK_DURATION_MS / 1000  # 1200 samples
chunk_bytes = chunk_samples * (BIT_DEPTH // 8) * CHANNELS  # 2400 bytes

Full-Duplex Audio Streaming

class WebSocketVoiceSession:
    def __init__(self, ws):
        self.ws = ws
        self.audio_buffer = bytearray()
        self.is_speaking = False

    async def run(self):
        """Run bidirectional audio session."""
        # Start audio output task (server → client)
        output_task = asyncio.create_task(self._send_audio())

        # Process incoming messages (client → server)
        async for message in self.ws:
            if isinstance(message, bytes):
                await self._handle_audio_input(message)
            else:
                await self._handle_event(json.loads(message))

        output_task.cancel()

    async def _handle_audio_input(self, audio_chunk):
        """Process incoming audio from client."""
        self.audio_buffer.extend(audio_chunk)
        # Feed to VAD/ASR pipeline
        await self.asr.process(audio_chunk)

    async def _send_audio(self):
        """Send agent audio to client."""
        while True:
            audio = await self.tts_queue.get()
            await self.ws.send(audio)  # Send PCM16 chunk

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Evaluation, Key Terms, Common Pitfalls, Interview Framing — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.