Phase 6: Real-Time Transport & Telephony · 60 min · Python · FastAPI · Twilio SDK

Twilio Media Streams: Streaming Bidirectional Audio on PSTN

Twilio Media Streams is how your voice agent answers actual phone calls. Master it.

Hiring signal: Twilio Media Streams integration is the most common telephony coding exercise in voice AI interviews.

What you will learn

Integrate Twilio Media Streams API for streaming bidirectional audio on PSTN calls
Configure TwiML <Connect><Stream> for inbound and outbound calls
Handle Twilio events: Start, Media, Stop
Convert between μ-law 8kHz (Twilio) and PCM 16kHz/24kHz (ASR/TTS)

The Problem

Your voice agent runs on a server. Real callers use phones on the PSTN (Public Switched Telephone Network). You need a bridge: the caller dials a phone number, Twilio connects the call, and streams bidirectional audio to your WebSocket server where your voice agent processes it.

The Concept

Twilio Media Streams Architecture

Component	Role
Twilio Phone Number	Caller dials this
TwiML `<Connect><Stream>`	Tells Twilio to stream audio to your WebSocket
WebSocket Server	Receives/sends audio chunks
μ-law 8kHz	Audio format on PSTN (G.711)
Your Voice Agent	ASR → LLM → TTS pipeline

TwiML Configuration

<Response>
  <Connect>
    <Stream url="wss://your-server.com/socket" />
  </Connect>
</Response>

# Python webhook handler
from fastapi import FastAPI, Response

app = FastAPI()

@app.post("/voice")
async def voice_webhook():
    """Twilio webhook — return TwiML to start media stream."""
    twiml = """<Response>
      <Connect>
        <Stream url="wss://your-server.com/socket" />
      </Connect>
    </Response>"""
    return Response(content=twiml, media_type="text/xml")

What audio format does Twilio Media Streams use by default?

MP3 44.1kHz

WebSocket Message Format

Twilio sends JSON messages over the WebSocket:

{
  "event": "media",
  "media": {
    "track": "inbound",
    "chunk": 1,
    "timestamp": "2024-01-01T00:00:00.000Z",
    "payload": "<base64-encoded μ-law audio>"
  }
}

Event	Direction	Purpose
`connected`	Server → Twilio	WebSocket connected
`start`	Twilio → Server	Call started, includes CallSid
`media`	Twilio → Server	Audio chunk (inbound from caller)
`media`	Server → Twilio	Audio chunk (outbound to caller)
`stop`	Twilio → Server	Call ended

Handling Audio Chunks

import base64
import json

async def handle_twilio_stream(websocket):
    """Handle Twilio Media Stream WebSocket."""
    call_sid = None

    async for message in websocket:
        data = json.loads(message)
        event = data["event"]

        if event == "start":
            call_sid = data["start"]["callSid"]
            print(f"Call started: {call_sid}")

        elif event == "media":
            # Decode base64 audio
            audio_b64 = data["media"]["payload"]
            audio_bytes = base64.b64decode(audio_b64)

            # Convert μ-law to PCM16
            pcm_audio = mulaw_to_pcm16(audio_bytes)

            # Feed to ASR pipeline
            await asr.process(pcm_audio)

            # When agent has response, send audio back
            if agent_audio := await get_agent_audio():
                # Convert PCM16 to μ-law
                mulaw_audio = pcm16_to_mulaw(agent_audio)
                # Send back to Twilio
                await websocket.send(json.dumps({
                    "event": "media",
                    "streamSid": data["streamSid"],
                    "media": {
                        "payload": base64.b64encode(mulaw_audio).decode()
                    }
                }))

        elif event == "stop":
            print(f"Call ended: {call_sid}")

μ-law to PCM16 Conversion

import audioop

def mulaw_to_pcm16(mulaw_bytes, source_rate=8000, target_rate=16000):
    """Convert μ-law 8kHz to PCM16 16kHz."""
    # Convert μ-law to PCM16
    pcm = audioop.ulaw2lin(mulaw_bytes, 2)  # 2 = 16-bit
    # Resample from 8kHz to 16kHz
    pcm_16k = audioop.ratecv(pcm, 2, 1, source_rate, target_rate)[0]
    return pcm_16k

def pcm16_to_mulaw(pcm_bytes, source_rate=16000, target_rate=8000):
    """Convert PCM16 16kHz to μ-law 8kHz."""
    # Resample from 16kHz to 8kHz
    pcm_8k = audioop.ratecv(pcm_bytes, 2, 1, source_rate, target_rate)[0]
    # Convert PCM16 to μ-law
    mulaw = audioop.lin2ulaw(pcm_8k, 2)
    return mulaw

Latency Budget with Twilio

Component	Latency	Notes
PSTN → Twilio	30ms	Phone network
Twilio → WebSocket	20ms	Network
ASR (8kHz)	150ms	Telephone quality
LLM TTFT	150ms	GPT-4o-mini
TTS	120ms	Cartesia
WebSocket → Twilio	20ms	Network
Twilio → PSTN	30ms	Phone network
Total	520ms	Slightly over 500ms budget

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Evaluation, Key Terms, Common Pitfalls, Interview Framing — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.