Phase 1: Audio Processing & Streaming · 45 min · Python · pydub · audioop

Audio Formats and Codecs: PCM, μ-law, Opus, G.711

Telephony speaks μ-law. The web speaks PCM. ASR expects 16kHz. You must translate between all three.

Hiring signal: Audio format conversion between telephony and web formats is a core voice AI engineering skill.

What you will learn

Convert between PCM, μ-law, Opus, and G.711 audio formats
Understand when each codec is used in the voice AI stack
Build a reusable audio format converter
Handle format conversion edge cases: clipping, aliasing, DC offset

The Problem

A caller's voice arrives as 8kHz μ-law (telephony). ASR needs 16kHz PCM. TTS outputs 24kHz PCM. WebRTC transport needs Opus at 48kHz. That's four format conversions in a single pipeline, each with quality and latency implications.

If you don't understand codecs, you'll ship agents with:

Garbled audio (wrong codec)
Silent calls (format mismatch)
Excessive bandwidth (uncompressed where compressed would work)
Quality degradation (lossy → lossy → lossy conversion chain)

The Concept

The Format Landscape

Format	Type	Bitrate	Latency	Quality	Where Used
PCM	Uncompressed	128-1536 kbps	0ms	Best	Internal processing, WAV files
μ-law (G.711)	Companded	64 kbps	~0ms	Basic	PSTN telephony (North America)
A-law (G.711)	Companded	64 kbps	~0ms	Basic	PSTN telephony (Europe)
Opus	Lossy	6-510 kbps	5-60ms	Excellent	WebRTC, VoIP
G.722	Wideband	64 kbps	~0ms	Good	HD telephony

μ-law Companding

μ-law is the telephony standard. It compresses 16-bit PCM to 8-bit using a logarithmic curve that gives more resolution to quiet sounds (where speech energy is concentrated) and less to loud sounds.

# μ-law encoding: 16-bit → 8-bit
MU = 255
def pcm_to_mulaw(sample):
    sign = 1 if sample >= 0 else -1
    magnitude = abs(sample) / 32768.0
    compressed = sign * np.log1p(MU * magnitude) / np.log1p(MU)
    return int((compressed + 1) * 127.5)

Why does μ-law use a logarithmic curve instead of linear quantization?

It produces smaller files

Opus: The WebRTC Codec

Opus is the mandatory codec for WebRTC. It's unique because it adapts:

Bitrate: 6-510 kbps (adapts to network conditions)
Frame size: 2.5-60ms (lower = lower latency, higher = less overhead)
Mode: SILK (voice-optimized) or CELT (music-optimized), auto-selected
Bandwidth: Narrowband (8kHz) to Fullband (48kHz)

For voice agents, Opus typically runs at:

20ms frames (standard for streaming)
16-24kHz bandwidth
20-40 kbps bitrate
SILK mode (voice-optimized)

The Conversion Pipeline

Quality Measurement: SNR

Signal-to-Noise Ratio (SNR) measures how much quality is lost in a conversion:

def compute_snr(original, converted):
    signal_power = np.mean(original.astype(np.float32) ** 2)
    noise_power = np.mean((original.astype(np.float32) - converted.astype(np.float32)) ** 2)
    if noise_power == 0:
        return float('inf')
    return 10 * np.log10(signal_power / noise_power)

Conversion	Typical SNR	Quality
PCM 24kHz → μ-law 8kHz	~35 dB	Noticeable quality loss
μ-law 8kHz → PCM 16kHz	∞ (no loss)	Lossless (just resampling)
PCM → Opus 24kbps	~45 dB	Minimal perceptible loss
Opus → PCM (decode)	~50 dB	Near-transparent

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Evaluation, Key Terms, Common Pitfalls, Interview Framing — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.