Phase 1: Audio Processing & Streaming · 45 min · Python · pydub · audioop

Audio Formats and Codecs: PCM, μ-law, Opus, G.711

Telephony speaks μ-law. The web speaks PCM. ASR expects 16kHz. You must translate between all three.

Real-world application: Audio format conversion between telephony and web formats is a core voice AI engineering skill.

What you will learn

Convert between PCM, μ-law, Opus, and G.711 audio formats
Understand when each codec is used in the voice AI stack
Build a reusable audio format converter
Handle format conversion edge cases: clipping, aliasing, DC offset

The Problem

A caller's voice arrives as 8kHz μ-law (telephony). ASR needs 16kHz PCM. TTS outputs 24kHz PCM. WebRTC transport needs Opus at 48kHz. That's four format conversions in a single pipeline, each with quality and latency implications.

If you don't understand codecs, you'll ship agents with:

Garbled audio (wrong codec)
Silent calls (format mismatch)
Excessive bandwidth (uncompressed where compressed would work)
Quality degradation (lossy → lossy → lossy conversion chain)

The Concept

The Format Landscape

Format	Type	Bitrate	Latency	Quality	Where Used
PCM	Uncompressed	128-1536 kbps	0ms	Best	Internal processing, WAV files
μ-law (G.711)	Companded	64 kbps	~0ms	Basic	PSTN telephony (North America)
A-law (G.711)	Companded	64 kbps	~0ms	Basic	PSTN telephony (Europe)
Opus	Lossy	6-510 kbps	5-60ms	Excellent	WebRTC, VoIP
G.722	Wideband	64 kbps	~0ms	Good	HD telephony

μ-law Companding

μ-law is the telephony standard. It compresses 16-bit PCM to 8-bit using a logarithmic curve that gives more resolution to quiet sounds (where speech energy is concentrated) and less to loud sounds.

# μ-law encoding: 16-bit → 8-bit
MU = 255
def pcm_to_mulaw(sample):
    sign = 1 if sample >= 0 else -1
    magnitude = abs(sample) / 32768.0
    compressed = sign * np.log1p(MU * magnitude) / np.log1p(MU)
    return int((compressed + 1) * 127.5)

Why does μ-law use a logarithmic curve instead of linear quantization?

Human speech has a wide dynamic range — whispers to shouts. Linear 8-bit quantization would give equal resolution across all levels, losing detail in quiet speech where intelligibility matters most. μ-law's logarithmic curve allocates more quantization levels to quiet sounds (where speech energy concentrates) and fewer to loud sounds, matching human auditory perception. This is the same principle as MP3's psychoacoustic model — allocate bits where they matter perceptually.

Opus: The WebRTC Codec

Opus is the mandatory codec for WebRTC. It's unique because it adapts:

Bitrate: 6-510 kbps (adapts to network conditions)
Frame size: 2.5-60ms (lower = lower latency, higher = less overhead)
Mode: SILK (voice-optimized) or CELT (music-optimized), auto-selected
Bandwidth: Narrowband (8kHz) to Fullband (48kHz)

For voice agents, Opus typically runs at:

20ms frames (standard for streaming)
16-24kHz bandwidth
20-40 kbps bitrate
SILK mode (voice-optimized)

The Conversion Pipeline

Quality Measurement: SNR

Signal-to-Noise Ratio (SNR) measures how much quality is lost in a conversion:

def compute_snr(original, converted):
    signal_power = np.mean(original.astype(np.float32) ** 2)
    noise_power = np.mean((original.astype(np.float32) - converted.astype(np.float32)) ** 2)
    if noise_power == 0:
        return float('inf')
    return 10 * np.log10(signal_power / noise_power)

Conversion	Typical SNR	Quality
PCM 24kHz → μ-law 8kHz	~35 dB	Noticeable quality loss
μ-law 8kHz → PCM 16kHz	∞ (no loss)	Lossless (just resampling)
PCM → Opus 24kbps	~45 dB	Minimal perceptible loss
Opus → PCM (decode)	~50 dB	Near-transparent

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.