Phase 0: Voice AI Fundamentals & Architecture · 50 min · Python · PyAudio · pydub
Audio Fundamentals: Sample Rates, Codecs, and Formats
If you don't understand sample rates and codecs, you will ship broken audio. Learn the fundamentals.
Hiring signal: Audio fundamentals knowledge distinguishes voice AI engineers from generalist AI engineers.
What you will learn
- Understand sample rates: 8kHz telephony, 16kHz ASR input, 24kHz output
- Explain bit depth, channels, and their relevance to voice agents
- Identify audio formats and codecs: PCM, μ-law, Opus, G.711
- Read and write WAV files programmatically
The Problem
Voice agents process audio. If you don't understand sample rates, bit depth, channels, and codecs, you'll ship agents with garbled audio, mismatched formats, and broken telephony integrations. These aren't theoretical concerns — they're the bugs that make voice agents sound terrible or crash in production.
Every voice AI interview includes a question about audio formats. "Why does telephony audio sound different from web audio?" is a screening question that eliminates candidates who haven't worked with real audio.
The Concept
Sample Rate
Sample rate is how many times per second the audio signal is measured. Higher sample rates capture more detail but require more bandwidth and processing.
| Sample Rate | Use Case | Quality | Bandwidth |
|---|
| 8 kHz | Telephony (PSTN, G.711) | Basic — sounds "phone-like" | Low |
| 16 kHz | ASR input (Deepgram, Whisper) | Good — sufficient for speech recognition | Medium |
| 22.05 kHz | Low-quality web audio | Fair | Medium |
| 24 kHz | TTS output (ElevenLabs, OpenAI) | Good — natural-sounding speech | Medium |
| 44.1 kHz | CD quality | High | High |
| 48 kHz | WebRTC default, professional audio | High | High |
Why do ASR providers like Deepgram recommend 16kHz audio instead of 48kHz?
48kHz is too expensive
Bit Depth
Bit depth determines the dynamic range — the difference between the quietest and loudest sounds that can be represented.
| Bit Depth | Dynamic Range | Use Case |
|---|
| 8-bit | ~48 dB | Low quality, telephony (μ-law) |
| 16-bit | ~96 dB | Standard for voice agents, CD quality |
| 24-bit | ~144 dB | Professional audio, studio recording |
| 32-bit float | ~1500 dB | Professional, no clipping |
For voice agents, 16-bit is the standard. It provides sufficient dynamic range for speech and is compatible with all ASR/TTS providers.
Channels
| Channels | Description | Use Case |
|---|
| Mono (1) | Single audio channel | Voice agents — both parties on one stream |
| Stereo (2) | Left and right channels | Music, spatial audio |
| Multi-channel | 4+ channels | Conference calls, speaker diarization |
Voice agents use mono. Stereo doubles the bandwidth with no benefit for speech.
Codecs
Codecs compress audio for transmission. The choice of codec affects latency, quality, and compatibility.
| Codec | Type | Bitrate | Latency | Quality | Use Case |
|---|
| PCM | Uncompressed | 128-1536 kbps | 0ms | Best | Internal processing, WAV files |
| G.711 (μ-law/A-law) | Companded | 64 kbps | ~0ms | Basic | PSTN telephony, TDM |
| Opus | Lossy | 6-510 kbps | 5-60ms | Excellent | WebRTC, VoIP, modern telephony |
| G.722 | Wideband | 64 kbps | ~0ms | Good | HD telephony |
| MP3 | Lossy | 32-320 kbps | High | Good | Music (not for voice agents) |
Opus is the codec of choice for WebRTC — it adapts bitrate to network conditions, has low latency (5-60ms frame sizes), and provides excellent quality at low bitrates.
Format Conversion in a Voice Agent Pipeline
The caller's audio arrives as 8kHz μ-law (telephony). It must be converted to 16kHz PCM for ASR. The TTS output is 24kHz PCM, which must be encoded as Opus for WebRTC transport back to the caller.
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Evaluation, Key Terms, Common Pitfalls, Interview Framing — plus a hands-on lab, quiz, and project artifact.
Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.
Browse all courses · View pricing · DeVenture Academy