Phase 1: Audio Processing & Streaming · 55 min · Python · noisereduce · numpy

Audio Chunking, Buffering, and Noise Suppression

20ms frames are the heartbeat of voice AI. Your buffer size is your latency floor.

Hiring signal: Buffering and frame management is tested in streaming audio coding rounds at Deepgram and ElevenLabs.

What you will learn

Implement audio chunking with 20ms frames as the standard unit
Build ring buffers and jitter buffers for streaming audio
Apply audio normalization, gain control, and noise suppression
Understand the trade-off between buffer size and latency

The Problem

Streaming audio doesn't arrive in neat packages. It comes in variable-size chunks from WebRTC, Twilio Media Streams, or WebSocket connections. You need to:

Chunk the incoming stream into fixed 20ms frames for processing
Buffer frames to handle network jitter and variable arrival times
Suppress noise that would degrade ASR accuracy

Get any of these wrong and you'll hear clicks, gaps, or have ASR transcribing background noise instead of speech.

The Concept

20ms Frames: The Fundamental Unit

Voice AI processes audio in 20ms frames. This is the standard because:

Low latency: 20ms is small enough for real-time response
VAD compatibility: Voice Activity Detection works on 20-30ms windows
Opus compatibility: 20ms is the default Opus frame size
ASR compatibility: Streaming ASR expects 20ms chunks

Sample Rate	Samples per 20ms	Bytes per 20ms (16-bit mono)
8kHz	160	320
16kHz	320	640
24kHz	480	960
48kHz	960	1920

Why is 20ms the standard frame size for voice agents?

It produces the best audio quality

Ring Buffers

A ring buffer (circular buffer) is the standard data structure for streaming audio. It allows a producer (network) to write audio and a consumer (processing pipeline) to read it at different rates.

class RingBuffer:
    def __init__(self, capacity_frames, frame_size):
        self.capacity = capacity_frames
        self.frame_size = frame_size
        self.buffer = [None] * capacity
        self.write_pos = 0
        self.read_pos = 0
        self.count = 0

    def write(self, frame):
        if self.count >= self.capacity:
            return False  # Overflow — drop frame
        self.buffer[self.write_pos] = frame
        self.write_pos = (self.write_pos + 1) % self.capacity
        self.count += 1
        return True

    def read(self):
        if self.count == 0:
            return None  # Underflow — no frame available
        frame = self.buffer[self.read_pos]
        self.read_pos = (self.read_pos + 1) % self.capacity
        self.count -= 1
        return frame

Jitter Buffers

Network audio arrives with variable timing (jitter). A jitter buffer holds a few frames to smooth out arrival times:

Too small: Underflows cause gaps (clicks in audio)
Too large: Adds unnecessary latency
Typical: 3-5 frames (60-100ms buffer)

Noise Suppression

Background noise degrades ASR accuracy. Common approaches:

Method	Quality	CPU	Latency	Use Case
Spectral subtraction	Good	Low	~20ms	Simple, effective for steady noise
noisereduce library	Good	Medium	~40ms	Easy to use, RNNoise-based
DeepFilterNet	Excellent	High	~20ms	ML-based, best for complex noise
WebRTC NS	Good	Low	~10ms	Built into WebRTC, real-time
noisereduce + VAD	Very good	Medium	~40ms	Only suppresses non-speech frames

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Evaluation, Key Terms, Common Pitfalls, Interview Framing — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.