Phase 5: Turn-Taking & Conversation Management · 55 min · Python · Silero VAD · webrtcvad

Voice Activity Detection (VAD): Silero, WebRTC, Semantic

VAD is the gatekeeper of every voice agent. Get it wrong and your agent either interrupts or stares in silence.

What you will learn

Implement Voice Activity Detection with Silero VAD and WebRTC VAD
Tune silence gap thresholds (300–800ms) to balance false interruptions vs awkward pauses
Compare VAD providers on clean and noisy audio samples
Measure false positive rate, false negative rate, and latency to turn detection

The Problem

Your voice agent receives a continuous audio stream. It needs to know:

When the caller starts speaking (to start ASR)
When the caller stops speaking (to trigger LLM response)
When it's background noise (to ignore)

This is Voice Activity Detection (VAD) — the gatekeeper of the entire pipeline.

The Concept

VAD Types

VAD Type	How It Works	Latency	Accuracy	Best For
Energy-based	Volume threshold	<1ms	Low	Clean environments
WebRTC VAD	GMM-based, 4 modes	~1ms	Good	Real-time, low CPU
Silero VAD	Neural network (ONNX)	~10ms	Excellent	Production voice agents
Semantic VAD	LLM judges if speech is complete	~200ms	Best	Natural conversations

WebRTC VAD

WebRTC VAD is a lightweight, GMM-based detector with 4 aggressiveness modes:

import webrtcvad

vad = webrtcvad.Vad(3)  # 0=least aggressive, 3=most aggressive

# Process 30ms frames
is_speech = vad.is_speech(audio_frame, sample_rate=16000)

Mode	Sensitivity	False Positives	False Negatives
0	Low	Many	Few
1	Medium-Low	Moderate	Few
2	Medium	Few	Moderate
3	High	Few	Many

Silero VAD

Silero VAD is a neural network model that's much more accurate:

import torch
from silero_vad import load_silero_vad, read_audio, get_speech_timestamps

model = load_silero_vad()
wav = read_audio('recording.wav', sampling_rate=16000)
timestamps = get_speech_timestamps(wav, model)
# Returns list of {start, end} for each speech segment

Why is Silero VAD better than WebRTC VAD for production voice agents?

Silero VAD uses a neural network trained on diverse audio — it correctly distinguishes speech from background noise (coughing, keyboard typing, traffic) that WebRTC's GMM-based approach misclassifies. WebRTC VAD mode 3 (aggressive) misses quiet speech; mode 0 (lenient) triggers on noise. Silero handles both with ~95% accuracy vs WebRTC's ~85%. The 10ms processing latency is negligible within a 500ms budget. For production voice agents, Silero's accuracy prevents false triggers (agent responds to noise) and missed speech (agent ignores the caller).

Semantic VAD

Semantic VAD uses the LLM to judge whether speech is complete:

async def semantic_vad(partial_transcript):
    """Use LLM to determine if the speaker is done talking."""
    prompt = f"""Is this a complete thought, or is the speaker likely to continue?
Transcript: "{partial_transcript}"
Respond with COMPLETE or CONTINUE."""

    response = await llm.generate(prompt, max_tokens=1)
    return response.strip() == "COMPLETE"

Scenario	Transcript	VAD Result
Complete	"I want to book a flight to Tokyo"	COMPLETE
Incomplete	"I want to book a flight to..."	CONTINUE
Complete	"What time does it arrive?"	COMPLETE
Incomplete	"What time does it..."	CONTINUE
Ambiguous	"I think maybe"	CONTINUE

VAD Pipeline Integration

Endpointing Thresholds

Setting	Value	Effect
Min speech duration	100ms	Ignore clicks/bangs
Max speech duration	30s	Force-stop long monologues
Silence threshold	500ms	Wait this long before ending
Pre-speech buffer	200ms	Include audio before VAD trigger
Post-speech buffer	300ms	Include trailing audio

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.