Phase 5: Turn-Taking & Conversation Management · 55 min · Python · Silero VAD · webrtcvad

Voice Activity Detection (VAD): Silero, WebRTC, Semantic

VAD is the gatekeeper of every voice agent. Get it wrong and your agent either interrupts or stares in silence.

Hiring signal: VAD implementation and tuning is the most common advanced coding exercise in voice AI interviews.

What you will learn

Implement Voice Activity Detection with Silero VAD and WebRTC VAD
Tune silence gap thresholds (300–800ms) to balance false interruptions vs awkward pauses
Compare VAD providers on clean and noisy audio samples
Measure false positive rate, false negative rate, and latency to turn detection

The Problem

Your voice agent receives a continuous audio stream. It needs to know:

When the caller starts speaking (to start ASR)
When the caller stops speaking (to trigger LLM response)
When it's background noise (to ignore)

This is Voice Activity Detection (VAD) — the gatekeeper of the entire pipeline.

The Concept

VAD Types

VAD Type	How It Works	Latency	Accuracy	Best For
Energy-based	Volume threshold	<1ms	Low	Clean environments
WebRTC VAD	GMM-based, 4 modes	~1ms	Good	Real-time, low CPU
Silero VAD	Neural network (ONNX)	~10ms	Excellent	Production voice agents
Semantic VAD	LLM judges if speech is complete	~200ms	Best	Natural conversations

WebRTC VAD

WebRTC VAD is a lightweight, GMM-based detector with 4 aggressiveness modes:

import webrtcvad

vad = webrtcvad.Vad(3)  # 0=least aggressive, 3=most aggressive

# Process 30ms frames
is_speech = vad.is_speech(audio_frame, sample_rate=16000)

Mode	Sensitivity	False Positives	False Negatives
0	Low	Many	Few
1	Medium-Low	Moderate	Few
2	Medium	Few	Moderate
3	High	Few	Many

Silero VAD

Silero VAD is a neural network model that's much more accurate:

import torch
from silero_vad import load_silero_vad, read_audio, get_speech_timestamps

model = load_silero_vad()
wav = read_audio('recording.wav', sampling_rate=16000)
timestamps = get_speech_timestamps(wav, model)
# Returns list of {start, end} for each speech segment

Why is Silero VAD better than WebRTC VAD for production voice agents?

It uses less memory

Semantic VAD

Semantic VAD uses the LLM to judge whether speech is complete:

async def semantic_vad(partial_transcript):
    """Use LLM to determine if the speaker is done talking."""
    prompt = f"""Is this a complete thought, or is the speaker likely to continue?
Transcript: "{partial_transcript}"
Respond with COMPLETE or CONTINUE."""

    response = await llm.generate(prompt, max_tokens=1)
    return response.strip() == "COMPLETE"

Scenario	Transcript	VAD Result
Complete	"I want to book a flight to Tokyo"	COMPLETE
Incomplete	"I want to book a flight to..."	CONTINUE
Complete	"What time does it arrive?"	COMPLETE
Incomplete	"What time does it..."	CONTINUE
Ambiguous	"I think maybe"	CONTINUE

VAD Pipeline Integration

Endpointing Thresholds

Setting	Value	Effect
Min speech duration	100ms	Ignore clicks/bangs
Max speech duration	30s	Force-stop long monologues
Silence threshold	500ms	Wait this long before ending
Pre-speech buffer	200ms	Include audio before VAD trigger
Post-speech buffer	300ms	Include trailing audio

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Evaluation, Key Terms, Common Pitfalls, Interview Framing — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.