Phase 5: Turn-Taking & Conversation Management · 50 min · Python · LiveKit Agents SDK · Pipecat

Endpointing: Silence-Based vs Semantic

Knowing when the user is done speaking is the hardest problem in voice AI. Semantic endpointing is the solution.

Hiring signal: Endpointing strategy selection and tuning is a senior voice AI engineering skill.

What you will learn

Compare silence-based vs semantic endpointing approaches
Implement semantic endpointing with LiveKit transformer-based turn detector
Use Pipecat SmartTurnDetection with configurable sensitivity
Tune endpointing for different conversation contexts

The Problem

After VAD detects speech, the agent must decide when the caller has finished their turn. Too early = interruption. Too late = awkward silence. This is endpointing — the hardest timing problem in voice AI.

The Concept

Endpointing Methods

Method	How	Latency	Accuracy	Complexity
Fixed silence	Wait N ms of silence	N ms	Low	Simple
Adaptive silence	Adjust N based on speech rate	N ms	Medium	Medium
Semantic endpointing	LLM checks if thought is complete	~200ms	High	Complex
Hybrid	Silence + semantic check	N + 200ms	Best	Complex

Fixed Silence Endpointing

class FixedSilenceEndpointer:
    def __init__(self, silence_threshold_ms=500):
        self.silence_threshold_ms = silence_threshold_ms
        self.last_speech_time = None

    def check(self, is_speech, current_time):
        """Check if endpoint is reached."""
        if is_speech:
            self.last_speech_time = current_time
            return False  # Still speaking

        if self.last_speech_time is None:
            return False  # No speech yet

        silence_duration = current_time - self.last_speech_time
        return silence_duration >= self.silence_threshold_ms

Threshold	Experience	Use Case
200ms	Snappy but interrupts	Fast-paced Q&A
500ms	Natural	General purpose
700ms	Slight pause	Thoughtful conversations
1000ms	Noticeable delay	Deliberate speaking

A caller says "I want to book a flight to... hmm... Tokyo" with a 600ms pause before "Tokyo". With a 500ms silence threshold, what happens?

The agent interrupts

Semantic Endpointing

async def semantic_endpoint(partial_transcript, silence_duration_ms):
    """Use LLM to check if the caller is done speaking."""
    # Only check after initial silence threshold
    if silence_duration_ms < 300:
        return False  # Too early to check

    # Ask LLM if this is a complete thought
    is_complete = await semantic_vad.is_complete(partial_transcript)

    if is_complete:
        return True  # Complete thought + silence = endpoint

    # Incomplete but silence > 1s = probably done anyway
    if silence_duration_ms > 1000:
        return True  # Force endpoint after long silence

    return False  # Wait for more speech

Hybrid Endpointing

The best approach combines fixed silence with semantic checking:

Adaptive Silence

class AdaptiveEndpointer:
    """Adjust silence threshold based on speech patterns."""
    def __init__(self, base_threshold=500):
        self.base_threshold = base_threshold
        self.speech_rates = []

    def get_threshold(self, recent_speech_duration):
        """Adjust threshold based on how fast the caller speaks."""
        self.speech_rates.append(recent_speech_duration)

        if len(self.speech_rates) < 3:
            return self.base_threshold

        avg_rate = np.mean(self.speech_rates[-5:])
        if avg_rate < 2.0:  # Fast speaker
            return 300  # Shorter threshold
        elif avg_rate > 5.0:  # Slow speaker
            return 700  # Longer threshold
        return self.base_threshold

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Evaluation, Key Terms, Common Pitfalls, Interview Framing — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.