Phase 8: Evaluation, Testing & Production · 55 min · Python · numpy · matplotlib

Latency Metrics: Glass-to-Glass, Per-Component, P50/P95/P99

If you can't measure P95 latency, you can't optimize it. Instrument everything.

Hiring signal: Latency measurement and optimization is the most common technical challenge in voice AI interviews.

What you will learn

Measure glass-to-glass latency: time from user speaking to agent responding
Break down latency per-component: ASR TTFT, LLM first-token, TTS TTFB, transport
Calculate P50, P95, P99 latency distributions
Build a latency profiling dashboard

The Problem

Users perceive voice agent latency as a single number: "how long until the agent responds?" But that number is the sum of many components — VAD, ASR, LLM, TTS, network, transport. To optimize, you must break it down and measure each piece. You also need percentiles (P50/P95/P99), not just averages, because outliers destroy user experience.

The Concept

Glass-to-Glass Latency

Glass-to-glass = from the moment sound hits the microphone glass to the moment sound leaves the speaker glass.

Component	Typical Latency	Optimizable?
Network (caller → server)	20-50ms	Use WebRTC
VAD	5-20ms	Use WebRTC VAD
Endpointing	200-500ms	Adaptive endpointing
ASR	100-200ms	Deepgram streaming
LLM TTFT	100-300ms	GPT-4o-mini, Groq
LLM streaming (first sentence)	200-500ms	Short responses
TTS TTFB	100-200ms	Cartesia Sonic
Network (server → caller)	20-50ms	Use WebRTC
Total glass-to-glass	500-1500ms	Target <500ms

Why is P95 latency more important than average latency for voice agents?

P95 captures the worst-case experience that real users encounter — if 5% of calls have 1.5s latency, that's 1 in 20 calls feeling broken, even if the average is 400ms

Per-Component Breakdown

class LatencyProfiler:
    def __init__(self):
        self.components = {}

    def record(self, component, latency_ms):
        """Record latency for a component."""
        if component not in self.components:
            self.components[component] = []
        self.components[component].append(latency_ms)

    def get_percentiles(self, component):
        """Get P50, P95, P99 for a component."""
        latencies = sorted(self.components[component])
        n = len(latencies)
        return {
            "p50": latencies[int(n * 0.50)],
            "p95": latencies[int(n * 0.95)],
            "p99": latencies[int(n * 0.99)],
            "avg": sum(latencies) / n,
        }

Percentile Calculation

Percentile	Meaning	Target
P50 (median)	50% of calls faster than this	<400ms
P95	95% of calls faster than this	<700ms
P99	99% of calls faster than this	<1000ms
P99.9	99.9% of calls faster than this	<1500ms

Latency Budget

LATENCY_BUDGET = {
    "network_in": 30,       # 30ms
    "vad": 10,              # 10ms
    "endpointing": 300,     # 300ms (silence detection)
    "asr": 150,             # 150ms
    "llm_ttft": 150,        # 150ms (time to first token)
    "llm_streaming": 200,   # 200ms (first sentence complete)
    "tts": 120,             # 120ms (time to first audio)
    "network_out": 30,      # 30ms
    "total": 990,           # 990ms — need to optimize
}

Optimization Strategies

Component	Optimization	Savings
Endpointing	Adaptive (reduce silence threshold)	-100ms
ASR	Deepgram streaming (partial results)	-50ms
LLM	GPT-4o-mini (faster TTFT)	-50ms
LLM	Short system prompt (fewer input tokens)	-20ms
TTS	Cartesia Sonic (streaming)	-30ms
Transport	WebRTC (vs WebSocket)	-30ms
Total savings		-280ms

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Evaluation, Key Terms, Common Pitfalls, Interview Framing — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.