Phase 3: Voice, Video, Safety & Tutoring · 30 hours · Python (agent + pipeline) · TypeScript (web client)

Capstone 03 — Real-Time Voice Assistant (ASR to LLM to TTS)

A voice agent that feels right has end-to-end latency under 800ms, knows when you have stopped talking, handles barge-in, and can call a tool without stalling.

Hiring signal: Can build capstone 03 end to end

Introduction

Retell, Vapi, LiveKit Agents, and Pipecat all hit this bar in 2026. They do it with the same shape: a streaming ASR, a turn-detector, a streaming LLM, and a streaming TTS, all wired through WebRTC with aggressive latency budgets at every hop. Build one, measure WER and MOS and false-cutoff rate, and run it under packet loss.

Phases exercised: P6 · P7 · P11 · P13 · P14 · P17

Type: Capstone Languages: Python (agent + pipeline), TypeScript (web client) Prerequisites: Phase 6 (speech and audio), Phase 7 (transformers), Phase 11 (LLM engineering), Phase 13 (tools), Phase 14 (agents), Phase 17 (infrastructure) Time: 30 hours

Objective

Learning objectives

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.