Phase 6: Automatic Speech Recognition · ~120 minutes · Python

Build a Voice Assistant Pipeline — The Phase 6 Capstone

Everything from lessons 01-11, stitched together. Build a voice assistant that listens, reasons, and talks back. In 2026 that is a solved engineering problem, not a research problem — but the integration details decide whether it ships.

Hiring signal: Understanding of build a voice assistant pipeline — the phase 6 capstone internals

What you will learn

Implement build a voice assistant pipeline — the phase 6 capstone from scratch
Understand the math and intuition behind the algorithm
Use production libraries for the same task
Ship a reusable artifact

Introduction

Type: Build Languages: Python Prerequisites: Phase 6 · 04, 05, 06, 07, 11; Phase 11 · 09 (Function Calling); Phase 14 · 01 (Agent Loop) Time: ~120 minutes

The Problem

Build an end-to-end assistant:

Captures mic input (16 kHz mono).
Detects start/end of user speech.
Transcribes streaming.
Passes transcript to an LLM that can call tools (timer, weather, calendar).
Streams LLM text to a TTS.
Plays audio back to the user.
Stops if the user interrupts mid-response.

Latency target: first TTS audio byte within 800 ms of the user finishing their utterance on a laptop CPU. Quality target: no missed words, no hallucinated subtitles on silence, no voice cloning leakage, no prompt injection success.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Concept, Build It, Use It, Pitfalls, Ship It, Exercises, Key Terms, Further Reading — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.