Phase 6: Automatic Speech Recognition · ~120 minutes · Python

Build a Voice Assistant Pipeline (Capstone)

Everything from lessons 01-11, stitched together. Build a voice assistant that listens, reasons, and talks back. In 2026 that is a solved engineering problem, not a research problem — but the integration details decide whether it ships.

What you will learn

Implement build a voice assistant pipeline — the phase 6 capstone from scratch
Understand the math and intuition behind the algorithm
Use production libraries for the same task
Ship a reusable artifact

Introduction

Type: Build
Languages: Python
Prerequisites: Speech Recognition (ASR) — CTC, RNN-T, Attention
Time: ~120 minutes

The Problem

Build an end-to-end assistant:

Captures mic input (16 kHz mono).
Detects start/end of user speech.
Transcribes streaming.
Passes transcript to an LLM that can call tools (timer, weather, calendar).
Streams LLM text to a TTS.
Plays audio back to the user.
Stops if the user interrupts mid-response.

Latency target: first TTS audio byte within 800 ms of the user finishing their utterance on a laptop CPU. Quality target: no missed words, no hallucinated subtitles on silence, no voice cloning leakage, no prompt injection success.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.

Key terms in this lesson

Streaming