Phase 6: Automatic Speech Recognition · ~75 minutes · Python

Real-Time Audio Processing

Batch pipelines process a file. Real-time pipelines process the next 20 milliseconds before the next 20 arrive. Every conversational AI, broadcast studio, and telephony bot lives and dies by this latency budget.

What you will learn

Implement real-time audio processing from scratch
Understand the math and intuition behind the algorithm
Use production libraries for the same task
Ship a reusable artifact

Introduction

Type: Build
Languages: Python
Prerequisites: Spectrograms, Mel Scale & Audio Features, Speech Recognition (ASR) — CTC, RNN-T, Attention, Text-to-Speech (TTS) — From Tacotron to F5 and Kokoro
Time: ~75 minutes

The Problem

You want a voice assistant that feels alive. Human conversational turn-taking latency is ~230 ms (silence-to-response). Anything above 500 ms feels robotic; above 1500 ms feels broken. The budget for a full hear → understand → respond → speak loop in 2026 is:

Stage	Budget
Mic → buffer	20 ms
VAD	10 ms
ASR (streaming)	150 ms
LLM (first token)	100 ms
TTS (first chunk)	100 ms
Render → speaker	20 ms
Total	~400 ms

Moshi (Kyutai, 2024) clocked 200 ms full-duplex. GPT-4o-realtime (2024) clocks ~320 ms. Cascaded pipelines in 2022 shipped at 2500 ms. The 10× improvement came from three techniques: (1) streaming everywhere, (2) asynchronous pipelining with partial results, (3) interruptible generation.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.