Phase 2: Speech-to-Text (ASR) · 50 min · Python

How ASR Works: Acoustic Models, CTC vs RNN-T vs Attention

You don't need to train ASR models, but you must understand how they work to choose and debug them.

Hiring signal: ASR architecture knowledge distinguishes speech AI engineers from generalist AI engineers.

What you will learn

Explain acoustic models and how they convert audio frames to phoneme probabilities
Compare CTC, RNN-T, and attention-based ASR architectures
Understand why RNN-T is preferred for streaming ASR
Relate ASR architecture choices to latency and accuracy trade-offs

The Problem

Every voice agent starts with ASR — converting the caller's speech to text. But ASR isn't a black box. The architecture choice (CTC, RNN-T, or attention) determines your latency profile, accuracy, and streaming capability. Engineers who understand these trade-offs make better provider choices and debug ASR errors more effectively.

The Concept

The ASR Pipeline

Feature Extraction

Raw audio → features the neural network can process:

Feature	Description	Dimensions
MFCC	Mel-Frequency Cepstral Coefficients	~13-40 coefficients per frame
Mel-spectrogram	Power spectrum on mel scale	80-128 mel bins per frame
Waveform	Raw samples (end-to-end models)	160-960 samples per frame

Modern ASR (Whisper, Nova-3) uses mel-spectrograms — 80-128 dimensional features extracted every 10ms.

Acoustic Models

The acoustic model is a neural network that converts audio features into phoneme or character probabilities:

Model Type	Architecture	Streaming	Example
CTC	LSTM/Transformer + CTC loss	Yes	Deepgram Nova-3
RNN-T	Encoder + Predictor + Joiner	Yes (natural)	Google USM, Amazon
Attention	Encoder-Decoder + Attention	No (batch)	Whisper, original Transformer

CTC (Connectionist Temporal Classification)

CTC allows the model to output one token per audio frame without knowing the alignment. It introduces a "blank" token that represents "no output this frame."

Why does CTC enable streaming ASR?

It uses smaller models

RNN-T (Recurrent Neural Network Transducer)

RNN-T has three components:

Encoder: Processes audio frames
Predictor: Processes previously output tokens
Joiner: Combines encoder and predictor outputs

RNN-T is the gold standard for streaming ASR because:

Natural streaming (no blank token hack)
Better accuracy than CTC
Lower latency than attention
Handles variable-length sequences natively

Attention-Based (Whisper)

Whisper uses an encoder-decoder Transformer with attention:

Encoder: Processes the full audio sequence
Decoder: Generates text tokens auto-regressively with cross-attention to encoder

This gives the best accuracy but requires the full audio before decoding starts — making it batch-only.

Architecture Comparison

Feature	CTC	RNN-T	Attention
Streaming	Yes (with blank token)	Yes (natural)	No (batch only)
Accuracy	Good	Very Good	Best
Latency	Low (~100ms)	Low (~100ms)	High (full audio)
Training complexity	Simple	Complex	Medium
Partial transcripts	Yes	Yes	No
Example	Deepgram Nova-3	Google USM	Whisper

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Evaluation, Key Terms, Common Pitfalls, Interview Framing — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.