Phase 3: Multimodal Generation · ~180 minutes · Python

Audio-Language Models: the Whisper to Audio Flamingo 3 Arc

Whisper (Radford et al., December 2022) settled speech recognition — 680k hours of weakly-supervised multilingual speech, a simple encoder-decoder transformer, a benchmark that made every subsequent ASR release cite it. But recognition is not reasoning. Asking "what instruments are in this recording" or "what emotion is the speaker expressing" or "what happened at minute 3" requires audio understanding, not transcription. Qwen-Audio, SALMONN, LTU, and NVIDIA's Audio Flamingo 3 (AF3, July 2025) progressively built that stack: keep Whisper-class encoders, bolt on Q-formers, train on audio-text instruction data, add chain-of-thought reasoning. This lesson walks the arc.

What you will learn

Compute a log-Mel spectrogram from a waveform: windowing, FFT, filter banks, log transform.
Compare encoder options: Whisper encoder, BEATs, AF-Whisper hybrid. When each wins.
Build an audio Q-former: N learnable queries cross-attending to spectrogram patches.
Explain cascaded (Whisper-then-LLM) vs end-to-end audio-LLM training: why end-to-end scales better for reasoning.

Introduction

Type: Build
Languages: Python (stdlib, log-Mel spectrogram + audio Q-former skeleton)
Prerequisites: the NLP & Speech Processing course, From CLIP to BLIP-2 — Q-Former as Modality Bridge
Time: ~180 minutes

Objective

Learning objectives

Compute a log-Mel spectrogram from a waveform: windowing, FFT, filter banks, log transform.
Compare encoder options: Whisper encoder, BEATs, AF-Whisper hybrid. When each wins.
Build an audio Q-former: N learnable queries cross-attending to spectrogram patches.
Explain cascaded (Whisper-then-LLM) vs end-to-end audio-LLM training: why end-to-end scales better for reasoning.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.