Phase 2: Audio-Visual Models · ~120 minutes · Python

Emu3: Next-Token Prediction for Image and Video Generation

BAAI's Emu3 (Wang et al., September 2024) is the 2024 result that should have ended the diffusion-versus-autoregressive debate. A single Llama-style decoder-only transformer, trained only on the next-token-prediction objective, across a unified vocabulary of text + VQ image tokens + 3D VQ video tokens, beats SDXL on image generation and LLaVA-1.6 on perception. No CLIP loss. No diffusion schedule. Classifier-free guidance is used at inference for quality, but the core training objective is next-token prediction with teacher forcing. Published in Nature. This lesson reads the Emu3 thesis — why a better tokenizer plus scale is all you need — and contrasts with diffusion approaches.

What you will learn

Explain why Emu3's single-loss next-token objective works despite the long-held assumption that diffusion is required for image quality.
Describe the 3D video tokenizer: what a spatiotemporal VQ codebook looks like, why patches span time.
Compare Emu3 vs Stable Diffusion XL on (training compute, inference cost, quality ceiling).
Name the three roles the same Emu3 model plays: Emu3-Gen (image gen), Emu3-Chat (perception), Emu3-Stage2 (video gen).

Introduction

Type: Learn
Languages: Python (stdlib, 3D video tokenizer math + autoregressive sampler skeleton)
Prerequisites: Chameleon and Early-Fusion Token-Only Multimodal Models
Time: ~120 minutes

Objective

Learning objectives

Explain why Emu3's single-loss next-token objective works despite the long-held assumption that diffusion is required for image quality.
Describe the 3D video tokenizer: what a spatiotemporal VQ codebook looks like, why patches span time.
Compare Emu3 vs Stable Diffusion XL on (training compute, inference cost, quality ceiling).
Name the three roles the same Emu3 model plays: Emu3-Gen (image gen), Emu3-Chat (perception), Emu3-Stage2 (video gen).

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.