Phase 2: Audio-Visual Models · ~180 minutes · Python
Transfusion: Autoregressive Text + Diffusion Image in One Transformer
Chameleon and Emu3 bet everything on discrete tokens. They work, but the quantization bottleneck is visible — the image quality plateaus below continuous-space diffusion models. Transfusion (Meta, Zhou et al., August 2024) takes the opposite bet: keep images continuous, drop the VQ-VAE entirely, and train one transformer with two losses. Text tokens get next-token-prediction. Image patches get a flow-matching / diffusion loss. Both objectives optimize the same weights. The architecture underlying Stable Diffusion 3 (MMDiT) is a close cousin. This lesson reads the Transfusion thesis, builds a toy two-loss trainer, and traces the attention mask that lets one transformer do both jobs.
Hiring signal: Understanding of transfusion: autoregressive text + diffusion image in one transformer internals
What you will learn
- Wire a transformer that runs two losses (NTP on text tokens, diffusion MSE on image patches) on one backbone.
- Explain why bidirectional attention across image patches plus causal attention over text tokens is the right mask choice.
- Compare Transfusion-style (continuous images, diffusion loss) to Chameleon-style (discrete images, NTP) on compute, quality, and code complexity.
- Name MMDiT's contribution: modality-specific weights at each block, joint attention at the residual stream.
Introduction
Type: Build Languages: Python (stdlib, two-loss trainer on MNIST-scale toy) Prerequisites: Phase 12 · 11 (Chameleon), Phase 8 (Generative AI) Time: ~180 minutes
Learning Objectives
- Wire a transformer that runs two losses (NTP on text tokens, diffusion MSE on image patches) on one backbone.
- Explain why bidirectional attention across image patches plus causal attention over text tokens is the right mask choice.
- Compare Transfusion-style (continuous images, diffusion loss) to Chameleon-style (discrete images, NTP) on compute, quality, and code complexity.
- Name MMDiT's contribution: modality-specific weights at each block, joint attention at the residual stream.
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Use It, Ship It, Exercises, Key Terms, Further Reading — plus a hands-on lab, quiz, and project artifact.
Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.
Browse all courses · View pricing · DeVenture Academy