Phase 1: Generative Adversarial Networks · ~75 minutes · Python
Latent Diffusion & Stable Diffusion
Pixel-space diffusion on 512×512 images is a computational war crime. Rombach et al. (2022) noticed that you do not need all 786k dimensions to generate an image — you need enough to capture semantic structure, and a separate decoder for the rest. Run diffusion inside a VAE's latent space. That one idea is Stable Diffusion.
Hiring signal: Understanding of latent diffusion & stable diffusion internals
What you will learn
- Implement latent diffusion & stable diffusion from scratch
- Understand the math and intuition behind the algorithm
- Use production libraries for the same task
- Ship a reusable artifact
Introduction
Type: Build Languages: Python Prerequisites: Phase 8 · 02 (VAE), Phase 8 · 06 (DDPM), Phase 7 · 09 (ViT) Time: ~75 minutes
The Problem
Pixel-space diffusion at 512² means the U-Net runs on tensors of shape [B, 3, 512, 512]. Each sampling step is ~100 GFLOPS for a 500M-param U-Net. Fifty steps is 5 TFLOPS per image. Train on a billion images and the compute bill is absurd.
Most of those FLOPs go to pushing perceptually unimportant details through the net — the high-frequency texture that a lossy VAE could compress away. Rombach's idea: train a VAE once (the first stage), freeze it, and run diffusion entirely in the 4-channel 64×64 latent space (the second stage). Same U-Net. 1/16th the pixels. ~64x fewer FLOPs for comparable quality.
This is the Stable Diffusion recipe. SD 1.x / 2.x used an 860M U-Net over 64×64×4 latents, SDXL used a 2.6B U-Net over 128×128×4, SD3 swapped the U-Net for a Diffusion Transformer (DiT) with flow matching. Flux.1-dev (Black Forest Labs, 2024) ships a 12B-param DiT-MMDiT. All run on the same two-stage substrate.
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers The Concept, Architecture variants, Build It, Pitfalls, Use It, Ship It, Exercises, Key Terms, Production note: running Flux-12B on an 8GB consumer GPU, Further Reading — plus a hands-on lab, quiz, and project artifact.
Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.
Browse all courses · View pricing · DeVenture Academy