Phase 1: Diffusion Model Fundamentals · 50 min · Python · PyTorch · Hugging Face Diffusers

Latent Diffusion & VAEs

Latent diffusion is why Stable Diffusion runs on a consumer GPU instead of a data center — the VAE compresses images to a fraction of their size.

Hiring signal: Understanding latent diffusion and VAEs is a core generative media engineering concept — interviewers test whether you know why diffusion runs in latent space, not pixel space.

What you will learn

Explain why pixel-space diffusion is too expensive (memory and compute)
Describe how VAEs compress images to latent representations (e.g., 512x512 → 64x64 latent)
Explain the latent diffusion process: running diffusion in compressed latent space
Describe the VAE decode step: converting latents back to pixels

The Problem

Diffusion in pixel space is expensive. A 512x512 RGB image has 512 × 512 × 3 = 786,432 values. Running 1000 diffusion steps on that many values requires enormous compute and memory. This is why early diffusion models needed data center GPUs.

Then in 2022, Rombach et al. published the Latent Diffusion paper. The insight was simple: don't diffuse in pixel space — compress the image first, then diffuse in the compressed latent space. This is why Stable Diffusion runs on a consumer GPU with 8GB VRAM instead of a data center A100.

What you'll build
Load a VAE from a Stable Diffusion model, encode an image to latent space, visualize the latent representation, decode it back, and measure the compression ratio. Then interpolate between two image latents to create a smooth transition.

The VAE: Variational Autoencoder

The VAE (Variational Autoencoder) is the compression model. It has two parts:

Encoder:  Image (512x512x3) → Latent (64x64x4)
Decoder:  Latent (64x64x4) → Image (512x512x3)

The encoder compresses the image by a factor of 8 in each spatial dimension (512→64) and reduces channels from 3 to 4. The total compression:

Pixel space:  512 × 512 × 3 = 786,432 values
Latent space: 64 × 64 × 4  = 16,384 values
Compression:  786,432 / 16,384 = 48x

This 48x compression means diffusion runs on 16,384 values instead of 786,432 — a massive reduction in compute and memory.

from diffusers import AutoencoderKL
import torch

# Load the VAE from Stable Diffusion
vae = AutoencoderKL.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", subfolder="vae")
vae.to("cuda")

# Encode: image → latent
image_tensor = torch.randn(1, 3, 512, 512).cuda()  # (B, C, H, W)
latent = vae.encode(image_tensor).latent_dist.sample()
# latent shape: (1, 4, 64, 64) — 48x compressed

# Decode: latent → image
reconstructed = vae.decode(latent).sample
# reconstructed shape: (1, 3, 512, 512) — back to pixel space

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Why Latent Diffusion Works, The Full Latent Diffusion Pipeline, VAE Quality Matters, Latent Space Interpolation, FLUX and Latent Diffusion, Key Takeaways, What's Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.