Phase 1: Diffusion Model Fundamentals · 45 min · Python · PyTorch

UNet vs DiT Architectures

Diffusion Transformers (DiTs) power FLUX, Sora, and Veo — UNets power SDXL. Knowing the difference is architecture literacy.

Hiring signal: Architecture knowledge (UNet vs DiT) demonstrates you understand the model layer, not just the API layer — a key differentiator in generative media interviews.

What you will learn

Describe UNet architecture: encoder-decoder with skip connections and attention mechanisms
Describe Diffusion Transformers (DiTs): transformer-based denoising replacing UNets
Explain why DiTs scale better than UNets: parallelism, training efficiency
Identify which models use which architecture: FLUX (DiT), Sora (DiT), Veo (DiT), SDXL (UNet)

The Problem

A team is choosing between SDXL and FLUX for their image generation pipeline. They look at sample outputs and pick FLUX because "it looks better." But they can't explain why FLUX is better — they don't know it uses a Diffusion Transformer while SDXL uses a UNet. When the interviewer asks "why did you choose FLUX over SDXL?", "it looks better" is not a sufficient answer.

Understanding the architecture behind generative models is what separates engineers from API callers. You don't need to implement these architectures from scratch, but you need to know what they are, how they differ, and why one scales better than the other.

What you'll build
Load both a UNet-based model (SDXL) and a DiT-based model (FLUX.1 Schnell), generate images with both, measure inference time, and compare output quality. Write a comparison report documenting the architecture differences and practical trade-offs.

UNet Architecture

The UNet was the original architecture for diffusion models, used in DDPM, Stable Diffusion 1.x, and SDXL.

Structure: Encoder-decoder with skip connections

Encoder (downsampling)          Decoder (upsampling)
[64x64] ──→ [32x32] ──→ [16x16] ──→ [8x8] (bottleneck)
  │             │            │
  │skip         │skip        │skip
  │             │            │
[64x64] ←── [32x32] ←── [16x16] ←── [8x8]

Key features:

Skip connections: connect encoder layers to corresponding decoder layers, preserving spatial detail
Attention mechanisms: self-attention at lower resolutions (16x16, 8x8) for global context
Convolutional: primarily conv-based, with attention layers at key points
Resolution processing: processes the image at multiple resolutions through down/up sampling

Models using UNet:

Stable Diffusion 1.5 (256x256 / 512x512)
Stable Diffusion XL (1024x1024)
DeepFloyd IF

# SDXL uses a UNet
from diffusers import StableDiffusionXLPipeline
pipe = StableDiffusionXLPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0")
# pipe.unet is a UNet2DConditionModel

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Diffusion Transformers (DiTs), Why DiTs Scale Better, Practical Comparison, When to Use Each, The Hybrid Approach, Key Takeaways, What's Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.