Phase 1: Diffusion Model Fundamentals · 50 min · Python · Hugging Face Diffusers · Civitai

Conditioning & Model File Formats

Checkpoints, LoRAs, VAEs, text encoders — knowing what each file does is the difference between downloading models blindly and engineering with them.

Hiring signal: Model file format fluency (checkpoints, LoRAs, VAEs, text encoders) is a practical skill that shows you've worked with production generative media systems, not just APIs.

What you will learn

Explain text conditioning: CLIP/T5 text encoders converting prompts to embeddings
Describe image conditioning: ControlNet and IP-Adapter for structural control
Navigate model file formats: checkpoints (.safetensors), LoRA adapters, VAEs, text encoders
Explain what each component does and how they combine in a generation pipeline

The Problem

A developer downloads a Stable Diffusion model from Civitai. It's a 6.4GB .safetensors file. They load it and generate images. Then they download a "LoRA" that's 150MB. They load it on top and the style changes. But they don't know what's inside these files, what each component does, or how they combine. When something breaks — wrong colors, distorted faces, style not applying — they can't debug because they don't understand the components.

Model file format fluency is a practical skill that shows you've worked with production generative media systems, not just APIs.

What you'll build
Load a base SDXL model, add a LoRA adapter, swap the VAE, and generate images with and without each component to demonstrate their effects. Document the file sizes, formats, and roles of each component.

Text Conditioning: CLIP and T5

Text conditioning is how the model "understands" your prompt. The text is encoded into embeddings that guide the diffusion process.

CLIP (Contrastive Language-Image Pre-training)

Used in Stable Diffusion 1.5 and SDXL.

SD 1.5: CLIP ViT-L/14 (768-dim embeddings)
SDXL: Dual CLIP — ViT-L/14 + ViT-bigG/14 (1280-dim embeddings, open_clip)

# SDXL uses two CLIP text encoders
from transformers import CLIPTextModel, CLIPTokenizer

tokenizer1 = CLIPTokenizer.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", subfolder="tokenizer")
text_encoder1 = CLIPTextModel.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", subfolder="text_encoder")

tokenizer2 = CLIPTokenizer.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", subfolder="tokenizer_2")
text_encoder2 = CLIPTextModel.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", subfolder="text_encoder_2")

# Encode prompt
tokens = tokenizer1("a cat on a windowsill", return_tensors="pt", padding="max_length", max_length=77)
embeddings = text_encoder1(tokens.input_ids).last_hidden_state
# embeddings shape: (1, 77, 768) — 77 tokens, 768-dim each

T5 (Text-to-Text Transformer)

Used in FLUX, Imagen, and Stable Diffusion 3.

FLUX.1: T5-XXL (4096-dim embeddings, 256 tokens)
SD 3.5: T5-XXL + CLIP (triple encoder)

T5 provides much longer context (256 tokens vs CLIP's 77) and better understanding of complex prompts. This is why FLUX handles detailed prompts better than SDXL.

Why does FLUX handle complex, detailed prompts better than SDXL?

FLUX uses T5-XXL for text encoding, which provides 4096-dim embeddings and 256 token context length. SDXL uses CLIP with 768/1280-dim embeddings and only 77 tokens. T5 is a language model trained on text-to-text tasks, giving it much better understanding of complex prompts. The 256 token limit means FLUX can process ~3x longer prompts than SDXL.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Image Conditioning: ControlNet and IP-Adapter, Model File Formats, Component Combination Order, File Format Summary, Key Takeaways, What's Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.

Key terms in this lesson

LoRA (Low-Rank Adaptation)