Phase 0: Multimodal Foundations · ~180 minutes · Python

CLIP and Contrastive Vision-Language Pretraining

OpenAI's CLIP (2021) proved a single idea big enough to power the next five years: align an image encoder and a text encoder in the same vector space using only noisy web image-caption pairs and a contrastive loss. Zero supervised labels. 400M pairs. The resulting embedding space does zero-shot classification, image-text retrieval, and plugs into every 2026 VLM as its vision tower. SigLIP 2 (2025) replaced softmax with sigmoid and scaled past CLIP at lower cost. This lesson walks the math from InfoNCE to sigmoid pairwise loss and builds the training step in stdlib Python.

Hiring signal: Understanding of clip and contrastive vision-language pretraining internals

What you will learn

Derive InfoNCE loss from mutual information and implement a numerically-stable vectorized version.
Explain why sigmoid pairwise loss (SigLIP) scales to batch 32768+ without the all-gather overhead softmax demands.
Run zero-shot ImageNet classification by constructing text templates (`a photo of a {class}`) and taking argmax over cosine similarity.
Name the four levers CLIP / SigLIP pretraining gives you: batch size, temperature, prompt template, data quality.

Introduction

Type: Build Languages: Python (stdlib, InfoNCE + sigmoid loss implementations) Prerequisites: Phase 12 · 01 (ViT patches), Phase 7 (Transformers) Time: ~180 minutes

Learning Objectives

Derive InfoNCE loss from mutual information and implement a numerically-stable vectorized version.
Explain why sigmoid pairwise loss (SigLIP) scales to batch 32768+ without the all-gather overhead softmax demands.
Run zero-shot ImageNet classification by constructing text templates (a photo of a {class}) and taking argmax over cosine similarity.
Name the four levers CLIP / SigLIP pretraining gives you: batch size, temperature, prompt template, data quality.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Use It, Ship It, Exercises, Key Terms, Further Reading — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.