Phase 3: Generative Vision Models · ~45 minutes · Python

Open-Vocabulary Vision — CLIP

Train an image encoder and a text encoder together so that matching (image, caption) pairs land at the same point in a shared space. That is the whole trick.

What you will learn

Explain CLIP's two-tower architecture and contrastive training objective
Use a pretrained CLIP (or SigLIP) for zero-shot classification without any task-specific training
Implement zero-shot classification from scratch: encode class prompts, compute cosine similarity, take argmax
Distinguish CLIP, SigLIP, OpenCLIP, and LLaVA/LLaMA-vision models — what each is for in 2026

Introduction

Type: Build + Use
Languages: Python
Prerequisites: Vision Transformers (ViT), Self-Supervised Vision — SimCLR, DINO, MAE
Time: ~45 minutes

Objective

Learning objectives

Explain CLIP's two-tower architecture and contrastive training objective
Use a pretrained CLIP (or SigLIP) for zero-shot classification without any task-specific training
Implement zero-shot classification from scratch: encode class prompts, compute cosine similarity, take argmax
Distinguish CLIP, SigLIP, OpenCLIP, and LLaVA/LLaMA-vision models — what each is for in 2026

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.