Phase 0: Multimodal Foundations · ~120 minutes · Python

Vision Transformers and the Patch-Token Primitive

Before anything multimodal, an image has to become a sequence of tokens a transformer can eat. The 2020 ViT paper answered this with 16x16 pixel patches, a linear projection, and a position embedding. Five years later every 2026 frontier model (Claude Opus 4.7 at 2576px native, Gemini 3.1 Pro, Qwen3.5-Omni) still begins this way — the encoder changed from ViT to DINOv2 to SigLIP 2, register tokens were added, the positional scheme became 2D-RoPE, but the primitive held. This lesson reads the patch-token pipeline end to end and builds it in stdlib Python so the rest of Phase 12 has a concrete mental model for "visual tokens."

What you will learn

Convert an HxWx3 image into a sequence of patch tokens with correct positional encoding.
Compute sequence length, parameter count, and FLOPs for a ViT of a given (patch size, resolution, hidden dim, depth).
Name the three upgrades that took ViT from 2020 research to 2026 production: self-supervised pretraining (DINO / MAE), register tokens, and native-resolution packing.
Pick between CLS pooling, mean pooling, and register tokens for a downstream task.

Introduction

Type: Learn
Languages: Python (stdlib, patch tokenizer + geometry calculator)
Prerequisites: the Transformers & LLMs from Scratch course, the Computer Vision Engineering course
Time: ~120 minutes

Objective

Learning objectives

Convert an HxWx3 image into a sequence of patch tokens with correct positional encoding.
Compute sequence length, parameter count, and FLOPs for a ViT of a given (patch size, resolution, hidden dim, depth).
Name the three upgrades that took ViT from 2020 research to 2026 production: self-supervised pretraining (DINO / MAE), register tokens, and native-resolution packing.
Pick between CLS pooling, mean pooling, and register tokens for a downstream task.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.