Phase 3: Vision-Language Model · ~90 minutes · Python

Projection Layer for Modality Alignment

A vision encoder produces image tokens.

Hiring signal: Can build projection layer for modality alignment end to end

What you will learn

Build a two-layer MLP projection that maps image features into the text embedding space.
Construct a mock text embedding table (no pretrained tokenizer, no real corpus).
Compute a cosine alignment loss between projected image tokens and a paired caption embedding.
Train the projection alone with a frozen vision encoder and a frozen text table.

Introduction

A text decoder consumes text tokens. The two live in different vector spaces. A small two-layer MLP projects image tokens into the text embedding space, and a cosine alignment loss against a paired caption pulls the two spaces into agreement. That projection is the smallest piece of a vision-language model and the one that matters most for transfer.

Type: Build Languages: Python Prerequisites: Phase 19 lessons 30-37 (Track B foundations) Time: ~90 minutes

Objective

Learning objectives

Build a two-layer MLP projection that maps image features into the text embedding space.
Construct a mock text embedding table (no pretrained tokenizer, no real corpus).
Compute a cosine alignment loss between projected image tokens and a paired caption embedding.
Train the projection alone with a frozen vision encoder and a frozen text table.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.

Key terms in this lesson

Embedding