Phase 3: Vision-Language Model · ~90 minutes · Python

Cross-Attention Fusion

The projection layer aligns one image vector with one caption vector.

Hiring signal: Can build cross-attention fusion end to end

What you will learn

Implement multi-head cross-attention where the query stream is text and the key/value stream is vision.
Compose a decoder block: causal self-attention + cross-attention + feed-forward.
Get the mask shapes right: causal mask for self-attention, no mask for cross-attention.
Run a forward pass with batched text tokens and a fixed pool of image tokens.

Introduction

A real vision-language decoder needs every text token to attend to every patch token, so the model can ground each word in a region. Cross-attention is how that grounding happens. The text queries; the vision keys and values answer. This lesson builds the cross-attention block, the causal text self-attention, and the mask shapes that keep both legal.

Type: Build Languages: Python Prerequisites: Phase 19 lessons 30-37 (Track B foundations) Time: ~90 minutes

Objective

Learning objectives

Implement multi-head cross-attention where the query stream is text and the key/value stream is vision.
Compose a decoder block: causal self-attention + cross-attention + feed-forward.
Get the mask shapes right: causal mask for self-attention, no mask for cross-attention.
Run a forward pass with batched text tokens and a fixed pool of image tokens.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.

Key terms in this lesson

Token