Phase 0: Multimodal Foundations · ~120 minutes · Python

Flamingo and Gated Cross-Attention for Few-Shot VLMs

DeepMind's Flamingo (2022) did two things before anyone else. It showed a single model could process arbitrarily interleaved sequences of images, videos, and text. And it showed VLMs could learn in-context — give a few-shot prompt with three example (image, caption) pairs and the model captions a new image without any gradient step. The mechanism: gated cross-attention layers, inserted between the frozen LLM's existing layers, with a learned tanh gate that starts at zero so the LLM's text capability is preserved at initialization. This lesson walks Flamingo's Perceiver resampler and gated cross-attention architecture — the ancestor of Gemini's interleaved inputs and Idefics2's visual tokens.

What you will learn

Explain how gated cross-attention preserves a frozen LLM's text capability at initialization via tanh(gate) = 0.
Walk through a Perceiver resampler: N image patches → K fixed "latent" queries via cross-attention.
Describe how Flamingo handles interleaved image-text sequences with causal masking that respects image placement.
Reproduce a few-shot multimodal prompt structure (3 image-caption examples then a query image).

Introduction

Type: Learn
Languages: Python (stdlib, gated cross-attention + Perceiver resampler demo)
Prerequisites: From CLIP to BLIP-2 — Q-Former as Modality Bridge
Time: ~120 minutes

Objective

Learning objectives

Explain how gated cross-attention preserves a frozen LLM's text capability at initialization via tanh(gate) = 0.
Walk through a Perceiver resampler: N image patches → K fixed "latent" queries via cross-attention.
Describe how Flamingo handles interleaved image-text sequences with causal masking that respects image placement.
Reproduce a few-shot multimodal prompt structure (3 image-caption examples then a query image).

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.