Phase 0: Multimodal Foundations · ~120 minutes · Python

Flamingo and Gated Cross-Attention for Few-Shot VLMs

DeepMind's Flamingo (2022) did two things before anyone else. It showed a single model could process arbitrarily interleaved sequences of images, videos, and text. And it showed VLMs could learn in-context — give a few-shot prompt with three example (image, caption) pairs and the model captions a new image without any gradient step. The mechanism: gated cross-attention layers, inserted between the frozen LLM's existing layers, with a learned tanh gate that starts at zero so the LLM's text capability is preserved at initialization. This lesson walks Flamingo's Perceiver resampler and gated cross-attention architecture — the ancestor of Gemini's interleaved inputs and Idefics2's visual tokens.

Hiring signal: Understanding of flamingo and gated cross-attention for few-shot vlms internals

What you will learn

Explain how gated cross-attention preserves a frozen LLM's text capability at initialization via tanh(gate) = 0.
Walk through a Perceiver resampler: N image patches → K fixed "latent" queries via cross-attention.
Describe how Flamingo handles interleaved image-text sequences with causal masking that respects image placement.
Reproduce a few-shot multimodal prompt structure (3 image-caption examples then a query image).

Introduction

Type: Learn Languages: Python (stdlib, gated cross-attention + Perceiver resampler demo) Prerequisites: Phase 12 · 03 (BLIP-2 Q-Former) Time: ~120 minutes

Learning Objectives

Explain how gated cross-attention preserves a frozen LLM's text capability at initialization via tanh(gate) = 0.
Walk through a Perceiver resampler: N image patches → K fixed "latent" queries via cross-attention.
Describe how Flamingo handles interleaved image-text sequences with causal masking that respects image placement.
Reproduce a few-shot multimodal prompt structure (3 image-caption examples then a query image).

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Use It, Ship It, Exercises, Key Terms, Further Reading — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.