Phase 3: Multimodal Generation · ~180 minutes · Python

Long-Video Understanding at Million-Token Context

A 1-hour 4K video at 24 FPS, patched and embedded, produces on the order of 60 million tokens. A 2-hour podcast episode transcribed is 30,000 tokens. A full Blu-ray feature film, even compressed with aggressive pooling, is hundreds of thousands of tokens. Google's Gemini 1.5 (March 2024) opened this era with a 10-million-token context, doing reliable needle-in-a-haystack recall over hour-long videos. LWM (Liu et al., February 2024) showed ring attention's scaling path. LongVILA and Video-XL scaled ingestion further. VideoAgent swapped raw context for agentic retrieval. Each approach is a different trade-off on compute, recall, and engineering complexity. This lesson reads them side by side.

What you will learn

Compute total visual-token counts for long-form video at varying FPS and pooling.
Explain the three scaling paths: brute context (Gemini 1.5), ring attention (LWM), token compression (LongVILA / Video-XL).
Compare raw-context video VLMs vs agentic-retrieval video VLMs (VideoAgent) on accuracy and latency.
Design a needle-in-a-haystack test for a 30-minute video and measure recall at a specific minute.

Introduction

Type: Build
Languages: Python (stdlib, needle-in-haystack simulator + agentic-retrieval router)
Prerequisites: Video-Language Models: Temporal Tokens and Grounding
Time: ~180 minutes

Objective

Learning objectives

Compute total visual-token counts for long-form video at varying FPS and pooling.
Explain the three scaling paths: brute context (Gemini 1.5), ring attention (LWM), token compression (LongVILA / Video-XL).
Compare raw-context video VLMs vs agentic-retrieval video VLMs (VideoAgent) on accuracy and latency.
Design a needle-in-a-haystack test for a 30-minute video and measure recall at a specific minute.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.