Phase 1: Vision-Language Models · ~180 minutes · Python

LLaVA-OneVision: Single-Image, Multi-Image, Video in One Model

Before LLaVA-OneVision (Li et al., August 2024) the open-VLM world had separate lineages: LLaVA-1.5 for single images, multi-image models like Mantis and VILA, video models like Video-LLaVA and Video-LLaMA. Each won its benchmark and failed at the others. LLaVA-OneVision argued a single curriculum could train one model to dominate all three scenarios, and that the emergent task-transfer effects (single-image skills exported to video, multi-image reasoning exported to single-image) beat the sum of specialists. The recipe is deceptively simple: a visual-token budget that stays constant across scenarios, plus an explicit curriculum that moves from single-image to OneVision (multi-image) to video. This lesson reads the budget, the curriculum, and the emergent behaviors.

What you will learn

Design a visual-token budget that holds constant across single-image, multi-image, and video inputs.
Order a training curriculum that transfers skills from single-image to video without catastrophic forgetting.
Explain why a single model beats specialists at the same parameter count when curriculum is done right.
Name the three emergent capabilities reported by LLaVA-OneVision: multi-camera reasoning, set-of-mark prompting, iPhone-screenshot agent.

Introduction

Type: Build
Languages: Python (stdlib, token budget solver + curriculum planner)
Prerequisites: LLaVA and Visual Instruction Tuning, Any-Resolution Vision: Patch-n'-Pack and NaFlex
Time: ~180 minutes

Objective

Learning objectives

Design a visual-token budget that holds constant across single-image, multi-image, and video inputs.
Order a training curriculum that transfers skills from single-image to video without catastrophic forgetting.
Explain why a single model beats specialists at the same parameter count when curriculum is done right.
Name the three emergent capabilities reported by LLaVA-OneVision: multi-camera reasoning, set-of-mark prompting, iPhone-screenshot agent.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.