Phase 1: Vision-Language Models · ~180 minutes · Python
LLaVA-OneVision: Single-Image, Multi-Image, Video in One Model
Before LLaVA-OneVision (Li et al., August 2024) the open-VLM world had separate lineages: LLaVA-1.5 for single images, multi-image models like Mantis and VILA, video models like Video-LLaVA and Video-LLaMA. Each won its benchmark and failed at the others. LLaVA-OneVision argued a single curriculum could train one model to dominate all three scenarios, and that the emergent task-transfer effects (single-image skills exported to video, multi-image reasoning exported to single-image) beat the sum of specialists. The recipe is deceptively simple: a visual-token budget that stays constant across scenarios, plus an explicit curriculum that moves from single-image to OneVision (multi-image) to video. This lesson reads the budget, the curriculum, and the emergent behaviors.
Hiring signal: Understanding of llava-onevision: single-image, multi-image, video in one model internals
What you will learn
- Design a visual-token budget that holds constant across single-image, multi-image, and video inputs.
- Order a training curriculum that transfers skills from single-image to video without catastrophic forgetting.
- Explain why a single model beats specialists at the same parameter count when curriculum is done right.
- Name the three emergent capabilities reported by LLaVA-OneVision: multi-camera reasoning, set-of-mark prompting, iPhone-screenshot agent.
Introduction
Type: Build Languages: Python (stdlib, token budget solver + curriculum planner) Prerequisites: Phase 12 · 05 (LLaVA), Phase 12 · 06 (any-resolution) Time: ~180 minutes
Learning Objectives
- Design a visual-token budget that holds constant across single-image, multi-image, and video inputs.
- Order a training curriculum that transfers skills from single-image to video without catastrophic forgetting.
- Explain why a single model beats specialists at the same parameter count when curriculum is done right.
- Name the three emergent capabilities reported by LLaVA-OneVision: multi-camera reasoning, set-of-mark prompting, iPhone-screenshot agent.
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Use It, Ship It, Exercises, Key Terms, Further Reading — plus a hands-on lab, quiz, and project artifact.
Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.
Browse all courses · View pricing · DeVenture Academy