Phase 0: Multimodal Foundations · ~180 minutes · Python

LLaVA and Visual Instruction Tuning

LLaVA (April 2023) is the most copied multimodal architecture on the planet. It replaced BLIP-2's Q-Former with a 2-layer MLP, replaced Flamingo's gated cross-attention with naive token concatenation, and trained on 158k visual-instruction turns generated by GPT-4 from text-only captions. Any practitioner who built a VLM between 2023 and 2026 built some variant of LLaVA. LLaVA-1.5 added AnyRes. LLaVA-NeXT bumped resolution. LLaVA-OneVision unified image, multi-image, and video in one recipe. This lesson reads the recipe, implements the projector, and explains why "simpler won."

Hiring signal: Understanding of llava and visual instruction tuning internals

What you will learn

Build a 2-layer MLP projector that maps ViT patch embeddings (dim 1024) to an LLM's embedding dim (dim 4096).
Walk the LLaVA two-stage recipe: (1) projector alignment on 558k caption pairs, (2) visual instruction tuning on 158k GPT-4-generated turns.
Construct a LLaVA-format prompt with the image token placeholder, system prompt, and user/assistant turns.
Explain why the community moved from Q-Former to MLP despite Q-Former's token-budget win.

Introduction

Type: Build Languages: Python (stdlib, projector + instruction-template builder) Prerequisites: Phase 12 · 02 (CLIP), Phase 11 (LLM Engineering — instruction tuning) Time: ~180 minutes

Learning Objectives

Build a 2-layer MLP projector that maps ViT patch embeddings (dim 1024) to an LLM's embedding dim (dim 4096).
Walk the LLaVA two-stage recipe: (1) projector alignment on 558k caption pairs, (2) visual instruction tuning on 158k GPT-4-generated turns.
Construct a LLaVA-format prompt with the image token placeholder, system prompt, and user/assistant turns.
Explain why the community moved from Q-Former to MLP despite Q-Former's token-budget win.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Use It, Ship It, Exercises, Key Terms, Further Reading — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.

Key terms in this lesson

Related lessons

Vision Transformers and the Patch-Token Primitive
CLIP and Contrastive Vision-Language Pretraining
From CLIP to BLIP-2 — Q-Former as Modality Bridge
Flamingo and Gated Cross-Attention for Few-Shot VLMs

Browse all courses · View pricing · DeVenture Academy