Phase 0: Multimodal Foundations · ~180 minutes · Python
LLaVA and Visual Instruction Tuning
LLaVA (April 2023) is the most copied multimodal architecture on the planet. It replaced BLIP-2's Q-Former with a 2-layer MLP, replaced Flamingo's gated cross-attention with naive token concatenation, and trained on 158k visual-instruction turns generated by GPT-4 from text-only captions. Any practitioner who built a VLM between 2023 and 2026 built some variant of LLaVA. LLaVA-1.5 added AnyRes. LLaVA-NeXT bumped resolution. LLaVA-OneVision unified image, multi-image, and video in one recipe. This lesson reads the recipe, implements the projector, and explains why "simpler won."
What you will learn
- Build a 2-layer MLP projector that maps ViT patch embeddings (dim 1024) to an LLM's embedding dim (dim 4096).
- Walk the LLaVA two-stage recipe: (1) projector alignment on 558k caption pairs, (2) visual instruction tuning on 158k GPT-4-generated turns.
- Construct a LLaVA-format prompt with the image token placeholder, system prompt, and user/assistant turns.
- Explain why the community moved from Q-Former to MLP despite Q-Former's token-budget win.
Introduction
- Type: Build
- Languages: Python (stdlib, projector + instruction-template builder)
- Prerequisites: CLIP and Contrastive Vision-Language Pretraining, the ML & AI Engineering course
- Time: ~180 minutes
Objective
Learning objectives
- Build a 2-layer MLP projector that maps ViT patch embeddings (dim 1024) to an LLM's embedding dim (dim 4096).
- Walk the LLaVA two-stage recipe: (1) projector alignment on 558k caption pairs, (2) visual instruction tuning on 158k GPT-4-generated turns.
- Construct a LLaVA-format prompt with the image token placeholder, system prompt, and user/assistant turns.
- Explain why the community moved from Q-Former to MLP despite Q-Former's token-budget win.
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.
Create a free account — the opening phases of 29 of 35 courses are free, no credit card.
Browse all courses · View pricing · DeVenture Academy