Phase 2: Audio-Visual Models · ~180 minutes · Python
Chameleon and Early-Fusion Token-Only Multimodal Models
Every VLM we have seen so far keeps images and text separate. Visual tokens come from a vision encoder, flow into a projector, then meet text inside the LLM. The vision and text vocabularies never overlap. Chameleon (Meta, May 2024) asked: what if they did? Train a VQ-VAE that turns an image into a sequence of discrete tokens from a shared vocabulary. Every multimodal document is now one sequence — text tokens and image tokens interleaved, a single autoregressive loss. Side effect: the model can generate mixed-modality outputs — alternating text and image tokens in a single inference call. This lesson reads the early-fusion thesis and builds a toy version end to end.
What you will learn
- Explain why a shared vocabulary + single loss changes what the model can do.
- Describe how a VQ-VAE tokenizes an image into a discrete sequence compatible with a transformer's next-token objective.
- Name Chameleon's training-stability tricks: QK-Norm, dropout placement, LayerNorm ordering.
- Compare Chameleon vs BLIP-2's Q-Former approach and describe when each is the right choice.
Introduction
- Type: Build
- Languages: Python (stdlib, VQ-VAE tokenizer + interleaved decoder)
- Prerequisites: LLaVA and Visual Instruction Tuning, the Generative AI Fundamentals course
- Time: ~180 minutes
Objective
Learning objectives
- Explain why a shared vocabulary + single loss changes what the model can do.
- Describe how a VQ-VAE tokenizes an image into a discrete sequence compatible with a transformer's next-token objective.
- Name Chameleon's training-stability tricks: QK-Norm, dropout placement, LayerNorm ordering.
- Compare Chameleon vs BLIP-2's Q-Former approach and describe when each is the right choice.
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.
Create a free account — the opening phases of 29 of 35 courses are free, no credit card.
Browse all courses · View pricing · DeVenture Academy