Phase 1: Vision-Language Models · ~120 minutes · Python

InternVL3: Native Multimodal Pretraining

Every open VLM before InternVL3 followed the same three-step recipe: take a text LLM trained on trillions of text tokens, bolt on a vision encoder, then fine-tune the seams. This works but has alignment debt — the text LLM has spent its full pretraining budget on pure text and does not natively understand visual tokens. When you add vision post-hoc, the LLM has to re-learn how to relate visual input to its text reasoning without forgetting the text. InternVL3 (Zhu et al., April 2025) rejects the post-hoc approach: one pretraining run, text and multimodal interleaved from step one. The result matches Gemini 2.5 Pro on MMMU-Pro at 78B params open. This lesson reads the case for native pretraining and what changes when you make it.

What you will learn

Explain why post-hoc VLM training accumulates alignment debt, citing the three measurable symptoms (catastrophic forgetting, answer drift, visual-text inconsistency).
Describe InternVL3's native pretraining corpus mix and why the ratio of text : interleaved : caption matters.
Compare V2PE (variable visual position encoding) to Qwen2-VL's M-RoPE.
Name the Visual Resolution Router (ViR) and Decoupled Vision-Language (DvD) deployment optimizations.

Introduction

Type: Learn
Languages: Python (stdlib, training-corpus mixer)
Prerequisites: LLaVA and Visual Instruction Tuning, Open-Weight VLM Recipes: What Actually Matters
Time: ~120 minutes

Objective

Learning objectives

Explain why post-hoc VLM training accumulates alignment debt, citing the three measurable symptoms (catastrophic forgetting, answer drift, visual-text inconsistency).
Describe InternVL3's native pretraining corpus mix and why the ratio of text : interleaved : caption matters.
Compare V2PE (variable visual position encoding) to Qwen2-VL's M-RoPE.
Name the Visual Resolution Router (ViR) and Decoupled Vision-Language (DvD) deployment optimizations.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.