Phase 1: Vision-Language Models · ~120 minutes · Python

InternVL3: Native Multimodal Pretraining

Every open VLM before InternVL3 followed the same three-step recipe: take a text LLM trained on trillions of text tokens, bolt on a vision encoder, then fine-tune the seams. This works but has alignment debt — the text LLM has spent its full pretraining budget on pure text and does not natively understand visual tokens. When you add vision post-hoc, the LLM has to re-learn how to relate visual input to its text reasoning without forgetting the text. InternVL3 (Zhu et al., April 2025) rejects the post-hoc approach: one pretraining run, text and multimodal interleaved from step one. The result matches Gemini 2.5 Pro on MMMU-Pro at 78B params open. This lesson reads the case for native pretraining and what changes when you make it.

Hiring signal: Understanding of internvl3: native multimodal pretraining internals

What you will learn

Explain why post-hoc VLM training accumulates alignment debt, citing the three measurable symptoms (catastrophic forgetting, answer drift, visual-text inconsistency).
Describe InternVL3's native pretraining corpus mix and why the ratio of text : interleaved : caption matters.
Compare V2PE (variable visual position encoding) to Qwen2-VL's M-RoPE.
Name the Visual Resolution Router (ViR) and Decoupled Vision-Language (DvD) deployment optimizations.

Introduction

Type: Learn Languages: Python (stdlib, training-corpus mixer) Prerequisites: Phase 12 · 05, Phase 12 · 07 (recipes) Time: ~120 minutes

Learning Objectives

Explain why post-hoc VLM training accumulates alignment debt, citing the three measurable symptoms (catastrophic forgetting, answer drift, visual-text inconsistency).
Describe InternVL3's native pretraining corpus mix and why the ratio of text : interleaved : caption matters.
Compare V2PE (variable visual position encoding) to Qwen2-VL's M-RoPE.
Name the Visual Resolution Router (ViR) and Decoupled Vision-Language (DvD) deployment optimizations.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Use It, Ship It, Exercises, Key Terms, Further Reading — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.