Phase 4: Training LLMs · ~60 minutes · Python

Multi-Token Prediction (MTP)

Every autoregressive LLM from GPT-2 to Llama 3 trains on one loss per position: predict the next token. DeepSeek-V3 added a second loss per position: predict the token after that. The extra 14B of parameters (on a 671B model) got distilled back into the main model through gradient flow, and the trained MTP heads were repurposed at inference as speculative-decoding drafters with 80%+ acceptance. 1.8× generation throughput came for free. This lesson builds the sequential MTP module from the DeepSeek technical report, computes the loss and the shared-head parameter layout, and explains why MTP keeps the causal chain while Gloeckle et al.'s original parallel MTP broke it.

Hiring signal: Understanding of multi-token prediction (mtp) internals

What you will learn

State the MTP training objective and derive the joint loss across prediction depths.
Explain the difference between Gloeckle et al.'s parallel MTP heads (2024) and DeepSeek-V3's sequential MTP modules and why the sequential design preserves the causal chain.
Compute the parameter and memory overhead of adding MTP modules to a pre-training run.
Implement one MTP module from scratch: the shared embedding, the per-depth transformer block, the projection, and the shared output head.

Introduction

Type: Build Languages: Python (stdlib) Prerequisites: Phase 10 · 04 (pre-training a mini GPT), Phase 10 · 15 (speculative decoding) Time: ~60 minutes

Learning Objectives

State the MTP training objective and derive the joint loss across prediction depths.
Explain the difference between Gloeckle et al.'s parallel MTP heads (2024) and DeepSeek-V3's sequential MTP modules and why the sequential design preserves the causal chain.
Compute the parameter and memory overhead of adding MTP modules to a pre-training run.
Implement one MTP module from scratch: the shared embedding, the per-depth transformer block, the projection, and the shared output head.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build It, Use It, Ship It, Exercises, Key Terms, Further Reading — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.