Phase 2: Pre-training & Fine-tuning · ~60 minutes · Python

Speculative Decoding — Draft, Verify, Repeat

Autoregressive decoding is serial. Each token waits for the previous one. Speculative decoding breaks the chain: a cheap model drafts N tokens, the expensive model verifies all N in one forward pass. When the draft is right you paid one big forward for N generations.

Hiring signal: Understanding of speculative decoding — draft, verify, repeat internals

What you will learn

Implement speculative decoding — draft, verify, repeat from scratch
Understand the math and intuition behind the algorithm
Use production libraries for the same task
Ship a reusable artifact

Introduction

Type: Build Languages: Python Prerequisites: Phase 7 · 07 (GPT Causal LM), Phase 7 · 12 (KV Cache & Flash Attention) Time: ~60 minutes

The Problem

A 70B LLM sampling one token takes ~30 ms on an H100. A 3B draft model takes ~3 ms. If we let the 3B draft 5 tokens ahead, then run the 70B once to verify all 5, the total is 5×3 + 30 = 45 ms for up to 5 accepted tokens — versus 5×30 = 150 ms for straight-line generation. That is the full speculative-decoding pitch: trade a small amount of extra GPU memory (draft model) for 2–4× lower decode latency.

The trick has to preserve the distribution. Speculative sampling, introduced by Leviathan et al. (2023) and by Chen et al. concurrently, guarantees that the output sequence is identically distributed to what the big model would have produced on its own. No quality tradeoff. Just faster.

Four families of draft-verifier pairs dominate 2026 inference:

Vanilla speculative (Leviathan 2023). Separate draft model (e.g., Llama 3 1B) + verifier (e.g., Llama 3 70B).
Medusa (Cai 2024). Multiple decoding heads on the verifier predict positions t+1..t+k in parallel. No separate draft model.
EAGLE family (Li 2024, 2025). Lightweight draft that reuses the verifier's hidden states; closer acceptance rate than vanilla; 3–4× typical.
Lookahead decoding (Fu 2024). Jacobi iteration; no draft model required at all. Self-speculation. Niche but dependency-free.

Every production inference stack in 2026 ships speculative decoding by default. vLLM, TensorRT-LLM, SGLang, and llama.cpp all support at least vanilla + EAGLE-2.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Concept, Build It, Use It, Ship It, Exercises, Key Terms, Further Reading — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.