Phase 2: Models, Serving & Observability · 30 hours · Python (serving) · C++ / CUDA (kernel inspection) · YAML (configs)

Capstone 14 — Speculative-Decoding Inference Server

Speculative decoding — a cheap draft proposes tokens, the target model verifies them in one pass — is now a production-ready optimization, not a research trick.

Hiring signal: Can build capstone 14 end to end

Introduction

EAGLE-3 in vLLM 0.7 ships 2.5-3x throughput on real traffic. P-EAGLE (AWS 2026) pushed parallel speculation even further. SGLang's SpecForge trained draft heads at scale. Red Hat's Speculators hub published aligned drafts for common open models. TensorRT-LLM made speculative decoding first-class on NVIDIA. The 2026 production serving stack is vLLM or SGLang with EAGLE-family drafts, FP8 or INT4 quantization, and HPA on queue-wait. This capstone is to serve two open models at 2.5x+ baseline throughput with a full tail-latency report.

Phases exercised: P3 · P7 · P10 · P17

Type: Capstone Languages: Python (serving), C++ / CUDA (kernel inspection), YAML (configs) Prerequisites: Phase 3 (deep learning), Phase 7 (transformers), Phase 10 (LLMs from scratch), Phase 17 (infrastructure) Time: 30 hours

Objective

Learning objectives

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.