Phase 3: LLM Architecture · ~120 minutes · Python

Inference Optimization

Two phases define LLM inference. Prefill processes your prompt in parallel -- compute-bound. Decode generates tokens one at a time -- memory-bound. Every optimization targets one or both.

Hiring signal: Understanding of inference optimization internals

What you will learn

Implement KV-cache to eliminate redundant computation during autoregressive token generation
Explain the prefill vs decode phases of LLM inference and why each has different bottlenecks (compute-bound vs memory-bound)
Implement continuous batching and PagedAttention concepts to maximize GPU utilization under concurrent requests
Compare inference optimization techniques (KV-cache, speculative decoding, flash attention) and their throughput/latency tradeoffs

Introduction

Type: Build Languages: Python Prerequisites: Phase 10, Lessons 01-08 (Transformer architecture, attention) Time: ~120 minutes

Learning Objectives

Implement KV-cache to eliminate redundant computation during autoregressive token generation
Explain the prefill vs decode phases of LLM inference and why each has different bottlenecks (compute-bound vs memory-bound)
Implement continuous batching and PagedAttention concepts to maximize GPU utilization under concurrent requests
Compare inference optimization techniques (KV-cache, speculative decoding, flash attention) and their throughput/latency tradeoffs

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build It, Use It, Ship It, Exercises, Key Terms, Further Reading — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.