Phase 9: LLMOps & Production Deployment · 60 min · vLLM · Text Generation Inference · Kubernetes

Serving LLMs: vLLM, Batching & Throughput

Generating one token at a time wastes a GPU. Serving LLMs well is the art of keeping it busy.

Hiring signal: Knows why LLM serving differs from REST and can reason about throughput vs latency

What you will learn

Explain why autoregressive generation makes LLM serving unique
Describe continuous batching and the KV cache (PagedAttention)
Reason about the throughput vs latency tradeoff and batch size
Stand up an OpenAI-compatible endpoint with vLLM
Plan autoscaling and estimate cost per million tokens

The Problem

Serving an LLM is not like serving a normal REST model. A classifier does one forward pass and returns. An LLM generates autoregressively — one token at a time, each depending on the last, often hundreds of steps per request. Naively, that leaves an expensive GPU mostly idle and your throughput in the floor. Specialized serving engines like vLLM exist precisely to solve this, and "how would you serve an LLM at scale?" is now a standard system-design question. Getting this wrong means a bill 5–10× higher than necessary, or latency that misses SLA.

The Concept

Two properties make LLM serving hard, and two techniques fix them:

Property                              Fix
- generation is sequential/long  -> continuous (in-flight) batching
- each request keeps growing KV  -> paged KV cache (PagedAttention)

KV cache : stores attention keys/values for tokens already generated so each new token
           is cheap. Grows with sequence length -> dominates GPU memory at scale.

With static batching, you wait for all 8 requests in a batch to finish before starting the next batch. One long request (500 tokens) blocks 7 short ones (50 tokens). How does continuous batching fix this?

Continuous (in-flight) batching treats each token position as an independent scheduling slot. When a short request finishes, its slot is immediately filled by a new request — the GPU never idles waiting for the long request. This is why vLLM and TGI achieve 3-5x higher throughput than naive static batching.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Evaluation, Exercises, Key Terms, Common Pitfalls, Interview Framing — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.