Home › Courses › LLM Inference & Production Infrastructure

LLM Inference & Production Infrastructure

Serve a model in production: paged attention, continuous batching, quantization, goodput, and the cost per million tokens

Everything between a trained model and a request that returns. You read what a serving engine actually does — PagedAttention, continuous batching, chunked prefill, prefix-cache reuse, speculative decoding, disaggregated prefill/decode — then the hardware and quantization decisions underneath it, the metrics that say whether a deployment is healthy (TTFT, TPOT, ITL, goodput, P99), and the unit economics that say whether it is solvent. It ends where production actually bites: shadow traffic and canaries for a system whose output cannot be diffed, load tests that do not lie, incident response, secrets, and the compliance regimes that apply.

5 phases · 28 lessons · 28 worked examples · Early access

Take Transformers & LLMs from Scratch first — this course builds on it.

Every phase, every lesson, every project

• Platforms & Inference Economics (5 lessons) — free — Where inference actually runs and what it costs: managed platforms, serverless GPU providers, batch APIs, model routing, and the gateway layer in front of them.
  • Managed LLM Platforms — Bedrock, Vertex AI, Azure OpenAI
  • Inference Platform Economics — Fireworks, Together, Baseten, Modal, Replicate, Anyscale
  • Batch APIs — the 50% Discount as Industry Standard
  • Model Routing as a Cost-Reduction Primitive
  • AI Gateways — LiteLLM, Portkey, Kong AI Gateway, Bifrost
• Serving Engine Internals (6 lessons) — What a serving engine does between the request and the first token: paged attention, continuous batching, prefix-cache reuse, speculative decoding, and disaggregated prefill/decode.
  • Serving Engine Internals — PagedAttention, Continuous Batching, Chunked Prefill
  • EAGLE-3 Speculative Decoding in Production
  • Prefix-Cache Serving — RadixAttention and KV Reuse
  • Disaggregated Prefill/Decode — NVIDIA Dynamo and llm-d
  • Production Serving Stack — KV Offloading and Cache-Aware Routing
  • Self-Hosted Serving Selection — Matching Engine to Hardware and Scale
• Hardware, Quantization & Placement (6 lessons) — Making the hardware the constraint it should be: GPU autoscaling, compilation for the target chip, production quantization, cold starts, region placement, and inference at the edge.
  • GPU Autoscaling on Kubernetes — Karpenter, KAI Scheduler, Gang Scheduling
  • Hardware-Specialized Inference Compilation — FP8 and NVFP4 on Blackwell
  • Production Quantization — AWQ, GPTQ, GGUF K-quants, FP8, MXFP4/NVFP4
  • Cold Start Mitigation for Serverless LLMs
  • Multi-Region LLM Serving and KV Cache Locality
  • Edge Inference — Apple Neural Engine, Qualcomm Hexagon, WebGPU/WebLLM, Jetson
• Measurement & Unit Economics (4 lessons) — The numbers that decide whether a deployment is healthy or solvent: TTFT, TPOT, goodput and P99, the observability stack that reports them, caching economics, and per-tenant cost attribution.
  • Inference Metrics — TTFT, TPOT, ITL, Goodput, P99
  • LLM Observability Stack Selection
  • Prompt Caching and Semantic Caching Economics
  • FinOps for LLMs — Unit Economics and Multi-Tenant Attribution
• Release, Reliability & Compliance (7 lessons) — Shipping changes to a system whose output cannot be diffed: shadow traffic, canaries, A/B tests, honest load testing, incident response, chaos, secrets, and the compliance regimes that apply.
  • Shadow Traffic, Canary Rollout, and Progressive Deployment for LLMs
  • A/B Testing LLM Features — GrowthBook, Statsig, and the Vibes Problem
  • Load Testing LLM APIs — Why k6 and Locust Lie
  • SRE for AI — Multi-Agent Incident Response, Runbooks, Predictive Detection
  • Chaos Engineering for LLM Production
  • Security — Secrets, API Key Rotation, Audit Logs, Guardrails
  • Compliance — SOC 2, HIPAA, GDPR, PCI-DSS, EU AI Act, ISO 42001

The technologies you will use

vLLM · SGLang · TensorRT-LLM · Kubernetes · LiteLLM · Prometheus

Common questions

Does this course have graded labs?

Not yet. Every lesson ships the worked example it refers to, and the exercises are stated at the end of each lesson, but nothing here is checked by the CLI the way the from-scratch courses are. These are operations decisions against running infrastructure, which is harder to grade than a function with a return value. The course is marked early access for that reason.

Do I need a GPU?

No. The lessons read real serving engines and real numbers rather than asking you to reproduce them, and the worked examples are stdlib Python — the continuous-batching lesson, for instance, ships a toy scheduler that runs anywhere.

Key terms in this course

Inference · Quantization · Speculative Decoding · vLLM · Token

Continue your learning path

ML & AI Engineering · Transformers & LLMs from Scratch · Agentic AI Engineering

Start the LLM Inference & Production Infrastructure course

Create a free account — the opening phases of 29 of 35 courses are free, no credit card. Or see Pro pricing.

All courses · Pricing · About · FAQ · Glossary