Phase 2: Hardware, Quantization & Placement · ~75 minutes · Python (stdlib · toy queue-depth autoscaler simulator)

GPU Autoscaling on Kubernetes — Karpenter, KAI Scheduler, Gang Scheduling

Three layers, not one.

Hiring signal: Can operate gpu autoscaling on kubernetes in production

What you will learn

Diagram the three autoscaling layers (node provisioning, gang scheduling, application-level) and name the tool used at each layer.
Explain why `DCGM_FI_DEV_GPU_UTIL` is the wrong HPA signal for vLLM and name two replacements (queue depth, KV cache utilization).
Describe gang scheduling and the partial-allocation failure mode KAI Scheduler prevents (7 of 8 GPUs idle).
Name the Karpenter consolidation policy (`WhenEmptyOrUnderutilized`) that terminates running GPU jobs and state the 2026 safe alternative.

Introduction

Karpenter provisions nodes dynamically (under one minute, 40% faster than Cluster Autoscaler). KAI Scheduler handles gang scheduling, topology awareness, and hierarchical queues — it prevents the 7-of-8 partial allocation trap where seven nodes wait and burn on one missing GPU. Application-level autoscalers (NVIDIA Dynamo Planner, llm-d Workload Variant Autoscaler) scale on inference-specific signals — queue depth, KV cache utilization — not CPU/DCGM duty cycle. The classic HPA trap is that DCGM_FI_DEV_GPU_UTIL is a duty-cycle measurement: 100% could be 10 requests or 100. vLLM pre-allocates KV cache memory, so memory never triggers scale-down. This lesson teaches you to compose the three layers and avoid the default Karpenter WhenEmptyOrUnderutilized policy that terminates running GPU jobs mid-inference.

Type: Learn Languages: Python (stdlib, toy queue-depth autoscaler simulator) Prerequisites: Phase 17 · 02 (Inference Platform Economics), Phase 17 · 04 (Serving Engine Internals) Time: ~75 minutes

Objective

Learning objectives

Diagram the three autoscaling layers (node provisioning, gang scheduling, application-level) and name the tool used at each layer.
Explain why DCGM_FI_DEV_GPU_UTIL is the wrong HPA signal for vLLM and name two replacements (queue depth, KV cache utilization).
Describe gang scheduling and the partial-allocation failure mode KAI Scheduler prevents (7 of 8 GPUs idle).
Name the Karpenter consolidation policy (WhenEmptyOrUnderutilized) that terminates running GPU jobs and state the 2026 safe alternative.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.

Key terms in this lesson

Related lessons

Hardware-Specialized Inference Compilation — FP8 and NVFP4 on Blackwell
Production Quantization — AWQ, GPTQ, GGUF K-quants, FP8, MXFP4/NVFP4
Cold Start Mitigation for Serverless LLMs
Multi-Region LLM Serving and KV Cache Locality

Browse all courses · View pricing · DeVenture Academy