Phase 2: Hardware, Quantization & Placement · ~75 minutes · Python (stdlib · toy memory and throughput comparison across formats)
Production Quantization — AWQ, GPTQ, GGUF K-quants, FP8, MXFP4/NVFP4
Quantization format is not a universal choice — it is a function of hardware, serving engine, and workload.
Hiring signal: Can operate production quantization in production
What you will learn
- Name the six production quantization formats and their sweet spots in 2026.
- Pick a format given hardware (CPU vs GPU, Hopper vs Blackwell), engine (vLLM, TRT-LLM, llama.cpp), and workload (routine chat, reasoning, multi-LoRA).
- Compute the weight memory saved and the KV cache left untouched for a chosen format.
- Name the calibration-dataset pitfall that degrades quantized models on domain traffic.
Introduction
GGUF Q4_K_M or Q5_K_M owns CPU and edge, delivered through llama.cpp and Ollama. GPTQ wins inside vLLM when you need multi-LoRA on the same base. AWQ with Marlin-AWQ kernels delivers ~741 tok/s on a 7B class model with the best Pass@1 at INT4 — the 2026 default for datacenter production. FP8 stays the middle ground on Hopper, Ada, and Blackwell — near-lossless and widely supported. NVFP4 and MXFP4 (Blackwell microscaling) are aggressive and require per-block validation. Two traps bite teams: calibration dataset must match deployment domain, and KV cache is separate from weight quantization — the AWQ lesson "my model is 4 GB now" forgets the 10-30 GB KV cache at production batch sizes.
Type: Learn Languages: Python (stdlib, toy memory and throughput comparison across formats) Prerequisites: Phase 10 · 13 (Quantization foundations), Phase 17 · 04 (Serving Engine Internals) Time: ~75 minutes
Objective
Learning objectives
- Name the six production quantization formats and their sweet spots in 2026.
- Pick a format given hardware (CPU vs GPU, Hopper vs Blackwell), engine (vLLM, TRT-LLM, llama.cpp), and workload (routine chat, reasoning, multi-LoRA).
- Compute the weight memory saved and the KV cache left untouched for a chosen format.
- Name the calibration-dataset pitfall that degrades quantized models on domain traffic.
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.
Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.
Browse all courses · View pricing · DeVenture Academy