Phase 9: LLMOps & Production Deployment · 60 min · PyTorch · bitsandbytes · ONNX Runtime

Model Optimization: Quantization & Distillation

The model that wins the leaderboard rarely wins production. Latency, memory, and cost decide what ships.

Hiring signal: Shrinks models for production without hand-waving the accuracy tradeoff

What you will learn

Explain quantization (FP32→INT8/4-bit) and its memory/latency wins
Distinguish post-training quantization from quantization-aware training
Describe knowledge distillation and when to use it
Reason about the accuracy vs cost/latency tradeoff with measurements
Choose an optimization path for a given deployment constraint

The Problem

A 70B-parameter model in FP32 needs ~280GB just to load the weights — multiple expensive GPUs before you serve a single request. Even a "small" model can blow your latency budget or your cloud bill. The gap between a model that works in a notebook and a model you can afford to serve is optimization: quantization and distillation. This is core LLMOps work and a frequent interview topic ("the model is too slow/expensive — what do you do?"). The danger is optimizing blindly and silently tanking quality, or never measuring whether the speedup was worth it.

The Concept

Three levers shrink a model's footprint and latency:

Quantization : store/compute weights in fewer bits (FP32 -> INT8 / 4-bit)
Distillation : train a small "student" to mimic a big "teacher"
Pruning      : remove low-importance weights/heads (less common in practice)

Quantization is the highest-leverage, most-used technique — start there.

A 7B parameter model in FP32 takes 28GB of GPU memory. How much memory does it need in INT4 quantization, and can it run on a single 16GB GPU?

FP32 = 4 bytes/param: 7B × 4 = 28GB. INT4 = 0.5 bytes/param: 7B × 0.5 = 3.5GB. The model fits easily on a 16GB GPU with ~12GB left for the KV cache and activations. This is why INT4 quantization (via GPTQ, AWQ, or GGUF) is the standard for running 7B-13B models on consumer GPUs.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Evaluation, Exercises, Key Terms, Common Pitfalls, Interview Framing — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.