Phase 1: Pretraining at Scale · ~90 minutes · Python

Gradient Clipping and Mixed Precision

The optimizer and schedule from the previous lesson assume gradients are sane.

Hiring signal: Can build gradient clipping and mixed precision end to end

What you will learn

Compute the global L2 norm over all parameter gradients and clip in place when it exceeds a configured threshold.
Wrap a training step in autocast plus a GradScaler so FP16 forward and backward passes survive overflow.
Detect NaN and Inf in the loss or gradient, skip the optimizer step, and log the skip.
Report the GradScaler's scaling factor every step so a long sequence of skips is visible immediately.

Introduction

They usually are not. A single bad batch can spike the gradient norm by three orders of magnitude. Mixed-precision training amplifies this by introducing FP16 overflow on the loss side. This lesson builds the two safety belts that production training cannot ship without: gradient clipping to a configured global L2 norm, and a mixed-precision loop with autocast and GradScaler that detects NaN and Inf, skips the step cleanly, and logs the scaling factor for forensics.

Type: Build Languages: Python Prerequisites: Phase 19 lessons 30-37 Time: ~90 minutes

Objective

Learning objectives

Compute the global L2 norm over all parameter gradients and clip in place when it exceeds a configured threshold.
Wrap a training step in autocast plus a GradScaler so FP16 forward and backward passes survive overflow.
Detect NaN and Inf in the loss or gradient, skip the optimizer step, and log the skip.
Report the GradScaler's scaling factor every step so a long sequence of skips is visible immediately.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.