Phase 1: Pretraining at Scale · ~90 minutes · Python

Cosine LR with Linear Warmup

The learning-rate schedule is the second most important decision after the loss function.

Hiring signal: Can build cosine lr with linear warmup end to end

What you will learn

Implement an AdamW optimizer wired to a cosine learning-rate schedule with linear warmup.
Compute the schedule's exact value at any step without floating-point drift across runs.
Log gradient L2 norm side by side with the learning rate so training health is observable.
Render the schedule to a text plot the eye can read and a CSV any tool can consume.

Introduction

AdamW with a cosine decay and a linear warmup is the modern default for language-model training because it lets the model see a small effective step size during the brittle first thousand updates, ramps up to a configured peak, and decays smoothly back toward zero. This lesson builds that schedule, plots the curve over training steps, logs gradient norms next to the schedule, and proves the schedule honors warmup, peak, and decay boundaries.

Type: Build Languages: Python Prerequisites: Phase 19 lessons 30-37 Time: ~90 minutes

Objective

Learning objectives

Implement an AdamW optimizer wired to a cosine learning-rate schedule with linear warmup.
Compute the schedule's exact value at any step without floating-point drift across runs.
Log gradient L2 norm side by side with the learning rate so training health is observable.
Render the schedule to a text plot the eye can read and a CSV any tool can consume.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.

Key terms in this lesson

Learning Rate