Phase 1: Pretraining at Scale · ~90 minutes · Python

Checkpoint Save and Resume

Train interrupts kill runs; checkpoints let them continue. Save model, optimizer, scheduler, loss history, step counter, and RNG state, atomically, so a kill at any moment leaves a valid file on disk.

Hiring signal: Can build checkpoint save and resume end to end

What you will learn

Capture the full training state into a single payload that can be reloaded into a fresh process.
Implement atomic save with write-to-temp then rename so a crash never leaves a half-written file.
Restore the RNG state for Python, NumPy, and PyTorch so the post-resume loss matches the uninterrupted baseline.
Build a sharded checkpoint layout for models that no longer fit in a single file, with hash-verified shards and a JSON index.

Introduction

Type: Build Languages: Python Prerequisites: Phase 19 lessons 42 to 45 Time: ~90 minutes

Objective

Learning objectives

Capture the full training state into a single payload that can be reloaded into a fresh process.
Implement atomic save with write-to-temp then rename so a crash never leaves a half-written file.
Restore the RNG state for Python, NumPy, and PyTorch so the post-resume loss matches the uninterrupted baseline.
Build a sharded checkpoint layout for models that no longer fit in a single file, with hash-verified shards and a JSON index.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.