Phase 6: Distributed Training · ~90 min · Python

Sharded Checkpoint and Atomic Resume

A 70B-parameter training job is paused by a node failure every few hours.

Hiring signal: Can build sharded checkpoint and atomic resume end to end

What you will learn

Save a multi-rank checkpoint as a per-rank shard file plus a manifest that records which rank owns what.
Use the atomic write pattern (write to a temp path then rename) so a crash mid-write never produces a half-finished checkpoint.
Resume from the manifest, verifying byte-equal state for both fp16 parameters and the ZeRO optimiser state on every rank.
Defend the manifest schema against the three failure modes: world-size change, shard count mismatch, and partial write.

Introduction

The checkpoint format decides whether you lose 30 minutes or 30 hours. A sharded checkpoint writes every rank's shard in parallel and records ownership in a manifest. Resume loads each rank's shard from its own file, reconstructs the state on the same world size, and the optimiser steps as if nothing happened. Atomic write keeps a half-finished checkpoint from poisoning the next resume.

Type: Build Languages: Python Prerequisites: Phase 19 Track C lessons 42-49 Time: ~90 min

Objective

Learning objectives

Save a multi-rank checkpoint as a per-rank shard file plus a manifest that records which rank owns what.
Use the atomic write pattern (write to a temp path then rename) so a crash mid-write never produces a half-finished checkpoint.
Resume from the manifest, verifying byte-equal state for both fp16 parameters and the ZeRO optimiser state on every rank.
Defend the manifest schema against the three failure modes: world-size change, shard count mismatch, and partial write.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.