Phase 6: Distributed Training · ~90 min · Python

ZeRO Optimizer State Sharding

Adam stores two moment estimates per parameter, both in float32.

Hiring signal: Can build zero optimizer state sharding end to end

What you will learn

Shard optimiser state (first moment, second moment, fp32 master copy) across N ranks so each rank owns 1/N.
Use reduce_scatter to deliver each rank only its shard's gradient sum, then allgather to broadcast the updated parameter shards back.
Compute the memory savings table for stage 1, stage 2, stage 3 against vanilla DDP.
Defend the choice of stage 1 vs stage 2 vs stage 3 on model size and bandwidth budget.

Introduction

A 7B-parameter model carries 56 GB of optimiser state. ZeRO stage 1 shards that across N ranks; each rank owns 1/N of the optimiser. After the local step the updated parameter shards broadcast back, every rank reconstructs the full model, and the next step begins. The win is a linear memory drop on the largest single allocation in the training stack.

Type: Build Languages: Python Prerequisites: Phase 19 Track C lessons 42-49 Time: ~90 min

Objective

Learning objectives

Shard optimiser state (first moment, second moment, fp32 master copy) across N ranks so each rank owns 1/N.
Use reduce_scatter to deliver each rank only its shard's gradient sum, then allgather to broadcast the updated parameter shards back.
Compute the memory savings table for stage 1, stage 2, stage 3 against vanilla DDP.
Defend the choice of stage 1 vs stage 2 vs stage 3 on model size and bandwidth budget.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.