Phase 1: Pretraining at Scale · ~90 minutes · Python

Distributed Data Parallel and FSDP from Scratch

Multi-rank training is two collectives and one rule. Broadcast the parameters at startup, average the gradients after backward, never let the ranks disagree about what step they are on.

Hiring signal: Can build distributed data parallel and fsdp from scratch end to end

What you will learn

Bring up a process group across N ranks with the `gloo` backend, no special hardware.
Implement a minimal DDP wrapper that broadcasts parameters at construction and all-reduces gradients after backward.
Prove that the all-reduce of per-rank gradients matches a single-process gradient on the concatenated input.
Sketch FSDP parameter sharding: each rank holds a slice, the full tensor is gathered for the forward pass and dropped after.

Introduction

Type: Build Languages: Python Prerequisites: Phase 19 lessons 42 to 45 Time: ~90 minutes

Objective

Learning objectives

Bring up a process group across N ranks with the gloo backend, no special hardware.
Implement a minimal DDP wrapper that broadcasts parameters at construction and all-reduces gradients after backward.
Prove that the all-reduce of per-rank gradients matches a single-process gradient on the concatenated input.
Sketch FSDP parameter sharding: each rank holds a slice, the full tensor is gathered for the forward pass and dropped after.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.