Phase 6: Distributed Training · ~90 min · Python

End-to-End Distributed Training

Lessons 76 through 80 each built one piece.

Hiring signal: Can build end-to-end distributed training end to end

What you will learn

Compose DDP (lesson 77) plus ZeRO-1 (lesson 78) plus sharded checkpoints (lesson 80) into one training loop.
Train a 2-layer transformer language model on a small synthetic corpus for 20 steps across 4 simulated ranks.
Print a per-step loss table, a per-rank memory profile, and a checkpoint manifest that resumes byte-equal on the same world size.
Defend the composition: each piece is independently testable in earlier lessons and this lesson proves they compose.

Introduction

This is the assembly: a tiny GPT trained across 4 simulated ranks with DDP for gradient sync, ZeRO-1 for optimiser-state sharding, and a sharded checkpoint at the halfway mark. The demo runs 20 steps, self-terminates, prints a loss curve plus a memory profile, and writes a resumable checkpoint.

Type: Build Languages: Python Prerequisites: Phase 19 Track C lessons 42-49 Time: ~90 min

Objective

Learning objectives

Compose DDP (lesson 77) plus ZeRO-1 (lesson 78) plus sharded checkpoints (lesson 80) into one training loop.
Train a 2-layer transformer language model on a small synthetic corpus for 20 steps across 4 simulated ranks.
Print a per-step loss table, a per-rank memory profile, and a checkpoint manifest that resumes byte-equal on the same world size.
Defend the composition: each piece is independently testable in earlier lessons and this lesson proves they compose.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.

Key terms in this lesson

Transformer