Phase 6: Distributed Training · ~90 min · Python

Collective Ops From Scratch

The four collective operations that hold distributed training together are allreduce, broadcast, allgather, and reduce_scatter.

Hiring signal: Can build collective ops from scratch end to end

What you will learn

Implement ring allreduce in two passes (reduce-scatter then allgather) and prove the per-rank communication volume is 2(N-1)/N bytes per element.
Build broadcast, allgather, and reduce_scatter on top of point-to-point sends over `multiprocessing.Queue`.
Verify every primitive against a `torch.distributed` gloo reference for the same input.
Defend the choice of ring versus tree on cluster shape, latency floor, and bandwidth ceiling.

Introduction

Every other primitive a training framework offers is a wrapper around these. Build them once over a multiprocessing.Queue mesh, verify them against a reference implementation, and the rest of the track becomes plumbing.

Type: Build Languages: Python Prerequisites: Phase 19 Track C lessons 42-49 Time: ~90 min

Objective

Learning objectives

Implement ring allreduce in two passes (reduce-scatter then allgather) and prove the per-rank communication volume is 2(N-1)/N bytes per element.
Build broadcast, allgather, and reduce_scatter on top of point-to-point sends over multiprocessing.Queue.
Verify every primitive against a torch.distributed gloo reference for the same input.
Defend the choice of ring versus tree on cluster shape, latency floor, and bandwidth ceiling.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.

Key terms in this lesson

Latency