Phase 3: Deep Learning Core · 75 min · PyTorch · DistributedDataParallel · NCCL

Distributed Training: DDP, Gradient Sync, and Multi-GPU

One model, many GPUs, one coherent gradient.

Hiring signal: Distributed training is a hard requirement for ML Engineer roles at any company training models at scale. Knowing DDP vs model parallelism, gradient synchronization, and common pitfalls (gradient staleness, NCCL timeouts) signals production experience.

What you will learn

Understand data parallelism vs model parallelism and when to use each
Implement DistributedDataParallel (DDP) training in PyTorch
Understand gradient synchronization via all-reduce and ring-all-reduce
Debug common distributed training failures (NCCL timeouts, gradient mismatch, deadlocks)
Use torchrun for multi-process launch and fault tolerance

The Problem

Your model fits on a single GPU, but training takes 3 days. You have 4 GPUs available. Naively, you could split your data into 4 chunks and train 4 models — but you'd get 4 different models, not one model trained 4x faster.

The challenge: how do you use multiple GPUs to train a single model faster, while ensuring every GPU contributes to the same parameter update?

The Concept

Data Parallelism vs Model Parallelism

Data parallelism: Each GPU has a full copy of the model. Each GPU processes a different batch of data. Gradients are averaged across GPUs before the optimizer step. This is the most common approach — it's what DDP does.

Model parallelism: The model is too large for one GPU, so different layers live on different GPUs. Activations flow GPU-to-GPU. Used for training models like 70B+ parameter LLMs. Tools: Megatron-LM, FSDP, pipeline parallelism.

How DDP Works (Step by Step)

Replicate: Copy the model to every GPU. All GPUs start with identical weights.
Scatter: Split the data batch across GPUs. GPU 0 gets mini-batch 0, GPU 1 gets mini-batch 1, etc.
Forward: Each GPU runs a forward pass on its mini-batch independently.
Backward: Each GPU computes gradients on its mini-batch independently.
All-Reduce: Gradients are synchronized across all GPUs using NCCL's all-reduce operation. Every GPU ends up with the average gradient across all mini-batches.
Optimizer Step: Each GPU updates its local copy of the model with the same averaged gradients. Since all GPUs had identical weights and now have identical gradients, they stay in sync.

You're running DDP with 4 GPUs. GPU 0 processes a batch where the loss is 2.0, GPU 1 gets loss 1.0, GPU 2 gets loss 3.0, GPU 3 gets loss 4.0. After all-reduce, what gradient does each GPU use for the optimizer step?

DDP's all-reduce computes the average gradient across all GPUs. This is critical: if each GPU used its own local gradient, the 4 GPUs would diverge immediately. By averaging, all GPUs take the same optimizer step and stay in sync. The effective batch size is 4x your per-GPU batch size.

Ring All-Reduce: Why It Scales

The naive approach to gradient synchronization would be: every GPU sends its gradients to a master, the master averages them, and sends the result back. This is O(N) in the number of GPUs for the master — it becomes a bottleneck.

Ring all-reduce solves this. GPUs are arranged in a logical ring. Each GPU sends a chunk of its gradient to the next GPU and receives a chunk from the previous one. After 2(N-1) steps, every GPU has the full averaged gradient. The bandwidth is optimally utilized — no single node is the bottleneck.

This is why NCCL (NVIDIA Collective Communications Library) is the backbone of distributed training — it implements ring all-reduce using NVIDIA's NVLink and PCIe topology for maximum throughput.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Ship It, Use It, Common Pitfalls, Sources, Evaluation, Exercises, Key Terms, Interview Framing — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.