Phase 3: LLM Architecture · ~120 minutes · Python

Scaling: Distributed Training, FSDP, DeepSpeed

Your 124M model trained on one GPU. Now try 7 billion parameters. The model doesn't fit in memory. The data takes weeks on a single machine. Distributed training isn't optional at scale. It's the only path forward.

Hiring signal: Understanding of scaling: distributed training, fsdp, deepspeed internals

What you will learn

Explain the three types of parallelism (data, tensor, pipeline) and when each is necessary based on model and cluster size
Implement data-parallel training using PyTorch DDP with gradient synchronization across multiple GPUs
Calculate the memory budget for a given model size (weights + optimizer states + gradients + activations) to determine the minimum hardware
Configure FSDP or DeepSpeed ZeRO stages to shard model states across GPUs and fit models that exceed single-GPU memory

Introduction

Type: Build Languages: Python Prerequisites: Phase 10, Lesson 04 (Pre-Training a Mini GPT) Time: ~120 minutes

Learning Objectives

Explain the three types of parallelism (data, tensor, pipeline) and when each is necessary based on model and cluster size
Implement data-parallel training using PyTorch DDP with gradient synchronization across multiple GPUs
Calculate the memory budget for a given model size (weights + optimizer states + gradients + activations) to determine the minimum hardware
Configure FSDP or DeepSpeed ZeRO stages to shard model states across GPUs and fit models that exceed single-GPU memory

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build It, Use It, Ship It, Exercises, Key Terms, Further Reading — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.