Phase 3: LLM Architecture · ~120 minutes · Python

Scaling: Distributed Training, FSDP, DeepSpeed

Your 124M model trained on one GPU. Now try 7 billion parameters. The model doesn't fit in memory. The data takes weeks on a single machine. Distributed training isn't optional at scale. It's the only path forward.

What you will learn

Explain the three types of parallelism (data, tensor, pipeline) and when each is necessary based on model and cluster size
Implement data-parallel training using PyTorch DDP with gradient synchronization across multiple GPUs
Calculate the memory budget for a given model size (weights + optimizer states + gradients + activations) to determine the minimum hardware
Configure FSDP or DeepSpeed ZeRO stages to shard model states across GPUs and fit models that exceed single-GPU memory

Introduction

Type: Build
Languages: Python
Prerequisites: Pre-Training a Mini GPT (124M Parameters)
Time: ~120 minutes

Objective

Learning objectives

Explain the three types of parallelism (data, tensor, pipeline) and when each is necessary based on model and cluster size
Implement data-parallel training using PyTorch DDP with gradient synchronization across multiple GPUs
Calculate the memory budget for a given model size (weights + optimizer states + gradients + activations) to determine the minimum hardware
Configure FSDP or DeepSpeed ZeRO stages to shard model states across GPUs and fit models that exceed single-GPU memory

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.