Phase 6: Distributed Training · ~90 min · Python

Pipeline Parallel and Bubble Analysis

Tensor parallelism splits the matrix multiply across ranks.

Hiring signal: Can build pipeline parallel and bubble analysis end to end

What you will learn

Split a sequential model into N stages and simulate a forward pipeline across N ranks.
Schedule M microbatches through the pipeline using the GPipe schedule (forward-only fill, then backward) and compute the bubble fraction.
Compare bubble against the interleaved 1F1B schedule used in Megatron-LM and PipeDream.
Defend stage assignment: equal compute per stage matters more than equal parameter count per stage.

Introduction

Pipeline parallelism splits the model across ranks, one stage per rank. Microbatches flow through the pipeline. The empty time at the start and end is the bubble; minimising it is the whole craft.

Type: Build Languages: Python Prerequisites: Phase 19 Track C lessons 42-49 Time: ~90 min

Objective

Learning objectives

Split a sequential model into N stages and simulate a forward pipeline across N ranks.
Schedule M microbatches through the pipeline using the GPipe schedule (forward-only fill, then backward) and compute the bubble fraction.
Compare bubble against the interleaved 1F1B schedule used in Megatron-LM and PipeDream.
Defend stage assignment: equal compute per stage matters more than equal parameter count per stage.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.