Phase 4: Training LLMs · ~60 minutes · Python

DualPipe Parallelism

DeepSeek-V3 was trained on 2,048 H800 GPUs with MoE experts scattered across nodes. Cross-node expert all-to-all communication cost 1 GPU-hour of comm for every 1 GPU-hour of compute. GPUs were idle half the time. DualPipe (DeepSeek, Dec 2024) is a bidirectional pipeline that overlaps forward and backward computation with the all-to-all comms they trigger. Bubbles drop, throughput climbs, and the keeping of two model-parameter copies (the "dual" that gives the name) is cheap once Expert Parallelism is already spreading experts across ranks anyway. This lesson is a Learn-type walkthrough of what DualPipe actually does and why Sea AI Lab's DualPipeV refinement drops the 2x parameter cost at the expense of a marginally tighter bubble.

What you will learn

Name the four components of a DualPipe forward-backward chunk and why each one gets its own overlap window.
Explain the pipeline bubble problem at scale, and what "bubble-free" means in practice versus in marketing.
Trace a DualPipe schedule by hand for 8 PP ranks and 16 micro-batches and confirm the forward and reverse streams fill each other's idle slots.
State the tradeoff DualPipeV (Sea AI Lab, 2025) makes: drops the 2x parameter replication at the cost of a slightly larger bubble when Expert Parallelism is inactive.

Introduction

Type: Learn
Languages: Python (stdlib, schedule simulator)
Prerequisites: Scaling: Distributed Training, FSDP, DeepSpeed, Open Models: Architecture Walkthroughs
Time: ~60 minutes

Objective

Learning objectives

Name the four components of a DualPipe forward-backward chunk and why each one gets its own overlap window.
Explain the pipeline bubble problem at scale, and what "bubble-free" means in practice versus in marketing.
Trace a DualPipe schedule by hand for 8 PP ranks and 16 micro-batches and confirm the forward and reverse streams fill each other's idle slots.
State the tradeoff DualPipeV (Sea AI Lab, 2025) makes: drops the 2x parameter replication at the cost of a slightly larger bubble when Expert Parallelism is inactive.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.

Key terms in this lesson

Mixture of Experts (MoE)