Phase 1: Pretraining at Scale · ~90 minutes · Python

Gradient Accumulation

Train at an effective batch you cannot afford, one micro-batch at a time. Scale the loss, hold the optimizer step, and let the gradients pile up.

Hiring signal: Can build gradient accumulation end to end

What you will learn

Derive the effective batch identity: `effective_batch = micro_batch * accum_steps`.
Implement loss-per-micro-batch scaling so the accumulated gradient matches a single full-batch backward.
Skip optimizer synchronization until the last micro-batch (sync-on-last-step).
Read a throughput against effective batch curve and explain the diminishing return.

Introduction

Type: Build Languages: Python Prerequisites: Phase 19 lessons 42 to 45 Time: ~90 minutes

Objective

Learning objectives

Derive the effective batch identity: effective_batch = micro_batch * accum_steps.
Implement loss-per-micro-batch scaling so the accumulated gradient matches a single full-batch backward.
Skip optimizer synchronization until the last micro-batch (sync-on-last-step).
Read a throughput against effective batch curve and explain the diminishing return.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.