Glossary › Training

Gradient Accumulation

Technique to simulate larger batch sizes by accumulating gradients over multiple mini-batches before updating weights. Effective batch = mini_batch × accumulation_steps.

Where this is taught

• Gradient Accumulation — Phase 1: Pretraining at Scale

Related terms

Batch Size · Gradient Descent

More in Training

Backpropagation · Batch Size · Fine-Tuning · Gradient Descent · LoRA (Low-Rank Adaptation) · RLHF · DPO (Direct Preference Optimization) · QLoRA

Learn this by building

DeVenture Academy teaches AI engineering through projects, with hands-on labs you run on your own machine using real tools.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card. Or see Pro pricing.

All courses · Pricing · About · FAQ · Glossary