Glossary › Training

Gradient Accumulation

Technique to simulate larger batch sizes by accumulating gradients over multiple mini-batches before updating weights. Effective batch = mini_batch × accumulation_steps.

Related terms

Batch Size · Gradient Descent

More in Training

Backpropagation · Batch Size · Fine-Tuning · Gradient Descent · LoRA (Low-Rank Adaptation) · RLHF · DPO (Direct Preference Optimization) · QLoRA

Learn this by building

DeVenture Academy teaches AI engineering through projects — every lesson is paired with a hands-on lab you run on your own machine with real tools.

Create a free account — the opening phases of 24 of 30 courses are free, no credit card. Or see Pro pricing.

All courses · Pricing · About · FAQ · Glossary