Glossary › Training

Gradient Descent

Optimization algorithm that iteratively updates parameters in the direction opposite to the gradient of the loss: θ = θ - lr × ∂L/∂θ. Variants: SGD, Adam, AdamW.

Where this is taught

• Gradient Descent & Optimization — Phase 2: Classical Machine Learning
• Linear Regression from Scratch — Phase 2: Classical Machine Learning
• Optimization Foundations: Why Training Neural Networks Is Hard — Phase 0: Development Environment & Mathematical Foundations
• Calculus for Machine Learning — Phase 1: Calculus & Optimization
• Convex Optimization — Phase 1: Calculus & Optimization
• Linear Regression — Phase 0: ML Foundations

Related terms

Learning Rate · Backpropagation

More in Training

Backpropagation · Batch Size · Fine-Tuning · LoRA (Low-Rank Adaptation) · RLHF · DPO (Direct Preference Optimization) · QLoRA · Learning Rate

Learn this by building

DeVenture Academy teaches AI engineering through projects, with hands-on labs you run on your own machine using real tools.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card. Or see Pro pricing.

All courses · Pricing · About · FAQ · Glossary