Glossary › Training

Learning Rate

Step size for parameter updates during gradient descent. Too high: diverge. Too low: slow or stuck. Typical: 1e-5 to 1e-3 for fine-tuning. Often uses warmup + cosine decay schedule.

Where this is taught

• Learning Rate Schedules and Warmup — Phase 2: Advanced Training Techniques
• Optimization Foundations: Why Training Neural Networks Is Hard — Phase 0: Development Environment & Mathematical Foundations
• Cosine LR with Linear Warmup — Phase 1: Pretraining at Scale
• Debugging Neural Networks — Phase 3: Frameworks & Debugging
• Gradient Descent & Optimization — Phase 2: Classical Machine Learning
• Optimization — Phase 1: Calculus & Optimization

Related terms

Gradient Descent

More in Training

Backpropagation · Batch Size · Fine-Tuning · Gradient Descent · LoRA (Low-Rank Adaptation) · RLHF · DPO (Direct Preference Optimization) · QLoRA

Learn this by building

DeVenture Academy teaches AI engineering through projects, with hands-on labs you run on your own machine using real tools.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card. Or see Pro pricing.

All courses · Pricing · About · FAQ · Glossary