Phase 1: Training Neural Networks · ~75 minutes · Python

Optimizers

Gradient descent tells you which direction to move. It says nothing about how far or how fast. SGD is a compass. Adam is GPS with traffic data.

What you will learn

Implement SGD, SGD with momentum, Adam, and AdamW optimizers from scratch in Python
Explain how Adam's bias correction compensates for zero-initialized moment estimates in early training steps
Demonstrate why AdamW produces better generalization than Adam with L2 regularization on the same task
Select the appropriate optimizer and default hyperparameters for transformers, CNNs, GANs, and fine-tuning

Introduction

Type: Build
Languages: Python
Prerequisites: Loss Functions
Time: ~75 minutes

Objective

Learning objectives

Implement SGD, SGD with momentum, Adam, and AdamW optimizers from scratch in Python
Explain how Adam's bias correction compensates for zero-initialized moment estimates in early training steps
Demonstrate why AdamW produces better generalization than Adam with L2 regularization on the same task
Select the appropriate optimizer and default hyperparameters for transformers, CNNs, GANs, and fine-tuning

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.