Phase 1: Training Neural Networks · ~75 minutes · Python

Optimizers

Gradient descent tells you which direction to move. It says nothing about how far or how fast. SGD is a compass. Adam is GPS with traffic data.

Hiring signal: Understanding of optimizers internals

What you will learn

Implement SGD, SGD with momentum, Adam, and AdamW optimizers from scratch in Python
Explain how Adam's bias correction compensates for zero-initialized moment estimates in early training steps
Demonstrate why AdamW produces better generalization than Adam with L2 regularization on the same task
Select the appropriate optimizer and default hyperparameters for transformers, CNNs, GANs, and fine-tuning

Introduction

Type: Build Languages: Python Prerequisites: Lesson 03.05 (Loss Functions) Time: ~75 minutes

Learning Objectives

Implement SGD, SGD with momentum, Adam, and AdamW optimizers from scratch in Python
Explain how Adam's bias correction compensates for zero-initialized moment estimates in early training steps
Demonstrate why AdamW produces better generalization than Adam with L2 regularization on the same task
Select the appropriate optimizer and default hyperparameters for transformers, CNNs, GANs, and fine-tuning

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build It, Use It, Ship It, Exercises, Key Terms, Further Reading — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.