Phase 1: Calculus & Optimization · ~75 minutes · Python

Optimization

Training a neural network is nothing more than finding the bottom of a valley.

Hiring signal: Understanding of optimization internals

What you will learn

Implement vanilla gradient descent, SGD with momentum, and Adam from scratch
Compare optimizer convergence on the Rosenbrock function and explain why Adam adapts per-weight learning rates
Distinguish convex from non-convex loss landscapes and explain the role of saddle points in high dimensions
Configure learning rate schedules (step decay, cosine annealing, warmup) for training stability

Introduction

Type: Build Language: Python Prerequisites: Phase 1, Lessons 04-05 (Derivatives, Gradients) Time: ~75 minutes

Learning Objectives

Implement vanilla gradient descent, SGD with momentum, and Adam from scratch
Compare optimizer convergence on the Rosenbrock function and explain why Adam adapts per-weight learning rates
Distinguish convex from non-convex loss landscapes and explain the role of saddle points in high dimensions
Configure learning rate schedules (step decay, cosine annealing, warmup) for training stability

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build It, Use It, Ship It, Exercises, Key Terms, Further Reading — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.