Phase 2: Classical Machine Learning · 60 min · Python · NumPy · Matplotlib

Gradient Descent & Optimization

Every model that learns is just gradient descent wearing a costume.

Hiring signal: Can explain and debug training dynamics and optimizer choice

What you will learn

Derive and implement batch, stochastic, and mini-batch gradient descent
Explain how learning rate, batch size, and momentum affect convergence
Implement Momentum, RMSProp, and Adam from scratch
Diagnose common training failures from loss curves

The Problem

You have a model with parameters and a loss function that measures how wrong it is. Training means finding the parameters that make the loss as small as possible. For almost every modern model, there is no closed-form solution — the loss surface is too complex. So we do the next best thing: we start somewhere, look at which direction reduces the loss fastest, take a step, and repeat.

That iterative procedure is gradient descent. Understanding it deeply is the difference between an engineer who can debug a model that "won't train" and one who just changes random numbers and hopes.

The Concept

The gradient ∇L is the vector of partial derivatives of the loss with respect to each parameter. It points in the direction of steepest increase. To minimize, we step in the opposite direction:

θ_new = θ_old - η · ∇L(θ_old)

Where:
  θ = parameters (weights, biases)
  η = learning rate (step size)
  ∇L = gradient of the loss w.r.t. parameters

You're training a model and the loss goes: 2.3, 2.1, 1.9, 1.8, 1.8, 1.8, 1.8... It plateaus quickly and never reaches the known minimum of 0.5. What's the most likely cause?

The loss decreases steadily but plateaus far from the minimum — classic sign of a learning rate that's too small. The steps are too tiny to escape the flat region. Try increasing the learning rate by 10x or using a scheduler with warmup.

Three flavors of "how much data per step"

Batch GD       : use ALL samples per update   -> stable, slow, memory-heavy
Stochastic GD  : use 1 sample per update      -> noisy, fast, can escape minima
Mini-batch GD  : use a small batch (32-512)   -> the practical default

Mini-batch is what essentially every real system uses: it balances the stability of batch with the speed and regularizing noise of stochastic.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Evaluation, Exercises, Key Terms, Common Pitfalls, Interview Framing — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.