Phase 3: Deep Learning Core · 80 min · Python · NumPy · Matplotlib

Backpropagation & Training from Scratch

If you can't implement backprop manually, you don't understand how your model learns.

Hiring signal: Deep understanding of gradient computation — not just calling .backward()

The Problem

An engineer fine-tunes a model. The loss isn't decreasing. They try: different learning rates, different optimizers, more data. Nothing works. They cannot diagnose the issue because they don't understand what happens during .backward(). Is it a vanishing gradient? An exploding one? A dead ReLU? Without understanding backprop, every training failure is a mystery.

The Concept

What Backpropagation Actually Does

Neural network training has two phases:

FORWARD PASS (compute loss):
  Input → Layer 1 → Layer 2 → ... → Output → Loss

BACKWARD PASS (compute gradients):
  Loss → ∂L/∂output → ∂L/∂layer2 → ∂L/∂layer1 → ∂L/∂input
                                                        ↓
                                                  Update weights

Backpropagation is just the chain rule applied recursively through a computation graph.

You forget to call optimizer.zero_grad() before loss.backward(). What happens?

PyTorch accumulates gradients by default (useful for gradient accumulation techniques). Without zero_grad(), each batch's gradients add to the previous batch's, so the update direction drifts and the loss will likely diverge or go to NaN.

The Chain Rule

If y = f(g(x)), then dy/dx = f'(g(x)) · g'(x)

In a neural network:

loss = L(σ(Wx + b))

∂loss/∂W = ∂loss/∂output × ∂output/∂pre_activation × ∂pre_activation/∂W
         = ∂L/∂σ       × σ'(z)                     × x

Computation Graph

Every operation creates a node. Gradients flow backward through edges:

Forward:   x ─→ [×W] ─→ z ─→ [ReLU] ─→ a ─→ [×W2] ─→ y ─→ [MSE] ─→ loss

Backward:  x ←── W ←── z ←── ReLU' ←── a ←── W2 ←── y ←── 2(y-target) ←── loss
           (gradients flow right to left)

Common Gradient Problems

Problem	Symptom	Cause	Fix
Vanishing	Loss stalls, early layers don't update	Sigmoid/tanh squash gradients	Use ReLU, residual connections
Exploding	Loss becomes NaN	Gradients multiply > 1 repeatedly	Gradient clipping, smaller LR
Dead ReLU	Some neurons output 0 forever	Negative input → 0 gradient	LeakyReLU, careful initialization
Saddle points	Loss plateaus then drops	Flat regions in high-dim loss landscape	Momentum, Adam optimizer

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Evaluation, Exercises, Key Terms, Common Pitfalls, Interview Framing — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.

Key terms in this lesson

Backpropagation