Phase 0: Development Environment & Mathematical Foundations · 75 min · Python · NumPy · Matplotlib
The Concept
Derivatives: The Slope of a Function
Imagine you're hiking on a hill. At any point, you can ask: "If I take a small step forward, how much does my elevation change?" That rate of change — elevation change per unit of horizontal movement — is the derivative. It's positive when you're going uphill, negative when downhill, and zero at the top of a peak or bottom of a valley.
For a function $f(x) = x^2$, the derivative $f'(x) = 2x$ tells you the slope at every point. At $x = 3$, the slope is 6 — meaning a tiny step in $x$ produces a change in $f(x)$ about 6 times as large. At $x = 0$, the slope is 0 — you're at the bottom of the valley, the minimum.
f(x) = x²
f'(x) = 2x
At x=3: f'(3) = 6 → a small change in x produces ~6x that change in f(x)
At x=0: f'(0) = 0 → flat — this is the minimum
This is the foundation of all model training: the loss function is the "hill," and the derivative tells us which direction is downhill. We adjust parameters to roll toward the valley (minimum loss).
Gradients: Derivatives in Multiple Dimensions
When a function depends on multiple variables — as all neural networks do, with millions of weights — we compute partial derivatives: the rate of change with respect to one variable while holding the others constant. The collection of all partial derivatives is the gradient, written as $\nabla f$:
f(x, y) = x² + 3xy + y²
∂f/∂x = 2x + 3y (partial derivative w.r.t. x)
∂f/∂y = 3x + 2y (partial derivative w.r.t. y)
∇f = (∂f/∂x, ∂f/∂y) = (2x + 3y, 3x + 2y) → the gradient vector
The gradient has a remarkable geometric property: it points in the direction of steepest increase. If you're standing on the loss surface and follow the gradient, you go uphill as fast as possible. To minimize loss — the goal of training — you move in the opposite direction: x_new = x - learning_rate * gradient. This is gradient descent, and it is the optimization engine behind every neural network ever trained.
If your loss is L(w) and the gradient ∇L = [0.5, -1.2], which direction do you move w to decrease the loss?
Gradient descent moves opposite to the gradient: w_new = w - lr * ∇L. So you add [-0.5, 1.2] (scaled by learning rate) to decrease the loss.
The Chain Rule: How Gradients Flow Through Compositions
The chain rule is the single most important mathematical identity in deep learning. It is what makes backpropagation possible, and therefore what makes training deep networks feasible.
The intuition is simple: if you compose two functions — say, $z = f(g(x))$ — then the rate of change of $z$ with respect to $x$ is the product of the local rates of change. If $g$ doubles $x$ and $f$ triples its input, then the composition multiplies by 6. The chain rule generalizes this to any differentiable composition:
dz/dx = dz/dg * dg/dx
In a neural network, the loss is a deep composition of many functions:
loss = loss_fn(activation_3(activation_2(activation_1(x, W1), W2), W3))
dL/dW1 = dL/dact3 * dact3/dact2 * dact2/dact1 * dact1/dW1
Each layer's gradient depends on the gradient from the layer above — gradients flow backward through the network. This backward flow is called backpropagation, and it is just the chain rule applied repeatedly to a computational graph.
Computational Graphs: Making the Chain Rule Automatic
Every differentiable computation can be represented as a directed graph of operations. Each node performs a simple operation (multiply, add, activate) and knows two things: how to compute its output (the forward pass) and how to compute the gradient of its input given the gradient of its output (the backward pass).
Forward pass (left to right):
x=2.0 → [×w] → [+] b → [ReLU] → [×w2] → [+] b2 → loss
Backward pass (right to left):
loss ← d/dw2 ← d/db2 ← d/ReLU ← d/dw ← d/db
The beauty of this representation is that you never need to derive the full gradient by hand. Each node only needs to know its local derivative — the derivative of its output with respect to its immediate inputs. The chain rule handles the rest: multiply the local derivative by the gradient flowing in from upstream, and pass it downstream. This is what frameworks like PyTorch do automatically, and it is called reverse-mode automatic differentiation.
Reverse-mode autodiff is efficient because one forward pass computes all outputs, and one backward pass computes all gradients — regardless of how many parameters the network has. This is why we use reverse-mode (not forward-mode) autodiff: for a function with $n$ inputs and 1 output (which is what a loss function is), reverse-mode needs one backward pass to get all $n$ gradients, while forward-mode would need $n$ passes.