Phase 0: Neural Network Foundations · ~120 minutes · Python

Backpropagation from Scratch

Backpropagation is the algorithm that makes learning possible. Without it, neural networks are just expensive random number generators.

Hiring signal: Understanding of backpropagation from scratch internals

What you will learn

Implement a Value-based autograd engine that builds a computational graph and computes gradients via topological sort
Derive the backward pass for addition, multiplication, and sigmoid using the chain rule
Train a multi-layer network on XOR and circle classification using only your from-scratch backpropagation engine
Identify the vanishing gradient problem in deep sigmoid networks and explain why gradients shrink exponentially

Introduction

Type: Build Languages: Python Prerequisites: Lesson 03.02 (Multi-Layer Networks) Time: ~120 minutes

Learning Objectives

Implement a Value-based autograd engine that builds a computational graph and computes gradients via topological sort
Derive the backward pass for addition, multiplication, and sigmoid using the chain rule
Train a multi-layer network on XOR and circle classification using only your from-scratch backpropagation engine
Identify the vanishing gradient problem in deep sigmoid networks and explain why gradients shrink exponentially

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build It, Use It, Ship It, Exercises, Key Terms, Further Reading — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.

Key terms in this lesson

Backpropagation