Phase 1: Training Neural Networks · ~75 minutes · Python

Activation Functions

Without nonlinearity, your 100-layer network is a fancy matrix multiply. Activations are the gates that let neural networks think in curves.

What you will learn

Implement sigmoid, tanh, ReLU, Leaky ReLU, GELU, Swish, and softmax with their derivatives from scratch
Diagnose the vanishing gradient problem by measuring activation magnitudes through 10+ layers with different activations
Detect dead neurons in a ReLU network and explain why GELU avoids this failure mode
Select the correct activation function for a given architecture (transformer, CNN, RNN, output layer)

Introduction

Type: Build
Languages: Python
Prerequisites: Backpropagation from Scratch
Time: ~75 minutes

Objective

Learning objectives

Implement sigmoid, tanh, ReLU, Leaky ReLU, GELU, Swish, and softmax with their derivatives from scratch
Diagnose the vanishing gradient problem by measuring activation magnitudes through 10+ layers with different activations
Detect dead neurons in a ReLU network and explain why GELU avoids this failure mode
Select the correct activation function for a given architecture (transformer, CNN, RNN, output layer)

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.