Phase 2: Advanced Training Techniques · ~90 minutes · Python

Weight Initialization and Training Stability

Initialize wrong and training never starts. Initialize right and 50 layers train as smoothly as 3.

What you will learn

Implement zero, random, Xavier/Glorot, and Kaiming/He initialization strategies and measure their effect on activation magnitudes through 50 layers
Derive why Xavier init uses Var(w) = 2/(fan_in + fan_out) and Kaiming uses Var(w) = 2/fan_in
Demonstrate the symmetry problem with zero initialization and explain why random scale alone is insufficient
Match the correct initialization strategy to the activation function: Xavier for sigmoid/tanh, Kaiming for ReLU/GELU

Introduction

Type: Build
Languages: Python
Prerequisites: Activation Functions, Regularization
Time: ~90 minutes

Objective

Learning objectives

Implement zero, random, Xavier/Glorot, and Kaiming/He initialization strategies and measure their effect on activation magnitudes through 50 layers
Derive why Xavier init uses Var(w) = 2/(fan_in + fan_out) and Kaiming uses Var(w) = 2/fan_in
Demonstrate the symmetry problem with zero initialization and explain why random scale alone is insufficient
Match the correct initialization strategy to the activation function: Xavier for sigmoid/tanh, Kaiming for ReLU/GELU

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.