Phase 2: Advanced Training Techniques · ~90 minutes · Python

Weight Initialization and Training Stability

Initialize wrong and training never starts. Initialize right and 50 layers train as smoothly as 3.

Hiring signal: Understanding of weight initialization and training stability internals

What you will learn

Implement zero, random, Xavier/Glorot, and Kaiming/He initialization strategies and measure their effect on activation magnitudes through 50 layers
Derive why Xavier init uses Var(w) = 2/(fan_in + fan_out) and Kaiming uses Var(w) = 2/fan_in
Demonstrate the symmetry problem with zero initialization and explain why random scale alone is insufficient
Match the correct initialization strategy to the activation function: Xavier for sigmoid/tanh, Kaiming for ReLU/GELU

Introduction

Type: Build Languages: Python Prerequisites: Lesson 03.04 (Activation Functions), Lesson 03.07 (Regularization) Time: ~90 minutes

Learning Objectives

Implement zero, random, Xavier/Glorot, and Kaiming/He initialization strategies and measure their effect on activation magnitudes through 50 layers
Derive why Xavier init uses Var(w) = 2/(fan_in + fan_out) and Kaiming uses Var(w) = 2/fan_in
Demonstrate the symmetry problem with zero initialization and explain why random scale alone is insufficient
Match the correct initialization strategy to the activation function: Xavier for sigmoid/tanh, Kaiming for ReLU/GELU

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build It, Use It, Ship It, Exercises, Key Terms, Further Reading — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.