Phase 3: Advanced Math for AI · ~60 minutes · Python

Information Theory

Information theory measures surprise. Loss functions are built on it.

What you will learn

Compute entropy, cross-entropy, and KL divergence from scratch and explain their relationship
Derive why minimizing cross-entropy loss is equivalent to maximizing log-likelihood
Calculate mutual information between features and a target to rank feature importance
Explain perplexity as the effective vocabulary size a language model chooses from

Introduction

Type: Learn
Language: Python
Prerequisites: Probability and Distributions
Time: ~60 minutes

Objective

Learning objectives

Compute entropy, cross-entropy, and KL divergence from scratch and explain their relationship
Derive why minimizing cross-entropy loss is equivalent to maximizing log-likelihood
Calculate mutual information between features and a target to rank feature importance
Explain perplexity as the effective vocabulary size a language model chooses from

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.