Phase 0: RL Foundations · ~75 minutes · Python

Monte Carlo Methods — Learning from Complete Episodes

Dynamic programming needs a model. Monte Carlo needs nothing but episodes. Run the policy, watch the returns, average them. The simplest idea in RL — and the one that unlocks everything downstream.

What you will learn

Implement monte carlo methods — learning from complete episodes from scratch
Understand the math and intuition behind the algorithm
Use production libraries for the same task
Ship a reusable artifact

Introduction

Type: Build
Languages: Python
Prerequisites: MDPs, States, Actions & Rewards, Dynamic Programming — Policy Iteration & Value Iteration
Time: ~75 minutes

The Problem

Dynamic programming is elegant, but it assumes you can query P(s' | s, a) for every state and action. Almost nothing in the real world works that way. A robot cannot analytically compute the distribution over camera pixels after a joint torque. A pricing algorithm cannot integrate over every possible customer reaction. An LLM cannot enumerate all possible continuations after a token.

You need a method that only needs the ability to sample from the environment. Run the policy. Get a trajectory s_0, a_0, r_1, s_1, a_1, r_2, …, s_T. Use it to estimate values. That is Monte Carlo.

The shift from DP to MC is philosophically important: we move from known model + exact backup to sampled rollouts + averaged return. The variance jumps, but the applicability explodes. Every RL algorithm after this lesson — TD, Q-learning, REINFORCE, PPO, GRPO — is a Monte Carlo estimator at heart, sometimes with bootstrapping layered on top.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.