The Problem
Monte Carlo works but it has two expensive demands. It needs episodes that terminate, and it only updates after the final return is in. If your episode is 1,000 steps, MC waits 1,000 steps to update anything. It is high-variance, low-bias, and slow in practice.
Dynamic programming has the opposite profile — zero-variance bootstrapped backups — but requires a known model.
Temporal difference (TD) learning splits the difference. From a single transition (s, a, r, s'), form a one-step target r + γ V(s') and nudge V(s) toward it. No model. No complete episodes. Bias from using an approximate V on the RHS, but dramatically lower variance than MC and online updates from step one.
This is the pivot on which all of modern RL — DQN, A2C, PPO, SAC — turns. The rest of this course is layers of function approximation and tricks built on top of the one-step TD update you will write in this lesson.
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.
Create a free account — the opening phases of 29 of 35 courses are free, no credit card.