Phase 0: RL Foundations · ~75 minutes · Python
Temporal Difference — Q-Learning & SARSA
Monte Carlo waits until the episode ends. TD updates after every step by bootstrapping the next value estimate. Q-learning is off-policy and optimistic; SARSA is on-policy and cautious. Both are one line of code. Both underpin every deep-RL method in this phase.
Hiring signal: Understanding of temporal difference — q-learning & sarsa internals
What you will learn
- Implement temporal difference — q-learning & sarsa from scratch
- Understand the math and intuition behind the algorithm
- Use production libraries for the same task
- Ship a reusable artifact
Introduction
Type: Build Languages: Python Prerequisites: Phase 9 · 01 (MDPs), Phase 9 · 02 (Dynamic Programming), Phase 9 · 03 (Monte Carlo) Time: ~75 minutes
The Problem
Monte Carlo works but it has two expensive demands. It needs episodes that terminate, and it only updates after the final return is in. If your episode is 1,000 steps, MC waits 1,000 steps to update anything. It is high-variance, low-bias, and slow in practice.
Dynamic programming has the opposite profile — zero-variance bootstrapped backups — but requires a known model.
Temporal difference (TD) learning splits the difference. From a single transition (s, a, r, s'), form a one-step target r + γ V(s') and nudge V(s) toward it. No model. No complete episodes. Bias from using an approximate V on the RHS, but dramatically lower variance than MC and online updates from step one.
This is the pivot on which all of modern RL — DQN, A2C, PPO, SAC — turns. The rest of Phase 9 is layers of function approximation and tricks built on top of the one-step TD update you will write in this lesson.
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers The Concept, Build It, Pitfalls, Use It, Ship It, Exercises, Key Terms, Further Reading — plus a hands-on lab, quiz, and project artifact.
Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.
Browse all courses · View pricing · DeVenture Academy