Phase 0: RL Foundations · ~75 minutes · Python
Dynamic Programming — Policy Iteration & Value Iteration
Dynamic programming is RL with cheating. You already know the transition and reward functions; you just iterate the Bellman equation until `V` or `π` stops moving. It is the benchmark every sampling-based method tries to approach.
Hiring signal: Understanding of dynamic programming — policy iteration & value iteration internals
What you will learn
- Implement dynamic programming — policy iteration & value iteration from scratch
- Understand the math and intuition behind the algorithm
- Use production libraries for the same task
- Ship a reusable artifact
Introduction
Type: Build Languages: Python Prerequisites: Phase 9 · 01 (MDPs) Time: ~75 minutes
The Problem
You have an MDP with a known model: you can query P(s' | s, a) and R(s, a, s') for any state-action pair. An inventory manager knows the demand distribution. A board game has deterministic transitions. A gridworld is four lines of Python. You have a model.
Model-free RL (Q-learning, PPO, REINFORCE) was invented for the case where you don't have a model — you can only sample from the environment. But when you do have one, there are faster, better methods: dynamic programming. Bellman designed them in 1957. They still define correctness: when people say "optimal policy for this MDP," they mean the policy DP would return.
You need them in 2026 for three reasons. First, every tabular environment in RL research (GridWorld, FrozenLake, CliffWalking) is solved with DP to produce the gold-standard policy. Second, exact values let you debug sampling methods: if Q-learning's estimate for V*(s_0) disagrees with the DP answer by 30%, your Q-learning has a bug. Third, modern offline RL and planning methods (MCTS, AlphaZero's search, model-based RL in Phase 9 · 10) all iterate a Bellman backup over a learned or given model.
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers The Concept, Build It, Pitfalls, Use It, Ship It, Exercises, Key Terms, Further Reading — plus a hands-on lab, quiz, and project artifact.
Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.
Browse all courses · View pricing · DeVenture Academy