Phase 1: Policy Gradient Methods · ~75 minutes · Python
Proximal Policy Optimization (PPO)
A2C throws away each rollout after one update. PPO wraps the policy gradient in a clipped importance ratio so you can do 10+ epochs on the same data without the policy exploding. Schulman et al. (2017). Still the default policy-gradient algorithm in 2026.
What you will learn
- Implement proximal policy optimization (ppo) from scratch
- Understand the math and intuition behind the algorithm
- Use production libraries for the same task
- Ship a reusable artifact
Introduction
- Type: Build
- Languages: Python
- Prerequisites: Policy Gradient — REINFORCE from Scratch, Actor-Critic — A2C and A3C
- Time: ~75 minutes
The Problem
A2C (Actor-Critic — A2C and A3C) is on-policy: the gradient E_{π_θ}[A · ∇ log π_θ] requires data sampled from the current π_θ. Take one update, and π_θ changes; the data you used is now off-policy. Re-use it and your gradient is biased.
Rollouts are expensive. On Atari, one rollout across 8 envs × 128 steps = 1024 transitions and a dozen seconds of environment time. Throwing that away after one gradient step is wasteful.
Trust Region Policy Optimization (TRPO, Schulman 2015) was the first fix: constrain each update so the KL divergence between old and new policy stays below δ. Theoretically clean, but requires a conjugate-gradient solve per update. Nobody runs TRPO in 2026.
PPO (Schulman et al. 2017) replaces the hard trust-region constraint with a simple clipped objective. One extra line of code. Ten epochs per rollout. No conjugate gradients. Good-enough theoretical guarantees. Nine years later it is still the default policy-gradient algorithm for everything from MuJoCo to RLHF.
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.
Create a free account — the opening phases of 29 of 35 courses are free, no credit card.
Browse all courses · View pricing · DeVenture Academy