Phase 1: Policy Gradient Methods · ~75 minutes · Python

Policy Gradient — REINFORCE from Scratch

Stop estimating value. Parameterize the policy directly, compute the gradient of expected return, step uphill. Williams (1992) wrote it in one theorem. It is why PPO, GRPO, and every LLM RL loop exist.

What you will learn

Implement policy gradient — reinforce from scratch from scratch
Understand the math and intuition behind the algorithm
Use production libraries for the same task
Ship a reusable artifact

Introduction

Type: Build
Languages: Python
Prerequisites: Backpropagation from Scratch (Deep Learning from Scratch), Monte Carlo Methods — Learning from Complete Episodes, Temporal Difference — Q-Learning & SARSA
Time: ~75 minutes

The Problem

Q-learning and DQN parameterize the value function. You pick actions by argmax Q. That is fine for discrete actions and discrete states. It breaks when actions are continuous (which argmax over a 10-dimensional torque?) or when you want a stochastic policy (argmax is deterministic by construction).

Policy gradients parameterize the policy instead. π_θ(a | s) is a neural net that outputs a distribution over actions. Sample from it to act. Compute the gradient of expected return with respect to θ. Step uphill. No argmax. No Bellman recursion. Just gradient ascent on J(θ) = E_{π_θ}[G].

The REINFORCE theorem (Williams 1992) tells you this gradient is computable: ∇J(θ) = E_π[ G · ∇_θ log π_θ(a | s) ]. Run an episode. Compute the return. Multiply by ∇ log π_θ(a | s) at every step. Average. Gradient-ascent. Done.

Every LLM-RL algorithm in 2026 — PPO, DPO, GRPO — is a refinement of REINFORCE. Understanding it in your fingers is the prerequisite for the rest of this phase, and for RLHF: Reward Model + PPO (Transformers & LLMs from Scratch) (RLHF implementation) and DPO: Direct Preference Optimization (Transformers & LLMs from Scratch) (DPO).

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.

Key terms in this lesson

Softmax