Phase 1: Policy Gradient Methods · ~75 minutes · Python

Actor-Critic — A2C and A3C

REINFORCE is noisy. Add a critic that learns `V̂(s)`, subtract it from the return, and you get an advantage that has the same expectation but far lower variance. That is actor-critic. A2C runs it synchronously; A3C runs it across threads. Both are the mental model for every modern deep-RL method.

Hiring signal: Understanding of actor-critic — a2c and a3c internals

What you will learn

Implement actor-critic — a2c and a3c from scratch
Understand the math and intuition behind the algorithm
Use production libraries for the same task
Ship a reusable artifact

Introduction

Type: Build Languages: Python Prerequisites: Phase 9 · 04 (TD Learning), Phase 9 · 06 (REINFORCE) Time: ~75 minutes

The Problem

Vanilla REINFORCE works, but its variance is terrible. Monte Carlo returns G_t can swing over a factor of 10 between episodes. Multiplying that noise by ∇ log π and averaging produces a gradient estimator that takes thousands of episodes to move the policy the same distance you could move it with far fewer DQN updates.

The variance comes from using raw returns. If you subtract a baseline b(s_t) — any function of state, including a learned value — the expectation is unchanged and the variance drops. The best tractable baseline is V̂(s_t). Now the quantity multiplying ∇ log π is the advantage:

A(s, a) = G - V̂(s)

An action is good if it produced above-average return; bad if below. REINFORCE with a learned critic is actor-critic. The critic gives the actor a low-variance teacher. This is every deep-policy method after 2015 (A2C, A3C, PPO, SAC, IMPALA).

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Concept, Build It, Pitfalls, Use It, Ship It, Exercises, Key Terms, Further Reading — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.