Phase 1: Policy Gradient Methods · ~75 minutes · Python

Deep Q-Networks (DQN)

2013: Mnih trained one Q-learning network on raw pixels, beat every classical RL agent on seven Atari games. 2015: extended to 49 games, published in Nature, sparked the deep-RL era. DQN is Q-learning plus three tricks that make function approximation stable.

What you will learn

Implement deep q-networks (dqn) from scratch
Understand the math and intuition behind the algorithm
Use production libraries for the same task
Ship a reusable artifact

Introduction

Type: Build
Languages: Python
Prerequisites: Backpropagation from Scratch (Deep Learning from Scratch), Temporal Difference — Q-Learning & SARSA
Time: ~75 minutes

The Problem

Tabular Q-learning needs a separate Q-value for every (state, action) pair. A chess board has ~10⁴³ states. An Atari frame is 210×160×3 = 100,800 features. Tabular RL dies at thousands of states, let alone billions.

The fix is obvious in hindsight: replace the Q-table with a neural network, Q(s, a; θ). But obvious-in-hindsight took decades. Naive function approximation with Q-learning diverges under the "deadly triad" — function approximation + bootstrapping + off-policy learning. Mnih et al. (2013, 2015) identified three engineering tricks that stabilize learning:

Experience replay decorrelates transitions.
Target network freezes the bootstrap target.
Reward clipping normalizes gradient magnitudes.

DQN on Atari was the first time a single architecture with a single hyperparameter set solved dozens of control problems from raw pixels. Everything "deep-RL" built since — DDQN, Rainbow, Dueling, Distributional, R2D2, Agent57 — is stacked on top of this three-trick base.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.