Phase 1: Policy Gradient Methods · ~75 minutes · Python

Deep Q-Networks (DQN)

2013: Mnih trained one Q-learning network on raw pixels, beat every classical RL agent on seven Atari games. 2015: extended to 49 games, published in Nature, sparked the deep-RL era. DQN is Q-learning plus three tricks that make function approximation stable.

Hiring signal: Understanding of deep q-networks (dqn) internals

What you will learn

Implement deep q-networks (dqn) from scratch
Understand the math and intuition behind the algorithm
Use production libraries for the same task
Ship a reusable artifact

Introduction

Type: Build Languages: Python Prerequisites: Phase 3 · 03 (Backpropagation), Phase 9 · 04 (Q-learning, SARSA) Time: ~75 minutes

The Problem

Tabular Q-learning needs a separate Q-value for every (state, action) pair. A chess board has ~10⁴³ states. An Atari frame is 210×160×3 = 100,800 features. Tabular RL dies at thousands of states, let alone billions.

The fix is obvious in hindsight: replace the Q-table with a neural network, Q(s, a; θ). But obvious-in-hindsight took decades. Naive function approximation with Q-learning diverges under the "deadly triad" — function approximation + bootstrapping + off-policy learning. Mnih et al. (2013, 2015) identified three engineering tricks that stabilize learning:

Experience replay decorrelates transitions.
Target network freezes the bootstrap target.
Reward clipping normalizes gradient magnitudes.

DQN on Atari was the first time a single architecture with a single hyperparameter set solved dozens of control problems from raw pixels. Everything "deep-RL" built since — DDQN, Rainbow, Dueling, Distributional, R2D2, Agent57 — is stacked on top of this three-trick base.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Concept, Build It, Pitfalls, Use It, Ship It, Exercises, Key Terms, Further Reading — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.