Phase 2: Classical Machine Learning · 50 min · NumPy · matplotlib

Reinforcement Learning: Learning from Reward, Not Labels

There is no label for the right move — only a reward that arrives late and blames the wrong action.

Hiring signal: Explains the third learning paradigm correctly and can trace RLHF back to the same MDP formalism — a standard AI engineer interview question when the conversation turns to alignment

What you will learn

Formalize a sequential decision problem as a Markov Decision Process: states, actions, rewards, transitions, policy
Implement tabular Q-learning from scratch and explain the Bellman update
Apply epsilon-greedy exploration and explain the exploration/exploitation tradeoff
Decide when a problem needs RL versus when supervised learning is the better (cheaper, more stable) choice
Trace how RLHF applies the same MDP formalism to align language models, connecting this lesson to fine-tuning in Phase 4

The Problem

Every algorithm so far in this phase has had one thing in common: a dataset that already tells you the answer. Linear regression has the true house price. Classification has the true class. Even clustering, label-free as it is, hands you a static pile of points to organize — nothing you do changes the data itself.

Reinforcement learning breaks that assumption. An RL agent takes an action, the environment changes because of it, and a reward comes back — sometimes immediately, often many steps later, frequently for a mix of good and bad decisions bundled together. A chess engine doesn't get told "move 14 was wrong"; it finds out 30 moves later that it lost. This is the credit assignment problem: when a reward finally arrives, which of the many actions that led up to it actually deserves the credit (or blame)? There's a second problem sitting on top of it: an agent that only ever repeats its best-known move never discovers a better one — the exploration/exploitation tradeoff.

This isn't an academic curiosity. Every frontier LLM you've used — GPT, Claude, Gemini — was aligned with RLHF (Reinforcement Learning from Human Feedback), and "explain how RLHF works" is a standard AI engineer interview question. You cannot answer it — or reason about why a model behaves the way it does after alignment — without the vocabulary this lesson builds.

The Concept

Reinforcement learning formalizes "an agent acting in a world" as a Markov Decision Process (MDP):

State (S)       : the current situation the agent observes
Action (A)      : a choice the agent can make from that state
Reward (R)      : a scalar signal from the environment after an action
Transition      : P(s' | s, a) — how the state changes given an action
Policy (π)      : the agent's strategy — a mapping from states to actions
Discount (γ)    : how much a future reward is worth today (0 to 1)

The "Markov" part is a specific, load-bearing assumption: the next state depends only on the current state and action — not on the full history that got you there. This is what makes the problem tractable.

A policy is judged by the expected discounted return it produces, not the reward of any single step:

G_t = R_t + γR_{t+1} + γ²R_{t+2} + γ³R_{t+3} + ...

γ close to 1 makes the agent care about long-term consequences; γ close to 0 makes it greedy for immediate reward. This single number is why an RL agent can learn to sacrifice a short-term reward (give up a pawn) for a long-term one (win the game) — something no single-step loss function can express.

An RL agent reaches a goal after 20 steps and receives a reward of +1 at the final step, with a small -0.04 penalty at every other step. With a discount factor γ close to 1, which of these does the algorithm most directly try to maximize?

This is the core idea an MDP formalizes: the agent doesn't optimize any single step in isolation, it optimizes the expected discounted RETURN — the sum of all rewards from now until the episode ends, weighted so that reward further in the future counts slightly less (controlled by γ). That's what lets it trade a small step penalty now for a much larger reward later.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Evaluation, Exercises, Key Terms, Common Pitfalls, Interview Framing — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.