Phase 3: LLM Architecture · ~90 minutes · Python

RLHF: Reward Model + PPO

SFT teaches the model to follow instructions. But it doesn't teach the model which response is BETTER. Two grammatically correct, factually accurate answers can differ enormously in helpfulness. RLHF is how you encode human judgment into the model's behavior. It's what makes Claude helpful and GPT polite.

What you will learn

Build a reward model that scores response quality from human preference pairs (chosen vs rejected)
Implement the PPO training loop that optimizes a language model policy against the reward model with a KL penalty
Explain why RLHF requires three models (SFT, reward, policy) and how the KL constraint prevents reward hacking
Evaluate the effect of RLHF by comparing response quality before and after preference optimization

Introduction

Type: Build
Languages: Python (with numpy)
Prerequisites: Instruction Tuning (SFT)
Time: ~90 minutes

Objective

Learning objectives

Build a reward model that scores response quality from human preference pairs (chosen vs rejected)
Implement the PPO training loop that optimizes a language model policy against the reward model with a KL penalty
Explain why RLHF requires three models (SFT, reward, policy) and how the KL constraint prevents reward hacking
Evaluate the effect of RLHF by comparing response quality before and after preference optimization

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.