Phase 3: LLM Architecture · ~90 minutes · Python

DPO: Direct Preference Optimization

RLHF works. It also requires training three models (SFT, reward model, policy), managing PPO's instability, and tuning a KL penalty. DPO asks: what if you could skip all of that? DPO directly optimizes the language model on preference pairs. No reward model. No PPO. One training loop. Same results.

What you will learn

Implement DPO training that directly optimizes a language model on preference pairs without a separate reward model
Derive the DPO loss function and explain how it implicitly represents a reward model through the policy's log probabilities
Compare DPO vs RLHF in terms of training stability, compute cost, and number of models required
Tune the beta parameter to control how far the trained policy diverges from the reference model

Introduction

Type: Build
Languages: Python (with numpy)
Prerequisites: RLHF: Reward Model + PPO
Time: ~90 minutes

Objective

Learning objectives

Implement DPO training that directly optimizes a language model on preference pairs without a separate reward model
Derive the DPO loss function and explain how it implicitly represents a reward model through the policy's log probabilities
Compare DPO vs RLHF in terms of training stability, compute cost, and number of models required
Tune the beta parameter to control how far the trained policy diverges from the reference model

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.