Phase 3: LLM Architecture · ~90 minutes · Python

Evaluation: Benchmarks, Evals, LM Harness

Goodhart's Law: when a measure becomes a target, it ceases to be a good measure. Every frontier lab games benchmarks. MMLU scores go up while models still can't reliably count the number of R's in "strawberry." The only eval that matters is YOUR eval -- on YOUR task, with YOUR data.

What you will learn

Build a custom evaluation harness that runs multiple-choice and open-ended benchmarks against a language model
Explain why standard benchmarks (MMLU, HumanEval) saturate and fail to differentiate frontier models
Implement task-specific evals with proper metrics: exact match, F1, BLEU, and LLM-as-judge scoring
Design a custom evaluation suite targeting your specific use case rather than relying solely on public leaderboards

Introduction

Type: Build
Languages: Python
Prerequisites: LLMs from Scratch
Time: ~90 minutes

Objective

Learning objectives

Build a custom evaluation harness that runs multiple-choice and open-ended benchmarks against a language model
Explain why standard benchmarks (MMLU, HumanEval) saturate and fail to differentiate frontier models
Implement task-specific evals with proper metrics: exact match, F1, BLEU, and LLM-as-judge scoring
Design a custom evaluation suite targeting your specific use case rather than relying solely on public leaderboards

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.