Phase 1: Pretraining at Scale · ~90 minutes · Python

Language Model Evaluation Harness

A model that does well on a task you cannot define is a model that does well by accident. The harness is the task definition, the metric, the runner, and the leaderboard, in one short, swappable shape.

Hiring signal: Can build language model evaluation harness end to end

What you will learn

Define a task as a JSONL file with `prompt`, `targets`, `metric`, and optional `extras` per example.
Implement five metrics: exact match, rouge-l F1, executable check, multiple choice, and substring contains.
Build a runner that batches examples per task and dispatches to a swappable model adapter.
Emit a leaderboard JSON with per-task scores, latency, and an overall average that is reproducible.

Introduction

Type: Build Languages: Python Prerequisites: Phase 19 lessons 42 to 45 Time: ~90 minutes

Objective

Learning objectives

Define a task as a JSONL file with prompt, targets, metric, and optional extras per example.
Implement five metrics: exact match, rouge-l F1, executable check, multiple choice, and substring contains.
Build a runner that batches examples per task and dispatches to a swappable model adapter.
Emit a leaderboard JSON with per-task scores, latency, and an overall average that is reproducible.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.

Key terms in this lesson

Latency