Phase 1: AI & ML Literacy for PMs · 45 min · Anthropic Transparency Hub · MMLU · Python
The Problem
A PM is choosing the model behind a new "summarize this support ticket" feature. The team pulls up a public leaderboard: Model A tops MMLU and a popular coding benchmark by a comfortable margin over Model B. The decision looks easy — ship Model A, cite the benchmark in the launch doc, move on. Six weeks later, support agents are complaining that summaries drop the one detail that actually matters (the customer's account tier, buried in message three of a nine-message thread) and that the tone reads as curt in a way that generates complaints. A quick side-by-side shows Model B — the "lower-ranked" one — handles both problems better on this specific task.
Nothing about the benchmark was wrong. MMLU measured what MMLU measures: broad multiple-choice knowledge across academic subjects, scored for accuracy. It never claimed to measure long-thread context retention, tone calibration for a support audience, or anything else this product actually needs. The team didn't misread the benchmark — they asked it a question it was never built to answer, and treated the answer as if it were.
This happens constantly because leaderboard position is legible and an eval built on your own product data is work. This lesson is about closing that gap: what benchmarks like MMLU actually measure, how to read a model card for what it discloses and what it conspicuously omits, how benchmark scores get inflated without real capability gains, and why the only benchmark that ultimately matters is the one built on your own use case.
What MMLU-style benchmarks actually measure
MMLU (Massive Multitask Language Understanding) is a multiple-choice test spanning 57 subjects — elementary math, US history, law, medicine, professional accounting, and more. A model's MMLU score is the percentage of questions it answers correctly. Benchmarks in this family (MMLU-Pro, GPQA, HellaSwag, and most coding benchmarks like HumanEval) share a shape: a fixed set of questions or tasks, a scorable correct answer, and a single aggregate number.
That shape is exactly what makes them useful for one narrow purpose — comparing raw knowledge recall and general reasoning across models on a standardized scale — and exactly what makes them useless for most product decisions. Here's what they don't measure:
- Your product's actual task. MMLU tests "does the model know this fact and pick the right multiple-choice answer." It says nothing about summarization quality, multi-turn coherence, structured extraction from messy real-world text, or any generative task where there's no single correct answer to grade against.
- Tone and format fit. A model can be highly accurate and still write in a voice that's wrong for your brand, too verbose for your UI, or too terse for your users.
- Latency and cost. The top-ranked model on a leaderboard is often the largest and slowest. A benchmark score never tells you the P95 latency or the per-token cost you'll actually pay at your volume.
- Safety and failure behavior on your domain. A model's general safety training doesn't guarantee it handles your specific edge cases — a healthcare-adjacent question, a legal disclaimer, an angry customer — the way your product needs it to.
None of this makes benchmarks worthless. It makes them a first filter, not a decision. A model that scores badly on MMLU is probably weak at broad reasoning, which is genuinely useful signal to rule something out early. A model that scores well on MMLU has cleared a low bar for "generally capable," which tells you almost nothing about whether it's right for your specific task.
How to read a model card
A model card (Anthropic and OpenAI both publish them — see the resources below) is the vendor's own disclosure document. Treat it the way you'd treat any vendor-authored document: read for what's stated, and just as carefully for what's missing.
- Intended uses and out-of-scope uses. This is usually the most honest section, because it protects the vendor legally. If a card says "not intended for high-stakes medical, legal, or financial decisions without human review," that's a real constraint — not boilerplate to skim past. Check whether your use case falls inside or outside the stated scope.
- Training data description. Look for the training data cutoff date and a general description of sources. This tells you whether the model can know about anything that happened after that date — critical if your product touches current events, recent APIs, or a fast-moving domain. Vendors rarely disclose exact datasets (competitive and legal reasons), so expect this section to be vaguer than the others.
- Evaluation results. The benchmark scores themselves, plus — if you're lucky — a methodology note on how they were run. Check whether the evals are the vendor's own internal suite or third-party/independent. Self-reported evals on benchmarks the vendor chose are a weaker signal than independently reproduced ones.
- Known limitations and risks. This section is where "what's conspicuously absent" matters most. A card that lists three limitations for a general-purpose model is probably not being exhaustive — compare against what other vendors disclose for similar models, and treat silence on a risk category (bias, multilingual performance, tool-use reliability) as unknown, not as "not a problem."
- Safety testing summary. Look for what testing was done (red-teaming, third-party audits, specific risk categories evaluated) and by whom. A vague "we conducted extensive safety testing" with no specifics is disclosure in name only.
Go pull up a real one — Anthropic's transparency hub or a specific model card — and read it section by section using this list before you finish this lesson. The gap between a card that says "we evaluated for X" with a methodology link and one that just asserts safety is the whole point of this exercise.
model cards are written by the vendor, for the vendor's incentives
A model card is simultaneously a genuine transparency effort and a document written by the party with every incentive to look good. Both things are true at once. Read the intended-use and limitations sections as the most reliable parts — they exist partly to limit vendor liability, so they tend to be conservative and specific. Read the evaluation section as the least independently verified part unless it cites third-party results. The skill isn't distrust; it's calibrating how much weight each section can bear before you build a product decision on top of it.
Benchmark contamination and gaming
A high benchmark score can mean real capability — or it can mean the benchmark leaked into training, or the model was specifically tuned to perform well on that exact test. Two distinct failure modes worth knowing apart:
Contamination happens when benchmark questions (or close paraphrases) end up in a model's training data, usually unintentionally — the questions were scraped from the public web, and so was the training corpus. A contaminated model isn't reasoning its way to the answer; it's closer to recalling it. This inflates the score without reflecting real generalization, and it's genuinely hard for outsiders to detect or rule out, since vendors rarely disclose exact training sources.
Benchmark gaming (also called "teaching to the test") happens when a lab optimizes specifically for a popular benchmark — including benchmark-style examples in fine-tuning, or tuning decoding strategies to match the benchmark's exact answer format — in a way that moves the score without moving general capability. This is why a model can top a leaderboard and still disappoint on tasks that are only slightly different in shape from the benchmark's format.
Neither failure mode requires bad faith. Both are structural risks of any fixed, public, reused benchmark. The practical takeaway: a public leaderboard number is a hypothesis about capability, not proof of it — and it degrades over time as more labs train on data that includes the benchmark.
A vendor's launch page says their new model "achieves state-of-the-art performance on MMLU, beating the previous leader by 2 points." You're deciding whether this model should power your product's document-classification feature. What should you ask before this claim moves your decision at all?
A 2-point MMLU lead tells you almost nothing about document-classification performance on your specific documents — MMLU is multiple-choice academic knowledge, not classification on your domain. Before this claim influences a real decision, you'd want to know if the eval was independently verified (self-reported numbers deserve more skepticism), what exactly was measured, and — the only claim that actually answers your question — how it performs on your own task-specific eval set. Cost matters too, but only after task fit is established, not instead of it.
Why leaderboard position isn't your answer
Put the last three sections together and the conclusion is unavoidable: a benchmark leaderboard tells you how models rank on a fixed, public, often-contaminated test of general knowledge. It does not tell you how a model performs on your product's actual task, with your actual data, against your actual users' expectations for tone, latency, and correctness. Two models can be a few points apart on MMLU and ten points apart on the task you actually care about — in either direction.
This is the setup for a theme that gets built out fully in Phase 5 of this course: the only benchmark that really settles a model-selection question is an eval built from your own product's task and data — a representative sample of real inputs, scored against a rubric or ground truth that reflects what "good" means for your product, not what "good" means for a 57-subject academic quiz. Public benchmarks are a fast, cheap first filter to narrow a shortlist. A task-specific eval is what you actually build a launch decision on.
Two candidate models are both being considered for a "flag risky clauses in a contract" feature. Model A scores 3 points higher on MMLU and 5 points higher on a public coding benchmark. Model B scores lower on both public benchmarks but was tested by the team on 200 real contracts from their own document set, where it correctly flagged 91% of clauses a legal reviewer had independently flagged, versus 78% for Model A. Which model should the team choose, and why?
MMLU measures general academic knowledge and the coding benchmark measures code generation — neither measures the ability to identify risky contract clauses, which is a specialized extraction and judgment task. Model B's 91% vs. 78% result comes from an eval built on the product's own data and its own definition of "correct" (matching an independent legal reviewer's flags), which is exactly the kind of task-specific evidence this lesson argues is the only benchmark that actually settles a model-selection question. Choosing Model A on the strength of unrelated public benchmarks, while ignoring a large, directly relevant performance gap on the real task, is precisely the mistake the lesson's opening scenario describes.