Phase 2: Classical Machine Learning · 55 min · scikit-learn · pandas · matplotlib

Error Analysis & Model Cards

The model's average score hides where it fails. Error analysis is how you find the next improvement.

Hiring signal: Debugs models by slice, not by leaderboard — and documents them responsibly

What you will learn

Run a structured error analysis on a trained model
Find underperforming data slices and rank failure categories
Distinguish data problems from model problems
Prioritize the next improvement by impact, not novelty
Write a model card documenting use, performance, and limitations

The Problem

You trained a model. It's 88% accurate. Now what? Beginners tune hyperparameters at random hoping the number rises. Strong ML engineers do error analysis: they look at what the model gets wrong, find the slices where it fails, and target the highest-impact fix. This is the single most valuable workflow in applied ML — and "tell me about a time you debugged a model" is a near-guaranteed interview question. The companion skill is documenting the model honestly with a model card, so others know where it's safe to use.

The Concept

The Mindset Shift: From Aggregate to Slice

Leaderboard mindset:  "raise the aggregate metric" -> random tuning, diminishing returns
Error-analysis mindset: "where and why does it fail?" -> targeted, high-impact fixes

The aggregate metric (overall accuracy, F1, AUC) is an average — it hides the distribution of errors. A model with 88% accuracy might be 99% accurate on the majority group and 60% accurate on a minority group that matters most. The aggregate says "good enough"; the slices say "here's where to focus."

Error analysis is the practice of systematically breaking down where the model fails and why. It replaces random hyperparameter tuning with targeted, high-impact fixes. The workflow is always the same two questions:

1. WHERE does it fail?   -> slice the data; find subgroups with worse metrics
2. WHY does it fail?     -> read actual errors; categorize the causes

The Four Categories of Failure

When you pull the actual misclassified examples and read them, failures typically fall into one of four categories:

Mislabeled: the ground truth is wrong. Fix: correct the labels, add label verification.
Edge case: the input is genuinely ambiguous or unusual. Fix: add features or training examples that cover this case.
Model limit: the model architecture can't represent the pattern. Fix: increase capacity (more layers, deeper trees) or switch models.
Data quality: the input features are corrupted, missing, or noisy. Fix: clean the data pipeline.

Once you've categorized the failures, rank them by impact = error_rate × slice_size × business_cost, and fix the highest-impact category first. This is how you go from "88% accuracy" to "95% accuracy on the segments that matter" — not by random tuning, but by understanding exactly what's broken.

Your spam classifier has 95% overall accuracy. But when you slice by email length, short emails (<50 chars) have only 72% accuracy. What should you do next?

The aggregate metric hides a weak slice. The next step is to pull the actual misclassified short emails, read them, and categorize why they fail (missing signal? ambiguous? mislabeled?). Then add targeted features or training examples. This is the core of error analysis: find the worst slice, understand the cause, fix it.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Model Card: [Model Name] v[Version], Use It, Ship It, Evaluation, Exercises, Key Terms, Common Pitfalls, Interview Framing — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.