Phase 2: Classical Machine Learning · 55 min · scikit-learn · matplotlib

Metrics & Validation Done Right

The wrong metric makes a useless model look great. Choosing it correctly is half of ML.

Hiring signal: Chooses metrics that match the business and validates without leakage

What you will learn

Choose classification metrics (precision, recall, F1, ROC-AUC, PR-AUC) by context
Read a confusion matrix and tune the decision threshold to costs
Pick regression metrics (MAE, RMSE, R²) appropriately
Design leakage-free cross-validation, including for time series
Avoid the accuracy trap on imbalanced data

The Problem

A fraud model that predicts "not fraud" for everyone is 99.9% accurate and completely worthless. Accuracy is the most over-used, most misleading metric in ML, and "what metric did you optimize and why?" is the question that most reliably exposes whether a candidate actually understands the problem they solved. Picking a metric that matches the business decision — and validating it without leaking the answer — is a skill that separates someone who gets a model to run from someone who knows if it's good.

The Concept

Why Metric Selection Is a Business Decision

The metric you optimize determines what the model is good at. This is not a technical choice — it's a business choice expressed through math. If you optimize accuracy on a fraud dataset where 0.1% of transactions are fraud, the model learns "always predict not fraud" and achieves 99.9% accuracy. It's useless, but the metric says it's excellent. The metric must mirror the cost of being wrong in your specific business context.

For each error type, ask: what does it cost? A false negative in cancer screening means a patient dies. A false positive means a follow-up test. The costs are asymmetric, and your metric should reflect that asymmetry. This is why there is no universally "best" metric — only the one that matches your decision.

Classification             measures...                     use when...
  Precision                of predicted positives, % right  false positives are costly (spam)
  Recall (sensitivity)     of actual positives, % caught     false negatives are costly (cancer, fraud)
  F1                       harmonic mean of P & R            you need a single balanced number
  ROC-AUC                  ranking quality across thresholds balanced classes, ranking
  PR-AUC                   precision/recall tradeoff         RARE positives (imbalanced)
Regression
  MAE                      avg absolute error                robust to outliers, interpretable
  RMSE                     penalizes big errors more         large errors are especially bad
  R²                       variance explained                relative goodness, communication

The Validation Discipline: Preventing Leakage

Choosing the right metric is half the battle. The other half is validating honestly — ensuring your evaluation doesn't accidentally give the model information about the test set. The most common leakage is in preprocessing: fitting a scaler on the full dataset (including test data) before splitting means the training data "saw" the test set's distribution. The model has indirect knowledge of test statistics, inflating your metrics.

The rule: fit preprocessing on training data only, then transform test data with those fitted parameters. Never fit anything on the full dataset. Never tune hyperparameters on the test set. The test set is touched exactly once — for the final evaluation.

You fit a StandardScaler on the full dataset, then split into train/test. Your test accuracy is 97%. Your colleague says the score is inflated. Why?

Fitting the scaler on the full dataset means the training data "saw" the test set's distribution (mean and std). This is data leakage — the model has indirect knowledge of test data through preprocessing. Always fit preprocessing on training data only, then transform test data with those fitted parameters.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Evaluation, Exercises, Key Terms, Common Pitfalls, Interview Framing — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.

Related lessons

Browse all courses · View pricing · DeVenture Academy