The Concept
Reading the Loss Curves: Three Regimes
Nearly every training problem falls into one of three regimes, and you can diagnose which one by reading the training and validation loss curves:
train loss val loss diagnosis fix
Underfitting high high too little capacity / LR too low
/ not trained enough
Overfitting low high (rising) memorizing -> regularize / more data
Good fit low low (tracks train)
LR too high spikes/NaN diverges unstable -> lower LR / clip / warmup
Underfitting means the model doesn't have enough capacity (too few parameters) or hasn't trained long enough to capture the pattern. Both training and validation loss are high and similar. Adding more data won't help — the model can't represent the pattern regardless. Fix: increase model capacity (more layers, wider layers), train longer, or reduce regularization.
Overfitting means the model has too much capacity and is memorizing the training data, including its noise. Training loss is low, but validation loss is high and often rising. The gap between train and val loss is the key signal. Fix: regularize (dropout, weight decay, early stopping), get more data, or reduce model capacity.
Good fit means both losses are low and track each other. The model has learned the pattern without memorizing the noise. This is the goal.
Why Regularization Works: The Intuition
Regularization adds a penalty for complexity to the training process. The core idea is a trade-off: you accept slightly worse training performance in exchange for better generalization. The model is forced to learn the dominant patterns in the data rather than memorizing every detail, including noise.
Dropout randomly zeros out a fraction of neurons during training. This prevents co-adaptation — neurons can't rely on any specific other neuron being present, so each must learn useful features independently. At test time, all neurons are active, giving an ensemble-like effect. The intuition: dropout is like training many sub-networks simultaneously and averaging their predictions at test time.
Weight decay (L2 regularization) adds a penalty proportional to the squared magnitude of weights to the loss function. This pushes weights toward zero, preventing any single feature from dominating. The intuition: large weights mean the model is very sensitive to small changes in input — weight decay encourages smoother, more robust decision boundaries.
Early stopping monitors validation loss and stops training when it starts rising. The intuition: during training, the model first learns general patterns (both train and val loss decrease), then starts memorizing noise (train loss keeps decreasing but val loss rises). Early stopping catches the transition point.
Batch normalization normalizes activations within each mini-batch to have zero mean and unit variance. This stabilizes training by keeping activations in a reasonable range, allows higher learning rates, and acts as a mild regularizer (the batch-level noise adds robustness).
The single most important hyperparameter is the learning rate — too high diverges, too low crawls or gets stuck. A good learning rate schedule (warmup + cosine decay) often matters more than the exact starting value.
Training loss: 0.12. Validation loss: 0.45 and rising. What's the diagnosis and what single fix should you try first?
Train loss is low (model fits training data) but val loss is high and rising (model fails to generalize) — classic overfitting. The model is memorizing. First fix: add dropout (0.1-0.3), weight decay (1e-2), or early stopping. More training data or augmentation also helps.