Overfitting and Underfitting

Definition

A model scores 99% in training and 60% on new data. What went wrong?

Overfitting is when a model learns the training data too closely, including its noise, so it does poorly on new data. Underfitting is when a model is too simple to capture the real pattern, so it does poorly even on the training data.

In simple words

An overfit student memorised last year's answers, and fails when the questions change. An underfit student barely studied, and fails everything. A good student understood the ideas, and handles new questions.

Why it matters

Generalising to new data is the whole point of ML, so this is the most common problem in the field. Spotting which one you have tells you exactly what to do next.

How it works

	Training score	Validation score	Fix
Underfitting	Low	Low	More features, a more complex model, longer training
Good fit	High	High, and close to training	You're done. Test it.
Overfitting	Very high	Much lower	More data, regularisation, a simpler model, early stopping

In code

Watch the training score climb as the tree grows deeper, while the validation score stops improving.

overfit.py

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

X, y = load_breast_cancer(return_X_y=True)
X_train, X_val, y_train, y_val = train_test_split(X, y, random_state=0)

for depth in (1, 2, 4, 8, None):
    tree = DecisionTreeClassifier(max_depth=depth, random_state=0).fit(X_train, y_train)
    print(f"depth {str(depth):>4}: train {tree.score(X_train, y_train):.3f}   validation {tree.score(X_val, y_val):.3f}")

Output

depth    1: train 0.923   validation 0.881
depth    2: train 0.941   validation 0.937
depth    4: train 0.993   validation 0.902
depth    8: train 1.000   validation 0.881
depth None: train 1.000   validation 0.881

Common mistakes

Celebrating a high training score. It says little by itself.
Adding complexity when the model already overfits.
Collecting more data for an underfit model. It usually won't help, because the model can't capture the pattern anyway.

Check yourself

1. Training 98%, validation 72%. What is this?

2. Training 62%, validation 60%. What should you try?

3. Which helps reduce overfitting?

Summary

Overfitting: great on training, poor on new data.
Underfitting: poor on both.
Compare training and validation scores to tell which.