Training, Validation and Test Sets

Definition

In simple words

Textbook problems are the training set. Mock tests are the validation set: you use them to change how you study. The board exam is the test set. If you saw the board paper in advance, your score would mean nothing, and the same holds for a model.

Why it matters

A score on data the model has already seen tells you almost nothing. Without a held-out set, you can ship a model that looks perfect and fails on real users.

How it works

Choosing how to split

• Stratified split: keeps class shares equal across sets. Use it for imbalanced classes.
• Time split: for time series, train on the past and test on the future. Never shuffle.
• Group split: keep one person's records in only one set, or the model can recognise that person.

In code

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
import numpy as np

X, y = load_iris(return_X_y=True)

X_rest, X_test, y_rest, y_test = train_test_split(X, y, test_size=0.2, random_state=0, stratify=y)
X_train, X_val, y_train, y_val = train_test_split(X_rest, y_rest, test_size=0.25, random_state=0, stratify=y_rest)

print("Sizes:", len(X_train), len(X_val), len(X_test))
print("Class counts in test:", np.bincount(y_test).tolist())

Sizes: 90 30 30
Class counts in test: [10, 10, 10]

Common mistakes

• Tuning on the test set. If you check it repeatedly, it stops being a test.
• Shuffling time-series data, which lets the model see the future.
• Scaling before splitting.

Check yourself

Summary

• Train to learn, validate to choose, test once to report.
• Split the right way: stratified, by time or by group.
• Fit all preprocessing on training data only.