Cross-Validation Strategies

Definition

Why trust one train-test split when a different random split could change the score?

Cross-validation estimates performance by repeatedly splitting the data into training and validation parts and averaging the scores. k-fold uses k equal folds; stratified k-fold keeps class proportions in each fold; group k-fold keeps all rows of a group (such as a patient) in the same fold; time-series split always validates on data later than the training data; nested cross-validation separates tuning from evaluation.

In simple words

With 5-fold cross-validation, every row is used for validation exactly once and for training four times. The average of five scores is more reliable than one, and their spread tells you how much luck was involved. The right way to split depends on how the model will be used.

Why it matters

The wrong strategy leaks information and overstates performance: random splits on time series, or patients split across folds. Choosing the strategy is part of designing an honest experiment.

How it works

Strategy	Use when	scikit-learn
k-fold	Independent rows, regression	KFold
Stratified k-fold	Classification, especially imbalanced	StratifiedKFold
Group k-fold	Several rows per person, device or store	GroupKFold
Time-series split	Forecasting; order matters	TimeSeriesSplit
Nested CV	Honest estimate after tuning	GridSearchCV inside cross_val_score

The maths

The cross-validation estimate and its spread

score = \frac{1}{k} i = 1 \sum k s_{i}, SE \approx \frac{sd ( s _{1} , \dots , s _{k} )}{k}

In code

Which rows each strategy puts in validation, on a tiny dataset, and a nested cross-validation estimate.

cv_strategies.py

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import GridSearchCV, GroupKFold, KFold, StratifiedKFold, TimeSeriesSplit, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X = np.arange(12).reshape(-1, 1)
y = np.array([0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1])
groups = np.repeat(["A", "B", "C", "D"], 3)
for name, splitter in [("KFold", KFold(3)), ("Stratified", StratifiedKFold(3)), ("Group", GroupKFold(4)), ("TimeSeries", TimeSeriesSplit(3))]:
    folds = [val.tolist() for _, val in splitter.split(X, y, groups)]
    print(f"{name:>10} validation rows: {folds}")

Xb, yb = load_breast_cancer(return_X_y=True)
inner = GridSearchCV(make_pipeline(StandardScaler(), SVC()), {"svc__C": [0.1, 1, 10]}, cv=3)
scores = cross_val_score(inner, Xb, yb, cv=StratifiedKFold(5, shuffle=True, random_state=0))
print(f"Nested CV accuracy: {scores.mean():.3f} +/- {scores.std() / np.sqrt(len(scores)):.3f}")

Output

     KFold validation rows: [[0, 1, 2, 3], [4, 5, 6, 7], [8, 9, 10, 11]]
Stratified validation rows: [[0, 1, 2, 8], [3, 4, 5, 9], [6, 7, 10, 11]]
     Group validation rows: [[9, 10, 11], [6, 7, 8], [3, 4, 5], [0, 1, 2]]
TimeSeries validation rows: [[3, 4, 5], [6, 7, 8], [9, 10, 11]]
Nested CV accuracy: 0.977 +/- 0.003

Try it yourself

Which strategy?

Match each problem to the cross-validation strategy it needs.

Forecast next week's sales from daily history
Detect a disease with 5 per cent positives
Ten scans per patient, predicting new patients
Report an honest score after tuning hyperparameters

Common mistakes

Shuffling time-series data before splitting.
Reporting the best inner-loop score of a grid search as the final estimate.
Ignoring the spread of fold scores.

Check yourself

1. In 5-fold CV, how often is each row used for validation?

2. Why use GroupKFold for patients?

3. What does nested cross-validation protect against?

Summary

Cross-validation averages scores over several splits.
Match the strategy to the data: stratified, grouped or time-ordered.
Use nested CV for an honest score after tuning.

Read it. Now try it yourself.

Open the interactive activity

Finished reading?

Sources: scikit-learn: Cross-validation · James et al., An Introduction to Statistical Learning