Definition
Why did a model with 99 per cent validation accuracy fail in real use?
Data leakage is any way in which information that would not be available at prediction time reaches the model during training or evaluation, making scores look better than they will be in real use. Validation design is choosing splits that mimic real use, so that the evaluation is honest.
In simple words
Leakage is the model seeing the answers before the exam. A feature recorded after the outcome, such as 'refund issued' when predicting fraud, gives the answer away. Preprocessing or feature selection done on all the data lets test information slip into training. Future data in the training set lets a forecast cheat.
Why it matters
Leakage is the most common reason for models that shine in notebooks and fail in production. It's invisible in the score; you only find it by reasoning about how and when each feature is created.
How it works
| Leak | Example | Prevention |
|---|---|---|
| Target leakage | A 'treatment given' feature when predicting disease | Use only features known before the prediction time |
| Preprocessing leakage | Scaling or selecting features on all data, then splitting | Put every step inside a Pipeline |
| Group leakage | Scans of the same patient in train and test | GroupKFold by patient |
| Temporal leakage | Random split of time-series data | Train on the past, test on the future |
| Duplicate leakage | The same row in train and test | Deduplicate before splitting |
In code
Pure noise has nothing to learn, so honest accuracy should be about 50 per cent. Selecting features on all the data first makes noise look predictive; doing it inside the pipeline does not.
import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
rng = np.random.default_rng(0)
X = rng.normal(size=(300, 2000)) # 2,000 random features
y = rng.integers(0, 2, size=300) # random labels: nothing to learn
# Leaky: choose the 20 'best' features using ALL rows, then cross-validate
X_leaky = SelectKBest(f_classif, k=20).fit_transform(X, y)
leaky = cross_val_score(LogisticRegression(max_iter=1000), X_leaky, y, cv=5).mean()
# Honest: feature selection happens inside each training fold
honest_model = make_pipeline(SelectKBest(f_classif, k=20), LogisticRegression(max_iter=1000))
honest = cross_val_score(honest_model, X, y, cv=5).mean()
print(f"Leaky accuracy on pure noise: {leaky:.2f}")
print(f"Honest accuracy on pure noise: {honest:.2f}")Output
Leaky accuracy on pure noise: 0.75 Honest accuracy on pure noise: 0.51
Try it yourself
Is there a leak?
For each setup, decide whether information leaks into training or evaluation.
Several X-rays per patient, split by patient with GroupKFold
Filling missing values with the mean of the whole dataset before splitting
Random split of daily sales to forecast next month
StandardScaler inside a Pipeline evaluated with cross_val_score
Predicting loan default using 'number of collection calls made'
Common mistakes
- Trusting a score that seems too good to be true.
- Doing feature selection or oversampling before the split.
- Using random splits for time series or grouped data.
Check yourself
Summary
- Leakage gives the model information it won't have in real use.
- Target, preprocessing, group, temporal and duplicate leaks are the common kinds.
- Design validation to mimic real use, and put preprocessing in a Pipeline.
Finished reading?
Sources: scikit-learn: Common pitfalls and data leakage · Hastie et al., The Elements of Statistical Learning