Machine Learning Pipelines

Definition

How do you make preprocessing and modelling one object you can fit, tune and deploy?

A machine learning pipeline chains preprocessing steps and a final estimator into a single object with fit and predict methods. Fitting the pipeline fits each step in order on the training data; predicting applies the already-fitted steps to new data. Pipelines can be cross-validated, tuned and saved as one unit.

In simple words

Without a pipeline you must remember to scale the training data, then scale the test data with the same numbers, then call the model, every time. A pipeline does all of that for you, in the right order, and makes it impossible to accidentally fit the scaler on test data.

Why it matters

Pipelines prevent leakage, make experiments reproducible, let grid search tune preprocessing and model together, and turn the deployed model into a single file that cannot drift out of step with its preprocessing.

How it works

Pipeline([("scale", StandardScaler()), ("model", SVC())]) names each step.
Tune parameters of any step with the name__parameter syntax, such as model__C.
Combine with ColumnTransformer for mixed data, and save with joblib.

In code

A pipeline cross-validated as one unit, then tuned through its step names.

pipeline.py

from sklearn.datasets import load_wine
from sklearn.model_selection import GridSearchCV, cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = load_wine(return_X_y=True)
pipe = Pipeline([("scale", StandardScaler()), ("model", SVC())])

print("Without scaling:", round(cross_val_score(SVC(), X, y, cv=5).mean(), 3))
print("Pipeline with scaling:", round(cross_val_score(pipe, X, y, cv=5).mean(), 3))

search = GridSearchCV(pipe, {"model__C": [0.1, 1, 10], "model__gamma": ["scale", 0.01]}, cv=5)
search.fit(X, y)
print("Best parameters:", search.best_params_)
print("Best cross-validated accuracy:", round(search.best_score_, 3))
print("Steps:", list(search.best_estimator_.named_steps))

Output

Without scaling: 0.663
Pipeline with scaling: 0.983
Best parameters: {'model__C': 10, 'model__gamma': 'scale'}
Best cross-validated accuracy: 0.989
Steps: ['scale', 'model']

Try it yourself

What does pipe.fit(X_train) do?

Put the internal steps of fitting a two-step pipeline in order.

1The fitted model predicts on the scaled test data
2The scaler learns the mean and standard deviation of X_train
3The scaler transforms X_train
4Later, pipe.predict(X_test) scales X_test with the stored numbers
5The model is fitted on the scaled training data

Common mistakes

Scaling outside the pipeline and then cross-validating the pipeline.
Forgetting the step__parameter naming when tuning.
Saving only the model, not the pipeline, so production forgets the preprocessing.

Check yourself

1. How do you tune C of a step named 'model'?

2. What does pipe.predict do with the scaler?

3. Why deploy the whole pipeline?

Summary

A pipeline chains preprocessing and a model into one estimator.
It prevents leakage in cross-validation and tuning.
Tune with step__parameter, and deploy the whole pipeline.

Read it. Now try it yourself.

Open the interactive activity

Finished reading?

Sources: scikit-learn: Pipelines and composite estimators