Stacking and Voting Ensembles

Definition

Can two very different experts together beat either one?

In simple words

A linear model, a k-NN model and a forest see the data differently. Where one is wrong, the others are often right. Voting lets them outvote each other's mistakes; stacking hires a manager who learns from experience which expert to listen to in which situation.

Why it matters

Ensembles of diverse models often give the last one or two per cent of accuracy in competitions and critical applications, at the cost of more complexity and slower prediction.

How it works

1. Choose diverse, individually good base models.
2. Voting: combine their votes or probabilities directly.
3. Stacking: generate out-of-fold predictions with cross-validation, so the meta-model never sees predictions on rows the base models trained on.
4. Train a simple meta-model, often logistic regression, on those predictions.

In code

Three different base models, hard and soft voting, and a stacked ensemble, all cross-validated.

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier, StackingClassifier, VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
base = [
    ("logreg", make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))),
    ("knn", make_pipeline(StandardScaler(), KNeighborsClassifier(7))),
    ("forest", RandomForestClassifier(n_estimators=200, random_state=0)),
]
for name, model in base:
    print(f"{name:>13}: {cross_val_score(model, X, y, cv=5).mean():.4f}")

ensembles = {
    "hard voting": VotingClassifier(base, voting="hard"),
    "soft voting": VotingClassifier(base, voting="soft"),
    "stacking": StackingClassifier(base, final_estimator=LogisticRegression(max_iter=1000), cv=5),
}
for name, model in ensembles.items():
    print(f"{name:>13}: {cross_val_score(model, X, y, cv=5).mean():.4f}")

       logreg: 0.9807
          knn: 0.9701
       forest: 0.9596
  hard voting: 0.9807
  soft voting: 0.9737
     stacking: 0.9789

Try it yourself

Common mistakes

• Stacking on in-sample predictions, which leaks and overfits the meta-model.
• Combining near-identical models, which adds cost but no diversity.
• Using soft voting with models whose probabilities are badly calibrated.

Check yourself

Summary

• Voting combines votes (hard) or probabilities (soft).
• Stacking learns a meta-model on out-of-fold predictions.
• Diverse, accurate base models make ensembles work.