Bagging and Random Forests

Definition

Why ask a crowd of imperfect experts instead of one?

Bagging (bootstrap aggregating) trains many models on bootstrap samples, random samples of the training data drawn with replacement, and averages their predictions to reduce variance. A random forest bags decision trees and also considers only a random subset of features at each split, which decorrelates the trees. Out-of-bag (OOB) error uses the rows each tree did not see as a built-in validation set.

In simple words

One deep tree memorises its training data and changes wildly if the data changes slightly. Grow hundreds, each on a slightly different sample and each looking at different features, and their individual mistakes cancel out. The vote of the forest is far more stable than any single tree.

Why it matters

Random forests are among the most reliable out-of-the-box models for tabular data: little tuning, no scaling, robust to outliers and with useful feature importances.

How it works

Draw a bootstrap sample of n rows with replacement (about 63 per cent unique rows).
Grow a deep tree, considering a random subset of features (often √p) at each split.
Repeat for many trees.
Predict by majority vote (classification) or averaging (regression).
Estimate error from out-of-bag rows without a separate validation set.

The maths

Variance of an average of B correlated trees

Var (\frac{1}{B} b = 1 \sum B T_{b}) = ρ σ^{2} + \frac{1 - ρ}{B} σ^{2}

$ρ$: Correlation between trees, reduced by feature sampling
$σ^{2}$: Variance of a single tree

Chance a row is left out of a bootstrap sample

(1 - \frac{1}{n})^{n} \approx e^{- 1} \approx 0.368

In code

One tree against a bagged ensemble and a random forest, with the out-of-bag estimate.

random_forest.py

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import BaggingClassifier, RandomForestClassifier
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier

X, y = load_breast_cancer(return_X_y=True)
models = {
    "single tree": DecisionTreeClassifier(random_state=0),
    "bagged trees": BaggingClassifier(DecisionTreeClassifier(), n_estimators=100, random_state=0),
    "random forest": RandomForestClassifier(n_estimators=200, random_state=0),
}
for name, model in models.items():
    print(f"{name:>13}: {cross_val_score(model, X, y, cv=5).mean():.3f}")

forest = RandomForestClassifier(n_estimators=200, oob_score=True, random_state=0).fit(X, y)
print("Out-of-bag accuracy:", round(forest.oob_score_, 3))

Output

  single tree: 0.917
 bagged trees: 0.958
random forest: 0.960
Out-of-bag accuracy: 0.965

Try it yourself

What does more of each do?

For a random forest, does each change mainly reduce variance, or mainly cost time with little gain?

Going from 2,000 trees to 20,000
Sampling fewer features per split
Going from 10 trees to 200

Common mistakes

Expecting a forest to extrapolate beyond the training range. Trees can't.
Reading impurity-based importances as exact. They favour high-cardinality features; use permutation importance.
Tuning hundreds of parameters. Defaults with enough trees are usually close to the best.

Check yourself

1. What does bagging mainly reduce?

2. What makes a random forest different from plain bagging?

3. About what share of rows is out-of-bag for each tree?

Summary

Bagging averages models trained on bootstrap samples.
Random forests add feature sampling to decorrelate trees.
OOB error is a free validation estimate.

Read it. Now try it yourself.

Open the interactive activity

Finished reading?

Sources: scikit-learn: Random forests · James et al., An Introduction to Statistical Learning