The Bias-Variance Trade-off

Definition

The bias-variance trade-off describes two sources of a model's error on new data. Bias is error from wrong or too-simple assumptions. Variance is error from being too sensitive to the particular training data. As a model gets more complex, bias tends to fall and variance tends to rise.

In simple words

Imagine throwing darts. High bias means your throws land close together but away from the centre: you're consistently wrong. High variance means your throws are scattered everywhere: sometimes close, but unreliable. You want low both: tight and on target.

Why it matters

It tells you how to improve a model. High bias means make it more complex. High variance means simplify it, regularise it or get more data. It explains why ensembles work, and it's among the most common interview questions.

How it works

	High bias	High variance
Also called	Underfitting	Overfitting
Training error	High	Low
Validation error	High, close to training	Much higher than training
Example	A line fit to a curve	A very deep tree
Fix	More complex model, better features	More data, regularisation, simpler model, ensembles

The maths

The decomposition for squared error

E [(y - \hat{f} (x))^{2}] = Bias^{2} (E [\hat{f} (x)] - f (x))^{2} + Variance E [(\hat{f} (x) - E [\hat{f} (x)])^{2}] + Noise σ^{2}

$f (x)$: The true function
$\hat{f} (x)$: What the model learned, from one training set
$σ^{2}$: Noise in the data, which no model can remove

In code

bias_variance.py

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures

rng = np.random.default_rng(0)
true = lambda x: np.sin(x)
test_x = np.array([[1.5]])

for degree in (1, 3, 15):
    predictions = []
    for _ in range(200):                       # many different training sets
        x = rng.uniform(0, 6, 20).reshape(-1, 1)
        y = true(x).ravel() + rng.normal(0, 0.3, 20)
        model = make_pipeline(PolynomialFeatures(degree), LinearRegression()).fit(x, y)
        predictions.append(model.predict(test_x)[0])
    predictions = np.array(predictions)
    bias_squared = (predictions.mean() - true(1.5)) ** 2
    print(f"degree {degree:2}: bias^2 = {bias_squared:.4f}, variance = {predictions.var():.4f}")

Output

degree  1: bias^2 = 0.1976, variance = 0.0322
degree  3: bias^2 = 0.0003, variance = 0.0192
degree 15: bias^2 = 0.0003, variance = 0.5668

Try it yourself

Which problem is shown?

Read each pair of training and validation results and diagnose it.

Training error 5 per cent, validation error 6 per cent
Training error 1 per cent, validation error 25 per cent
Validation error falls as you add training data, but stays above training error
Training error 30 per cent, validation error 31 per cent

Common mistakes

Confusing this statistical bias with social bias. They're different things that share a word.
Adding data to fix high bias. It mainly helps variance.

Check yourself

1. Training error is low, and validation error is high. Which is it?

2. Which part of the error can't be reduced by any model?

Summary

Bias: too simple. Variance: too sensitive to the data.
Error = Bias² + Variance + Noise.
Diagnose which one you have, then apply the right fix.