Regularisation: Ridge, Lasso and Elastic Net

Definition

How do you stop a flexible model from memorising the noise?

Regularisation adds a penalty on the size of the weights to the loss, trading a little bias for a large reduction in variance. Ridge (L2) penalises the sum of squared weights and shrinks them smoothly. Lasso (L1) penalises the sum of absolute weights and drives some exactly to zero, performing feature selection. Elastic Net combines both. The strength λ (alpha in scikit-learn) is tuned by validation.

In simple words

An overfitted model uses huge positive and negative weights that cancel out on the training data. The penalty says 'every unit of weight costs you something', so the model keeps only weights that genuinely reduce error. Lasso is stricter: it would rather switch a feature off completely.

Why it matters

Regularisation is how almost every modern model, from linear regression to large neural networks with weight decay, stays general. Lasso also gives sparse, interpretable models when you have many candidate features.

How it works

	Ridge (L2)	Lasso (L1)	Elastic Net
Penalty	λ Σ wⱼ²	λ Σ \|wⱼ\|	A mix of both
Effect	Shrinks all weights	Sets some weights to exactly 0	Sparse, but stable with correlated features
Use when	Many small effects, correlated features	You expect few relevant features	Many correlated features, some irrelevant

The maths

Ridge objective

J (w) = ∥ y - X w ∥^{2} + λ j = 1 \sum p w_{j}^{2}

Ridge closed form

w = (X^{⊤} X + λ I)^{- 1} X^{⊤} y

Lasso objective

J (w) = \frac{1}{2 n} ∥ y - X w ∥^{2} + λ j = 1 \sum p ∣ w_{j} ∣

$λ$: Regularisation strength: larger means smaller weights

In code

Ridge and Lasso on the diabetes dataset: as alpha grows, ridge shrinks every weight while lasso switches features off.

regularisation.py

import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.linear_model import Lasso, Ridge, RidgeCV
from sklearn.preprocessing import StandardScaler

X, y = load_diabetes(return_X_y=True)
X = StandardScaler().fit_transform(X)

print("alpha   ridge |w| sum   lasso |w| sum   lasso zero weights")
for alpha in [0.01, 1, 10, 50]:
    ridge = Ridge(alpha=alpha).fit(X, y)
    lasso = Lasso(alpha=alpha, max_iter=10000).fit(X, y)
    print(f"{alpha:>5}   {np.abs(ridge.coef_).sum():13.1f}   {np.abs(lasso.coef_).sum():13.1f}   {np.sum(lasso.coef_ == 0):>18}")

best = RidgeCV(alphas=np.logspace(-3, 3, 13)).fit(X, y)
print("Alpha chosen by cross-validation:", best.alpha_)

Output

alpha   ridge |w| sum   lasso |w| sum   lasso zero weights
 0.01           164.3           160.0                    0
    1           143.7            90.7                    3
   10           105.0            52.1                    6
   50            93.8             0.0                   10
Alpha chosen by cross-validation: 1.0

Try it yourself

Which penalty gives zeros?

Assign each property to Ridge or Lasso.

Spreads weight across correlated features
Has a closed-form solution
Sets some weights exactly to zero
Performs automatic feature selection

Common mistakes

Regularising unscaled features, so the penalty hits some features harder than others.
Penalising the intercept. Libraries leave it out by default; your own code should too.
Picking alpha by hand instead of cross-validation.

Check yourself

1. Which regularisation can remove features entirely?

2. As λ increases, the weights…

3. Regularisation mainly reduces…

Summary

Ridge adds λΣw² and shrinks; Lasso adds λΣ|w| and zeroes.
Elastic Net mixes both for correlated features.
Standardise first and tune λ by cross-validation.

Read it. Now try it yourself.

Open the interactive activity

Finished reading?

Sources: James et al., An Introduction to Statistical Learning · scikit-learn: Linear models