Multiple and Polynomial Regression

Definition

When is a straight line not enough?

Multiple regression uses several features at once. Polynomial regression models curved relationships by adding powers and products of features, such as x² and x₁x₂, and then fitting an ordinary linear model on the expanded features. The degree controls flexibility, and is chosen by validation error.

In simple words

The model is still 'linear' in its weights, which is why the same least-squares maths works, but the inputs are bent. Degree 1 is a line, degree 2 a parabola, degree 9 a wiggly curve that can pass through every training point and still predict badly between them.

Why it matters

Real relationships are rarely straight. Polynomial features are the simplest way to add flexibility, and choosing the degree is the clearest example of the bias-variance trade-off you'll meet in this module.

How it works

PolynomialFeatures(degree=d) creates every power and product up to degree d.
The number of features grows fast: with 10 features, degree 3 gives 286 columns.
Scale features before high degrees, or powers become enormous.
Choose the degree with the lowest validation error, not training error.

The maths

Polynomial regression in one feature

\overset{y}{^} = w_{0} + w_{1} x + w_{2} x^{2} + \dots + w_{d} x^{d}

Number of polynomial features for p inputs and degree d (with bias)

(d p + d)

In code

Fitting degrees 1 to 12 to a curved signal, and comparing training and validation error.

polynomial.py

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler

rng = np.random.default_rng(0)
x = rng.uniform(-3, 3, 60).reshape(-1, 1)
y = np.sin(x).ravel() + rng.normal(scale=0.2, size=60)
x_tr, x_va, y_tr, y_va = train_test_split(x, y, test_size=0.4, random_state=0)

print("degree  train MSE  validation MSE")
for d in [1, 2, 3, 5, 8, 12]:
    model = make_pipeline(PolynomialFeatures(d), StandardScaler(), LinearRegression()).fit(x_tr, y_tr)
    tr = mean_squared_error(y_tr, model.predict(x_tr))
    va = mean_squared_error(y_va, model.predict(x_va))
    print(f"{d:>6}  {tr:9.3f}  {va:14.3f}")
print("Columns for 10 features at degree 3:", PolynomialFeatures(3).fit(np.zeros((1, 10))).n_output_features_)

Output

degree  train MSE  validation MSE
     1      0.188           0.329
     2      0.188           0.329
     3      0.039           0.053
     5      0.032           0.046
     8      0.022           0.072
    12      0.020           0.087
Columns for 10 features at degree 3: 286

Try it yourself

Best degree?

Training error always falls as degree rises. Classify each situation.

Degree 3: low training error, lowest validation error
Degree 12: near-zero training error, validation error rising sharply
Degree 1 on a sine wave: high training and validation error

Common mistakes

Choosing the degree by training error.
Using high degrees without scaling, causing numerical problems.
Extrapolating a polynomial far outside the training range, where it shoots off.

Check yourself

1. Why is polynomial regression still a linear model?

2. As the degree increases, training error…

3. How should the degree be chosen?

Summary

Polynomial features let a linear model fit curves.
Degree controls flexibility; feature count grows fast.
Pick the degree by validation error and scale first.

Read it. Now try it yourself.

Open the interactive activity

Finished reading?

Sources: James et al., An Introduction to Statistical Learning · scikit-learn: Polynomial features