Linear Regression | AI Pathshala

Definition

Linear regression is a supervised learning algorithm that predicts a number by fitting a straight line (or a flat plane, with many features) through the data. It chooses the line that minimises the average squared difference between predictions and true values.

In simple words

Plot flat sizes against prices, and the dots roughly form a rising band. Draw the straight line that passes as close as possible to all the dots. To price a new flat, find its size on that line and read off the price. The whole job of the algorithm is to find the best line.

Why it matters

It's the simplest useful model, it's fast, and you can read exactly what it learned. It's usually the first baseline for any number-prediction problem. It's also the base of logistic regression and of neural networks, since a neuron is linear regression plus an activation.

How it works

Pick a line with a slope w and an intercept b.
Measure the error: how far each point is from the line, squared and averaged.
Adjust w and b to reduce the error, with gradient descent or the direct formula.

Assumptions

The relationship is roughly a straight line.
Errors are independent of each other, with similar spread throughout.
Features aren't near-duplicates of each other, or coefficients become unstable.

The maths

The model

\overset{y}{^} = w x + b

$\overset{y}{^}$: The prediction
$w$: The slope: how much y changes when x rises by 1
$b$: The intercept: the prediction when x is 0

The loss: mean squared error

J (w, b) = \frac{1}{n} i = 1 \sum n (y_{i} - (w x_{i} + b))^{2}

Best slope and intercept, for one feature

w = \frac{\sum _{i} ( x _{i} - x ˉ ) ( y _{i} - y ˉ )}{\sum _{i} ( x _{i} - x ˉ ) ^{2}}, b = \overset{y}{ˉ} - w \overset{x}{ˉ}

Gradient descent updates

w \leftarrow w + \frac{2 η}{n} i \sum x_{i} (y_{i} - \overset{y}{^}_{i}), b \leftarrow b + \frac{2 η}{n} i \sum (y_{i} - \overset{y}{^}_{i})

$η$: The learning rate

Points: (1, 2), (2, 4), (3, 5). The means are x̄ = 2 and ȳ = 3.67.

Σ(x−x̄)(y−ȳ) = (−1)(−1.67) + (0)(0.33) + (1)(1.33) = 3
Σ(x−x̄)²     = 1 + 0 + 1 = 2
w = 3 / 2 = 1.5
b = 3.67 − 1.5 × 2 = 0.67

The line: ŷ = 1.5x + 0.67
Prediction for x = 4: ŷ = 6.67

In code

First from scratch, using the formula above, then with scikit-learn. Both find the same line.

linear_regression.py

import numpy as np
from sklearn.linear_model import LinearRegression

x = np.array([1, 2, 3])
y = np.array([2, 4, 5])

w = np.sum((x - x.mean()) * (y - y.mean())) / np.sum((x - x.mean()) ** 2)
b = y.mean() - w * x.mean()
print(f"From scratch: y = {w:.2f}x + {b:.2f}")

model = LinearRegression().fit(x.reshape(-1, 1), y)
print(f"scikit-learn: y = {model.coef_[0]:.2f}x + {model.intercept_:.2f}")
print("Prediction for x = 4:", round(model.predict([[4]])[0], 2))

Output

From scratch: y = 1.50x + 0.67
scikit-learn: y = 1.50x + 0.67
Prediction for x = 4: 6.67

Common mistakes

Using it on a clearly curved relationship. Plot the data first.
Reading a coefficient as cause and effect.
Predicting far outside the range of the training data.

Check yourself

1. In ŷ = 3x + 10, what does 3 mean?

2. Why square the errors?

3. ŷ = 2x + 1. What's the prediction at x = 5?

Summary

It fits the straight line ŷ = wx + b.
It minimises the mean squared error.
Use the formula or gradient descent, and use it as your first baseline.

Read it. Now try it yourself.

Open the interactive activity

Finished reading?

Sources: scikit-learn user guide · Google Machine Learning Crash Course