Correlation | AI Pathshala

Definition

Correlation measures how strongly two variables move together, on a scale from −1 to +1. Pearson's correlation measures straight-line relationships. Spearman's correlation applies the same idea to ranks, so it catches any steady rising or falling relationship.

In simple words

Hours studied and marks usually rise together, so that's a positive correlation. Price and sales often move oppositely, a negative one. Shoe size and marks don't move together at all: near zero.

Why it matters

In EDA, correlation shows which features may help and which are duplicates. Two features with 0.99 correlation carry nearly the same information, a problem called multicollinearity, which can make linear models unstable.

How it works

Value of r	Meaning
+1	A perfect rising straight line
+0.7	Strong and rising
0	No straight-line relationship
−0.7	Strong and falling
−1	A perfect falling straight line

The maths

Pearson correlation

r = \frac{\sum _{i} ( x _{i} - x ˉ ) ( y _{i} - y ˉ )}{\sum _{i} ( x _{i} - x ˉ ) ^{2} \sum _{i} ( y _{i} - y ˉ ) ^{2}} = \frac{cov ( X , Y )}{σ _{X} σ _{Y}}

$r$: The correlation, from −1 to +1
$cov$: Covariance: do they move together?

x = [1, 2, 3], y = [2, 4, 7]. The means are 2 and 4.33.

x − x̄ = −1, 0, 1           y − ȳ = −2.33, −0.33, 2.67
Σ(x−x̄)(y−ȳ) = 2.33 + 0 + 2.67 = 5
Σ(x−x̄)² = 2       Σ(y−ȳ)² = 5.44 + 0.11 + 7.11 = 12.67
r = 5 / (√2 × √12.67) = 5 / 5.03 ≈ 0.993

In code

correlation.py

import numpy as np
import pandas as pd

x = np.array([1, 2, 3])
y = np.array([2, 4, 7])
print("Pearson r:", round(np.corrcoef(x, y)[0, 1], 3))

curve_x = np.array([-3, -2, -1, 0, 1, 2, 3])
print("r for y = x squared:", round(np.corrcoef(curve_x, curve_x ** 2)[0, 1], 3))

df = pd.DataFrame({"a": [1, 2, 3, 4, 5], "b": [1, 8, 27, 64, 125]})
print("Pearson:", round(df.corr(method="pearson").loc["a", "b"], 3))
print("Spearman:", round(df.corr(method="spearman").loc["a", "b"], 3))

Output

Pearson r: 0.993
r for y = x squared: 0.0
Pearson: 0.943
Spearman: 1.0

Common mistakes

Claiming that one thing causes another because they correlate.
Trusting r without a plot. Very different datasets can share the same r.
Using Pearson on data with outliers. One extreme point can swing it. Consider Spearman.

Check yourself

1. r = −0.9 means:

2. Students who sleep more score higher. What can you conclude?

Summary

Correlation runs from −1 to +1. The sign is direction, and the size is strength.
Pearson is for straight lines, and Spearman for any steady trend.
Correlation is not causation, and always plot the data.