Principal Component Analysis

Definition

In simple words

Photograph a teapot. From some angles you can tell what it is, and from others you can't. PCA finds the angle that shows the most: it keeps the view with the most variation and drops what adds little. You lose a little detail, and gain a much simpler picture.

Why it matters

Fewer features mean faster training, less overfitting and less noise. It also lets you plot high-dimensional data in 2D, which helps you see the structure. It's widely used to compress data and remove correlated features.

How it works

1. Standardise the features, so large-scale columns don't dominate.
2. Compute the covariance matrix.
3. Find its eigenvectors and eigenvalues.
4. Sort by eigenvalue, largest first. Each eigenvalue shows how much variance that direction holds.
5. Keep the top k eigenvectors, and project the data onto them.

The maths

In code

from sklearn.datasets import load_breast_cancer
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

X, _ = load_breast_cancer(return_X_y=True)
X_scaled = StandardScaler().fit_transform(X)
print("Original features:", X.shape[1])

pca = PCA(n_components=2).fit(X_scaled)
print("Variance kept by 2 components:", round(pca.explained_variance_ratio_.sum(), 3))

keep_95 = PCA(n_components=0.95).fit(X_scaled)
print("Components needed for 95%:", keep_95.n_components_)

Original features: 30
Variance kept by 2 components: 0.632
Components needed for 95%: 10

Common mistakes

• Forgetting to standardise. A column in lakhs will swamp a column between 0 and 1.
• Fitting PCA on all the data, including the test set. That's leakage.
• Expecting components to be easy to interpret. Each is a mixture of all the features.

Check yourself

Summary

• PCA reduces features by keeping the directions with the most variance.
• Those directions are eigenvectors of the covariance matrix.
• Standardise first, and fit on training data only.