Distance and Similarity

Definition

A norm is a function that gives a vector's size as a single non-negative number. A distance measures how far apart two points are, usually as the norm of their difference. The most common are the L2 (Euclidean) norm and the L1 (Manhattan) norm.

In simple words

You want to go 3 blocks east and 4 blocks north. A bird flies straight: 5 blocks. That's the L2 distance. A car follows the roads: 3 + 4 = 7 blocks. That's the L1 distance. Same two places, two fair ways to measure.

Why it matters

KNN and k-means decide everything using distance. Ridge and Lasso regression penalise large weights with L2 and L1 norms. Gradient clipping and normalising embeddings both use norms. Choosing the wrong one can quietly hurt a model.

How it works

Name	In words	Used for
L1 (Manhattan)	Add the absolute values	Lasso, data with outliers
L2 (Euclidean)	Square root of the sum of squares	KNN, k-means, Ridge
L∞ (Max)	The largest absolute value	Worst-case limits, adversarial examples

The maths

L2 norm

∥ x ∥_{2} = i = 1 \sum n x_{i}^{2}

L1 norm

∥ x ∥_{1} = i = 1 \sum n ∣ x_{i} ∣

General Lp norm

∥ x ∥_{p} = (i = 1 \sum n ∣ x_{i} ∣^{p})^{1/ p}

$p$: Choose 1 for Manhattan, 2 for Euclidean

Euclidean distance between two points

d (a, b) = ∥ a - b ∥_{2} = i = 1 \sum n (a_{i} - b_{i})^{2}

For x = [3, −4]:

L1 = |3| + |−4| = 7
L2 = √(3² + (−4)²) = √25 = 5
L∞ = max(|3|, |−4|) = 4

In code

norms.py

import numpy as np

x = np.array([3, -4])
print("L1:", np.linalg.norm(x, 1))
print("L2:", np.linalg.norm(x, 2))
print("Max:", np.linalg.norm(x, np.inf))

a = np.array([1, 2])
b = np.array([4, 6])
print("Distance from a to b:", np.linalg.norm(a - b))

Output

L1: 7.0
L2: 5.0
Max: 4.0
Distance from a to b: 5.0

Common mistakes

Forgetting the absolute value in L1. Negative numbers would cancel out and the size would look small.
Running KNN or k-means on unscaled features.

Check yourself

1. What is the L1 norm of [1, −2, 3]?

2. What is the L2 norm of [6, 8]?

Summary

A norm gives a vector's size. A distance is the norm of a difference.
L1 adds absolute values, and L2 is the straight-line length.
Scale your features before using distance.

Finished reading?

Sources: Deisenroth, Faisal & Ong, Mathematics for Machine Learning