k-Nearest Neighbours and Distance Metrics

Definition

Are you like your neighbours? A classifier that simply asks them.

In simple words

To guess whether a new student will pass, look at the three students with the most similar study hours and attendance, and go with the majority. Small k follows every quirk of the data; large k smooths everything out. In very many dimensions, all points end up roughly equally far apart, which weakens k-NN.

Why it matters

k-NN is a strong, assumption-free baseline, the basis of recommendation by similarity, and the core idea behind vector search in retrieval-augmented LLM systems, which find the nearest embeddings to a query.

How it works

The maths

In code

How k and scaling change k-NN accuracy on the wine dataset, plus the three distances computed by hand.

import numpy as np
from sklearn.datasets import load_wine
from sklearn.model_selection import cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_wine(return_X_y=True)
print(" k   unscaled   scaled")
for k in [1, 3, 5, 15, 51]:
    raw = cross_val_score(KNeighborsClassifier(k), X, y, cv=5).mean()
    scaled = cross_val_score(make_pipeline(StandardScaler(), KNeighborsClassifier(k)), X, y, cv=5).mean()
    print(f"{k:>2}   {raw:8.3f}   {scaled:6.3f}")

a, c = np.array([1.0, 2.0, 3.0]), np.array([2.0, 0.0, 3.0])
print("Euclidean:", round(float(np.linalg.norm(a - c)), 3))
print("Manhattan:", float(np.abs(a - c).sum()))
print("Cosine distance:", round(float(1 - a @ c / (np.linalg.norm(a) * np.linalg.norm(c))), 3))

 k   unscaled   scaled
 1      0.725    0.950
 3      0.703    0.944
 5      0.691    0.949
15      0.714    0.955
51      0.714    0.961
Euclidean: 2.236
Manhattan: 3.0
Cosine distance: 0.185

Try it yourself

Common mistakes

• Forgetting to scale features, so one large-unit feature dominates the distance.
• Using an even k for two classes, which allows ties.
• Applying k-NN to hundreds of raw features, where distances lose meaning.

Check yourself

Summary

• k-NN predicts from the k most similar training points.
• Scale features; choose k and the distance by validation.
• Its idea powers vector search in modern AI systems.