K-Nearest Neighbours (KNN): the complete guide

What is KNN?

K-Nearest Neighbours (KNN) predicts the answer for something new by looking at the k most similar examples it has already seen. For a category, those neighbours vote. For a number, their values are averaged.

KNN is a supervised learning algorithm, because the examples come with correct answers. It is also called a lazy learner: it does no work during training except storing the data. All the calculating happens when you ask for a prediction.

How it works, in four steps

1. Choose k, the number of neighbours to ask.
2. Measure the distance from the new point to every stored example.
3. Keep the k closest examples.
4. Take a majority vote for a category, or the average for a number.

A full example, worked by hand

Seven students are recorded by hours studied and hours slept, and whether they passed. A new student studied 5 hours and slept 4. Will they pass?

distance = √[ (study₁ − study₂)² + (sleep₁ − sleep₂)² ]

Student D:  √[ (5 − 6)² + (4 − 5)² ] = √[ 1 + 1 ] = √2 = 1.41

Do the same for every student, then sort from nearest to farthest:

Now watch how the answer depends on k. Move the slider and count the votes yourself:

How to choose k

• Use cross-validation: try several values of k and keep the one with the best average score. Code is below.
• With two classes, use an odd k so votes can't tie.
• The popular shortcut k ≈ √n (n = number of training examples) is only a starting point, not an answer.

Distance metrics

“Nearest” needs a definition. Here are the common ones, measured between the new student (5, 4) and student F (8, 6):

Why scaling is not optional

This is the mistake beginners make most often with KNN. Compare a 26-year-old earning ₹53,000 with two people: one is 25 and earns ₹50,000, the other is 60 and earns ₹52,000.

import math

new_person = (26, 53000)
young = (25, 50000)
old = (60, 52000)

print("Without scaling")
print("  to the 25-year-old:", round(math.dist(new_person, young), 1))
print("  to the 60-year-old:", round(math.dist(new_person, old), 1))

# Put both features on a similar scale: age / 50, income / 20000
def scale(person):
    return (person[0] / 50, person[1] / 20000)

print("With scaling")
print("  to the 25-year-old:", round(math.dist(scale(new_person), scale(young)), 2))
print("  to the 60-year-old:", round(math.dist(scale(new_person), scale(old)), 2))

Without scaling
  to the 25-year-old: 3000.0
  to the 60-year-old: 1000.6
With scaling
  to the 25-year-old: 0.15
  to the 60-year-old: 0.68

Without scaling, the 60-year-old looks “closer”, because income numbers are huge and drown out age. After scaling, the 25-year-old is correctly closer. In real work, use StandardScaler, as the scikit-learn code below does.

KNN from scratch in Python

This runs the worked example above, with no libraries beyond Python's own. Press Run to try it, then change the new student.

import math
from collections import Counter

# (hours studied, hours slept), result
students = [
    ((1, 2), "Fail"), ((2, 1), "Fail"), ((2, 3), "Fail"), ((4, 4), "Fail"),
    ((6, 5), "Pass"), ((7, 7), "Pass"), ((8, 6), "Pass"),
]

def predict(new_student, k):
    # Step 2: measure the distance to everyone
    distances = [(math.dist(new_student, point), result) for point, result in students]
    # Step 3: keep the k closest
    nearest = sorted(distances, key=lambda row: row[0])[:k]
    # Step 4: majority vote
    votes = Counter(result for _, result in nearest)
    return votes.most_common(1)[0][0]

for k in [1, 3, 5, 7]:
    print("k =", k, "->", predict((5, 4), k))

k = 1 -> Fail
k = 3 -> Fail
k = 5 -> Pass
k = 7 -> Fail

KNN with scikit-learn, the way it's done at work

Here is a real dataset: 150 iris flowers, 4 measurements each, 3 species. A pipeline scales the data, cross-validation picks k, and the held-back test set gives an honest final score.

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)

# Try each k using only the training data
best_k, best_score = None, 0
for k in [1, 3, 5, 7, 9, 11, 15, 21]:
    model = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=k))
    score = cross_val_score(model, X_train, y_train, cv=5).mean()
    print(f"k={k:2}  cross-validation accuracy {score:.3f}")
    if score > best_score:
        best_k, best_score = k, score

# Only now touch the test set, once
final = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=best_k))
final.fit(X_train, y_train)
print("Best k:", best_k)
print(f"Test accuracy: {final.score(X_test, y_test):.3f}")

k= 1  cross-validation accuracy 0.946
k= 3  cross-validation accuracy 0.946
k= 5  cross-validation accuracy 0.946
k= 7  cross-validation accuracy 0.955
k= 9  cross-validation accuracy 0.955
k=11  cross-validation accuracy 0.955
k=15  cross-validation accuracy 0.946
k=21  cross-validation accuracy 0.902
Best k: 7
Test accuracy: 0.947

Predicting a number: KNN regression

Swap the vote for an average, and KNN predicts numbers.

from sklearn.neighbors import KNeighborsRegressor

# House size in square feet, and price in lakh rupees
sizes = [[500], [700], [900], [1100], [1300], [1500]]
prices = [20, 28, 35, 44, 52, 60]

model = KNeighborsRegressor(n_neighbors=2).fit(sizes, prices)
print("Predicted price for 1000 sq ft:", model.predict([[1000]])[0], "lakh")

Predicted price for 1000 sq ft: 39.5 lakh

Speed and the curse of dimensionality

With many features, every point ends up about equally far from every other, so “nearest” stops meaning much. This is the curse of dimensionality. Remove useless features or use PCA before KNN.

Strengths, weaknesses and where it's used

• Recommendations: “people similar to you also liked this”.
• Filling in missing values from similar rows (KNNImputer).
• Searching for similar images or documents using embeddings.
• A quick baseline: a simple score that a fancier model must beat.

Common mistakes

• Forgetting to scale features.
• Fitting the scaler on all the data before splitting, which leaks test information. A pipeline prevents this.
• Choosing k using the test set. Use cross-validation on the training data.
• Trusting accuracy when one class is rare. A large k may always predict the common class.

Interview questions

Practice

1. By hand: using the table above, what does k = 3 predict for a student at (7, 5)? Check with the scratch code.
2. Change the scratch code to use Manhattan distance. Does any prediction change?
3. In the iris code, remove StandardScaler. Does the score change? Why might iris be less affected than the income example?