Vector Operations and the Dot Product

Definition

How does a recommendation system know two people have similar taste? Often, with one small formula.

The dot product takes two vectors of the same length, multiplies them element by element, and adds up the results. The answer is one number. It also equals the product of their lengths and the cosine of the angle between them.

In simple words

You buy 2 pens at ₹10 and 3 notebooks at ₹40. Your bill is 2×10 + 3×40 = ₹140. That sum is a dot product of [2, 3] and [10, 40]. Geometrically it measures how much two arrows point in the same direction: large when they agree, zero when they're at right angles, negative when they oppose.

Why it matters

It is everywhere. A neuron computes a dot product of its weights and inputs. Search and RAG compare texts with cosine similarity, which is built on it. Attention in Transformers uses dot products to decide which words matter. Matrix multiplication is many dot products.

How it works

Result	Angle between vectors	Meaning
Positive	Less than 90°	They point in a similar direction
Zero	90°	Unrelated (orthogonal)
Negative	More than 90°	They point in opposite directions

Cosine similarity

A long document has big word counts just because it's long. To compare meaning and not size, divide the dot product by both lengths. The result is always between −1 and 1. Almost every vector database uses this.

The maths

Dot product, by elements

a \cdot b = i = 1 \sum n a_{i} b_{i} = a_{1} b_{1} + a_{2} b_{2} + \dots + a_{n} b_{n}

$a_{i}, b_{i}$: Element i of each vector
$n$: The length of each vector. They must be equal.

Dot product, by geometry

a \cdot b = ∥ a ∥ ∥ b ∥ cos θ

$∥ a ∥$: The length of vector a
$θ$: The angle between the two vectors

Cosine similarity

cos θ = \frac{a \cdot b}{∥ a ∥ ∥ b ∥}

Take a = [1, 2, 3] and b = [4, 0, −1].

a·b  = (1×4) + (2×0) + (3×−1) = 4 + 0 − 3 = 1
‖a‖  = √(1² + 2² + 3²) = √14 ≈ 3.742
‖b‖  = √(4² + 0² + (−1)²) = √17 ≈ 4.123
cos θ = 1 / (3.742 × 4.123) ≈ 0.065

A cosine of 0.065 is close to zero, so these vectors are nearly unrelated.

In code

dot_product.py

import numpy as np

a = np.array([1, 2, 3])
b = np.array([4, 0, -1])

print("By hand:", sum(x * y for x, y in zip(a, b)))
print("NumPy:", np.dot(a, b))

cosine = np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
print("Cosine similarity:", round(cosine, 3))

# Word counts for two documents about similar things, one twice as long
short_doc = np.array([1, 2, 0])
long_doc = np.array([2, 4, 0])
print("Same direction:", round(np.dot(short_doc, long_doc) / (np.linalg.norm(short_doc) * np.linalg.norm(long_doc)), 3))

Output

By hand: 1
NumPy: 1
Cosine similarity: 0.065
Same direction: 1.0

Common mistakes

Trying it on vectors of different lengths. It isn't defined.
Multiplying element by element and forgetting to add. a * b in NumPy gives a vector, and np.dot(a, b) gives one number.
Comparing raw dot products when vector sizes differ. Use cosine similarity.

Check yourself

1. What is [2, 3] · [4, 5]?

2. The dot product of two non-zero vectors is 0. What does that mean?

3. Why use cosine similarity, not the raw dot product, to compare documents?

Summary

Multiply element by element, then add. The answer is one number.
It measures how much two vectors point the same way.
Cosine similarity is the dot product divided by both lengths.

Finished reading?

Sources: Deisenroth, Faisal & Ong, Mathematics for Machine Learning