Self-Attention Step by Step

Definition

In "The animal didn't cross the road because it was tired", how does a model know what "it" refers to?

Self-attention is a mechanism in which each token in a sequence looks at all the other tokens and decides how much each one matters to it. The token's new representation is a weighted average of the others, with weights computed from the data. It's the core of the Transformer.

In simple words

In a meeting, when someone says "it", you look around for what they meant. Each word does this. It asks a question (query), every word offers a label (key), and where they match, that word's information (value) flows in. So "it" gathers meaning from "animal".

Why it matters

Unlike RNNs, it connects distant words directly and can run in parallel, which made huge models practical. Since the 2017 paper "Attention Is All You Need", it has become the base of GPT, BERT, Claude, Gemini and modern vision models.

How it works

Each token's embedding is multiplied by three learned matrices, giving a query Q, a key K and a value V.
Compare each query with every key, using dot products.
Divide by the square root of the key size, which keeps the numbers stable.
Apply softmax to turn the scores into weights that sum to 1.
The output is the weighted sum of the values.

Idea	Why it's needed
Multi-head attention	Several attentions run side by side, so each can focus on a different relationship
Positional encoding	Attention ignores order, so position information must be added
Masking	When generating, a token mustn't see future tokens

The maths

Scaled dot-product attention

Attention (Q, K, V) = softmax (\frac{Q K ^{⊤}}{d _{k}}) V

$Q$: Queries: what each token is looking for
$K$: Keys: what each token offers to be matched on
$V$: Values: the information each token passes on
$d_{k}$: The size of the keys

Three tokens, each with a 2-number query and key, where Q = K = [[1, 0], [0, 1], [1, 1]] and d_k = 2. Scores for the first token:

Q₁·K₁ = 1     Q₁·K₂ = 0     Q₁·K₃ = 1
Divide by √2:  0.707, 0, 0.707
Softmax:       0.401, 0.198, 0.401
Token 1's output = 0.401·V₁ + 0.198·V₂ + 0.401·V₃

In code

attention.py

import numpy as np

def softmax(x):
    e = np.exp(x - x.max(axis=-1, keepdims=True))
    return e / e.sum(axis=-1, keepdims=True)

def attention(Q, K, V):
    d_k = K.shape[-1]
    scores = Q @ K.T / np.sqrt(d_k)
    weights = softmax(scores)
    return weights @ V, weights

embeddings = np.array([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]])
output, weights = attention(embeddings, embeddings, embeddings)

print("Attention weights:")
print(weights.round(3))
print("Each row adds up to:", weights.sum(axis=1).round(3))
print("New representations:")
print(output.round(3))

Output

Attention weights:
[[0.401 0.198 0.401]
 [0.198 0.401 0.401]
 [0.248 0.248 0.503]]
Each row adds up to: [1. 1. 1.]
New representations:
[[0.802 0.599]
 [0.599 0.802]
 [0.752 0.752]]

Common mistakes

Reading attention weights as a full explanation of why a model decided something. They're only a partial clue.
Forgetting positional encoding. Without it, "dog bites man" and "man bites dog" look alike.
Skipping the mask when training a language model, which lets it cheat by seeing the next word.

Check yourself

1. What do the attention weights for one token add up to?

2. Why are positional encodings needed?

3. What's an advantage over RNNs?

Summary

Each token gathers information from all the others, weighted by relevance.
Weights come from query-key dot products and softmax.
It's the core of every Transformer and LLM.

Finished reading?

Sources: Vaswani et al., Attention Is All You Need (2017)