Backpropagation and Gradient Descent

Definition

A network may have billions of weights. How can it know which way to change every one of them?

In simple words

A team loses a match. The coach traces it back: the last player missed because the pass was late, and the pass was late because of the player before that. Each one gets a share of the blame. Backpropagation hands out blame like this, so that every weight learns how much it contributed to the error.

Why it matters

It makes training deep networks practical. Checking each weight separately would need one pass through the network per weight, which is impossible at this scale. Backpropagation gets all the gradients for about the cost of one extra pass.

How it works

1. Forward pass: compute the output, layer by layer, and save the values.
2. Compute the loss.
3. Backward pass: starting at the loss, use the chain rule to get each layer's gradients, reusing what was computed in the layer after it.
4. Update: each weight changes by −learning rate × its gradient.

The maths

Take x = 1, y = 1, w₁ = 0.5, b₁ = 0, w₂ = 0.5, b₂ = 0, with a sigmoid activation.

FORWARD
z = 0.5×1 + 0 = 0.5
h = σ(0.5) = 0.6225
ŷ = 0.5 × 0.6225 = 0.3112
L = ½(1 − 0.3112)² = 0.2372

BACKWARD
∂L/∂ŷ  = −(1 − 0.3112) = −0.6888
∂ŷ/∂h  = w₂ = 0.5
∂h/∂z  = σ(z)(1 − σ(z)) = 0.6225 × 0.3775 = 0.2350
∂z/∂w₁ = x = 1
∂L/∂w₁ = −0.6888 × 0.5 × 0.2350 × 1 = −0.0809

UPDATE, with η = 1
w₁ = 0.5 − 1 × (−0.0809) = 0.5809

The gradient is negative, so raising w₁ lowers the loss, and the update does exactly that.

In code

The worked example in code. The numeric check confirms the hand calculation.

import math

sigmoid = lambda z: 1 / (1 + math.exp(-z))
x, y = 1.0, 1.0
w1, w2 = 0.5, 0.5

def forward(w1):
    h = sigmoid(w1 * x)
    y_hat = w2 * h
    return h, y_hat, 0.5 * (y - y_hat) ** 2

h, y_hat, loss = forward(w1)
print("Loss:", round(loss, 4))

gradient = -(y - y_hat) * w2 * h * (1 - h) * x
print("Gradient by backprop:", round(gradient, 4))

step = 1e-6
numeric = (forward(w1 + step)[2] - loss) / step
print("Gradient by numeric check:", round(numeric, 4))

for _ in range(200):
    h, y_hat, loss = forward(w1)
    w1 -= 1.0 * (-(y - y_hat) * w2 * h * (1 - h) * x)
print("Loss after 200 updates:", round(forward(w1)[2], 4))

Loss: 0.2372
Gradient by backprop: -0.0809
Gradient by numeric check: -0.0809
Loss after 200 updates: 0.1301

Common mistakes

• Using sigmoid in deep hidden layers. Its derivative is at most 0.25, so many multiplied together vanish.
• Forgetting optimizer.zero_grad() in PyTorch, so gradients pile up across steps.
• Thinking it's how the brain learns. There's no evidence the brain does this.

Check yourself

Summary

• Backpropagation gets every gradient using the chain rule.
• Forward for the loss, backward for the gradients.
• Gradient descent then updates the weights.