Gradients and Gradient Descent

Definition

Lost on a foggy hill, you feel the ground and step downhill. Repeat. That's how nearly every AI model is trained.

Gradient descent is an optimisation algorithm that finds low values of a function by repeatedly moving the parameters a small step in the direction of the negative gradient. The size of the step is set by the learning rate.

In simple words

Picture a ball rolling down a valley. At each moment it checks the slope under it and moves a little downhill. Eventually it settles at the bottom. Here the valley is the loss, and the ball's position is the model's weights.

Why it matters

Most models are too complex for a direct formula to give the best weights. Gradient descent needs only the gradient, so it scales to models with billions of parameters. Linear regression, logistic regression, neural networks and LLMs can all be trained with it.

How it works

Start with some parameter values, often random.
Compute the loss and its gradient.
Update each parameter: subtract the learning rate times the gradient.
Repeat until the loss stops improving.

The learning rate matters most

Learning rate	What happens
Too small	Very slow. May never get there in the time you have.
About right	The loss falls steadily.
Too large	It overshoots, bounces around, and the loss may blow up.

Three versions

Version	Each step uses	Note
Batch	All the data	Accurate, but slow on large data
Stochastic (SGD)	One example	Fast and noisy
Mini-batch	A small group, such as 32	The standard choice in deep learning

The maths

The update rule

θ_{t + 1} = θ_{t} - η \nabla_{θ} L (θ_{t})

$θ_{t}$: The parameters at step t
$η$: The learning rate (eta), a small positive number such as 0.01
$\nabla_{θ} L$: The gradient of the loss with respect to the parameters

Minimise L(w) = (w − 3)². The gradient is 2(w − 3). Start at w = 0 with η = 0.1.

Step 1: gradient = 2(0 − 3) = −6       w = 0 − 0.1(−6) = 0.6
Step 2: gradient = 2(0.6 − 3) = −4.8   w = 0.6 + 0.48 = 1.08
Step 3: gradient = 2(1.08 − 3) = −3.84 w = 1.08 + 0.384 = 1.464

w is moving towards 3, the true minimum, and the steps shrink as the slope flattens.

In code

Try changing the learning rate to 1.1 and watch it blow up.

gradient_descent.py

def gradient(w):
    return 2 * (w - 3)

w = 0.0
learning_rate = 0.1

for step in range(1, 26):
    w = w - learning_rate * gradient(w)
    if step in (1, 2, 3, 10, 25):
        print(f"step {step:2}: w = {w:.4f}, loss = {(w - 3)**2:.6f}")

Output

step  1: w = 0.6000, loss = 5.760000
step  2: w = 1.0800, loss = 3.686400
step  3: w = 1.4640, loss = 2.359296
step 10: w = 2.6779, loss = 0.103763
step 25: w = 2.9887, loss = 0.000128

Common mistakes

Adding the gradient. That climbs the hill and increases the loss.
Keeping a learning rate that makes the loss jump around or become NaN. Reduce it.
Not scaling the features. Very different scales make the valley long and narrow, so training zigzags.

Check yourself

1. w = 2, the gradient is 4, and η = 0.5. What is the new w?

2. The loss shoots up during training. What is the likely cause?

3. Which version is the standard in deep learning?

Summary

New parameters = old parameters − learning rate × gradient.
The learning rate is the setting that matters most.
Mini-batch is the normal choice for deep learning.

Read it. Now try it yourself.

Open the interactive activity

Finished reading?

Sources: Deisenroth, Faisal & Ong, Mathematics for Machine Learning · Goodfellow, Bengio & Courville, Deep Learning