Slope and the Derivative

Definition

If you knew which way is downhill, you could always find the bottom. A derivative tells you that.

The derivative of a function at a point is its instantaneous rate of change there, which is the slope of the curve at that point. It says how much the output changes for a tiny change in the input.

In simple words

Your car's speedometer shows a derivative: how fast distance is changing right now, not your average for the trip. A positive derivative means the function is going up, a negative one means going down, and zero means flat, which is where peaks and valleys are.

Why it matters

Training a model means making the loss smaller. The derivative tells the model which way to change each weight, and how strongly. Without derivatives there is no gradient descent, and without that, no modern deep learning.

How it works

Rule	Function	Derivative
Constant	c	0
Power	xⁿ	n xⁿ⁻¹
Constant multiple	c·f(x)	c·f′(x)
Sum	f(x) + g(x)	f′(x) + g′(x)
Product	f(x)·g(x)	f′g + f g′
Quotient	f(x)/g(x)	(f′g − f g′) / g²
Exponential	eˣ	eˣ
Logarithm	ln x	1/x

The maths

Definition of the derivative

f^{'} (x) = \frac{df}{d x} = h \to 0 lim \frac{f ( x + h ) - f ( x )}{h}

$h$: A tiny step in x, shrinking towards zero
$f (x + h) - f (x)$: How much the output changed

Power rule

\frac{d}{d x} x^{n} = n x^{n - 1}

Take f(x) = x² − 4x + 5. Where is the bottom of this curve?

f′(x) = 2x − 4
At x = 0: f′ = −4   the curve is falling
At x = 3: f′ = +2   the curve is rising
Set f′(x) = 0: 2x − 4 = 0, so x = 2
The minimum is at x = 2, where f(2) = 4 − 8 + 5 = 1

In code

Computers can estimate a derivative using the definition itself, with a very small h.

derivative.py

def f(x):
    return x**2 - 4*x + 5

def derivative(func, x, h=1e-6):
    return (func(x + h) - func(x)) / h

for x in [0, 2, 3]:
    print(f"x = {x}: exact {2*x - 4}, estimated {derivative(f, x):.4f}")

Output

x = 0: exact -4, estimated -4.0000
x = 2: exact 0, estimated 0.0000
x = 3: exact 2, estimated 2.0000

Common mistakes

Thinking a zero derivative always means a minimum. It can be a maximum or a flat stretch too.
Forgetting that the derivative is a function of x. It changes from point to point.

Check yourself

1. What is the derivative of x³?

2. At some point the loss has a negative derivative with respect to a weight. To reduce the loss, should the weight go up or down?

3. What is the derivative of 7?

Summary

A derivative is the slope: how fast the output changes with the input.
Power rule: xⁿ becomes n·xⁿ⁻¹.
Setting the derivative to zero finds peaks and valleys.

Finished reading?

Sources: Deisenroth, Faisal & Ong, Mathematics for Machine Learning