Multivariable Calculus and Gradients

Definition

In simple words

You're standing on a hilly field in fog. Take a small step east and notice how much you rise: that's one partial derivative. Take a small step north: that's another. Put them together and you have the gradient, an arrow pointing straight up the steepest slope. To go downhill, walk the opposite way.

Why it matters

A real model has thousands or billions of weights, so its loss has that many inputs. The gradient tells you how to change all of them at once to reduce the loss. Every optimiser, from plain gradient descent to Adam, is built on it.

How it works

1. To get ∂f/∂x, treat every other variable as a constant and differentiate normally.
2. Repeat for each variable.
3. Stack the results into a vector. That's the gradient.

The maths

Take f(x, y) = x² + 3xy. Evaluate its gradient at (2, 1).

∂f/∂x = 2x + 3y   (y treated as a constant)
∂f/∂y = 3x        (x treated as a constant)

At (2, 1): ∂f/∂x = 4 + 3 = 7
           ∂f/∂y = 6
∇f(2, 1) = [7, 6]

In code

def f(x, y):
    return x**2 + 3*x*y

def gradient(x, y, h=1e-6):
    df_dx = (f(x + h, y) - f(x, y)) / h
    df_dy = (f(x, y + h) - f(x, y)) / h
    return round(df_dx, 3), round(df_dy, 3)

print("Gradient at (2, 1):", gradient(2, 1))

Gradient at (2, 1): (7.0, 6.0)

Common mistakes

• Stepping in the direction of the gradient to reduce a loss. That increases it.
• Forgetting to hold the other variables fixed. In ∂/∂x of 3xy, y is a constant, so the result is 3y.

Check yourself

Summary

• A partial derivative changes one input and holds the others fixed.
• The gradient collects them all and points uphill.
• Training steps the opposite way to the gradient.