Forward Propagation and Loss

Definition

A loss function gives a single number that measures how wrong a model's predictions are. Training is the process of making this number smaller. Regression usually uses mean squared error, and classification usually uses cross-entropy.

In simple words

It's the marks on a practice test, in reverse: a lower score is better. It tells the model how badly it did, and gradient descent uses that to improve. Whatever you put in the loss is what the model will care about, and nothing else.

Why it matters

The loss defines what "good" means for the model. A poor choice trains a model to do the wrong thing, however good the architecture is.

How it works

Task	Loss	Note
Regression	MSE	Punishes big errors heavily
Regression with outliers	MAE or Huber	Less sensitive to outliers
Binary classification	Binary cross-entropy	With a sigmoid output
Multi-class	Categorical cross-entropy	With a softmax output
Imbalanced detection	Focal loss	Focuses on hard examples
SVM	Hinge	Maximises the margin

The maths

Mean squared error

MSE = \frac{1}{n} i = 1 \sum n (y_{i} - \overset{y}{^}_{i})^{2}

Mean absolute error

MAE = \frac{1}{n} i = 1 \sum n ∣ y_{i} - \overset{y}{^}_{i} ∣

Cross-entropy

L = - k \sum y_{k} lo g \overset{y}{^}_{k}

$y_{k}$: 1 for the true class, and 0 otherwise
$\overset{y}{^}_{k}$: The predicted probability of class k

True values [3, 5], predictions [2, 8]:

MSE = [(3−2)² + (5−8)²] / 2 = (1 + 9) / 2 = 5
MAE = [|3−2| + |5−8|] / 2 = (1 + 3) / 2 = 2

The error of 3 counts for 9 in MSE, which is why MSE reacts so strongly to outliers.

In code

losses.py

import numpy as np

y = np.array([3.0, 5.0])
pred = np.array([2.0, 8.0])
print("MSE:", np.mean((y - pred) ** 2))
print("MAE:", np.mean(np.abs(y - pred)))

true_class = 1
for probabilities in ([0.1, 0.8, 0.1], [0.3, 0.4, 0.3], [0.8, 0.1, 0.1]):
    print(probabilities, "cross-entropy:", round(-np.log(probabilities[true_class]), 3))

Output

MSE: 5.0
MAE: 2.0
[0.1, 0.8, 0.1] cross-entropy: 0.223
[0.3, 0.4, 0.3] cross-entropy: 0.916
[0.8, 0.1, 0.1] cross-entropy: 2.303

Common mistakes

Using MSE for classification. Cross-entropy trains much better with a sigmoid or softmax.
Confusing the loss with the metric.

Check yourself

1. Which is more sensitive to outliers?

2. Which loss suits a 5-class image classifier?

Summary

The loss is the single number training minimises.
MSE or MAE for regression, and cross-entropy for classification.
You'll get what you put in the loss.

Finished reading?

Sources: Goodfellow, Bengio & Courville, Deep Learning