Entropy, Cross-Entropy and KL Divergence

Definition

Cross-entropy measures the average surprise when you use a predicted distribution q to describe events that really follow p. KL divergence is the extra surprise this causes compared with knowing p itself: cross-entropy minus entropy. KL is never negative, and is zero only when p and q are the same.

In simple words

A forecaster says 90% chance of rain. If it rains, the forecast was nearly right. If it doesn't, the forecast was badly wrong and earns a big penalty. Cross-entropy is exactly this scoring rule. Being confident and wrong costs much more than being unsure and wrong.

Why it matters

Cross-entropy is the standard loss for classification, from logistic regression to GPT, which is trained to predict the next token. Minimising it is the same as maximising likelihood. KL divergence appears in VAEs, in distribution drift monitoring and in knowledge distillation.

How it works

In classification, the true distribution p is simply 1 for the right class and 0 for the rest. All terms but one vanish, so the loss is just −log of the probability the model gave to the right answer.

Probability given to the right class	Loss = −ln(p)
0.99	0.01
0.90	0.11
0.50	0.69
0.10	2.30
0.01	4.61

The maths

Cross-entropy

H (p, q) = - x \sum p (x) lo g q (x)

$p$: The true distribution
$q$: The model's predicted distribution

KL divergence

D_{K L} (p ∥ q) = x \sum p (x) lo g \frac{p ( x )}{q ( x )} = H (p, q) - H (p)

Binary cross-entropy (log loss)

L = - \frac{1}{n} i = 1 \sum n [y_{i} lo g \overset{y}{^}_{i} + (1 - y_{i}) lo g (1 - \overset{y}{^}_{i})]

$y_{i}$: The true label, 0 or 1
$\overset{y}{^}_{i}$: The predicted probability of the positive class (label 1)

Three categories. The right answer is the second one. The model says [0.2, 0.7, 0.1].

p = [0, 1, 0]
H(p,q) = −[0×ln 0.2 + 1×ln 0.7 + 0×ln 0.1] = −ln 0.7 ≈ 0.357

A better model says [0.05, 0.9, 0.05]: loss = −ln 0.9 ≈ 0.105

In code

cross_entropy.py

import numpy as np

def cross_entropy(p, q):
    return -np.sum(p * np.log(q))

def kl(p, q):
    return np.sum(p * np.log(p / q))

true_class = np.array([0, 1, 0])
print("Okay model:", round(cross_entropy(true_class, np.array([0.2, 0.7, 0.1])), 3))
print("Better model:", round(cross_entropy(true_class, np.array([0.05, 0.9, 0.05])), 3))
print("Confident and wrong:", round(cross_entropy(true_class, np.array([0.98, 0.01, 0.01])), 3))

p = np.array([0.4, 0.6])
q = np.array([0.5, 0.5])
print("KL(p||q):", round(kl(p, q), 4), " KL(q||p):", round(kl(q, p), 4))

Output

Okay model: 0.357
Better model: 0.105
Confident and wrong: 4.605
KL(p||q): 0.0201  KL(q||p): 0.0204

Common mistakes

Passing q values of exactly 0. log(0) is infinite, so frameworks clip probabilities or work from raw logits.
Adding softmax twice. PyTorch's CrossEntropyLoss already applies it, so give it raw logits.
Calling KL divergence a distance. It isn't symmetric.

Check yourself

1. A model gives the right class probability 1.0. What is the cross-entropy loss?

2. Which earns the highest penalty?

3. When is KL(p‖q) zero?

Summary

Cross-entropy scores predicted probabilities, and punishes confident mistakes hard.
In classification, it's −log of the right class's probability.
KL divergence = cross-entropy − entropy. It's never negative and isn't symmetric.

Finished reading?

Sources: Goodfellow, Bengio & Courville, Deep Learning, ch. 3