Entropy

Definition

In simple words

A fair coin is hard to predict, so it has high entropy: exactly 1 bit. A coin that lands heads 99% of the time is easy to predict, so it has low entropy. A two-headed coin has none. Entropy tells you how surprised you should expect to be.

Why it matters

Decision trees pick splits that reduce entropy the most. Cross-entropy, the standard loss for classification and LLMs, is built directly on it. It also underlies text compression and how sampling temperature behaves.

How it works

The maths

A box has 3 red balls and 1 blue, so p = 0.75 and 0.25.

H = −[0.75 × log₂(0.75) + 0.25 × log₂(0.25)]
  = −[0.75 × (−0.415) + 0.25 × (−2)]
  = 0.311 + 0.5
  ≈ 0.811 bits

In code

import math

def entropy(probabilities):
    return 0.0 - sum(p * math.log2(p) for p in probabilities if p > 0)

print("Fair coin:", round(entropy([0.5, 0.5]), 3))
print("Biased coin:", round(entropy([0.9, 0.1]), 3))
print("3 red, 1 blue:", round(entropy([0.75, 0.25]), 3))
print("Fair die:", round(entropy([1/6] * 6), 3))
print("Certain:", entropy([1.0]))

Fair coin: 1.0
Biased coin: 0.469
3 red, 1 blue: 0.811
Fair die: 2.585
Certain: 0.0

Common mistakes

• Forgetting the minus sign. Logs of probabilities are negative, and the minus makes entropy non-negative.
• Computing log(0). By convention, 0·log 0 counts as 0, so skip zero probabilities.

Check yourself

Summary

• Entropy is average surprise, or uncertainty.
• It's highest for equal chances, and zero when certain.
• Decision trees and cross-entropy loss are built on it.