Maximum Likelihood and MAP Estimation

Definition

In simple words

You toss a coin 10 times and get 7 heads. What's your best guess for the coin's chance of heads? Almost everyone says 0.7. MLE is why: of all possible values, 0.7 makes getting 7 heads out of 10 most likely.

Why it matters

MLE is where common loss functions come from. Assume Gaussian noise and MLE gives you squared error. Assume a Bernoulli output and it gives cross-entropy. So training a neural network with these losses is doing maximum likelihood.

How it works

1. Choose a model for the data, such as Bernoulli for coin tosses.
2. Write the likelihood: the product of each example's probability.
3. Take the log. Products become sums, which are easier to work with and don't underflow.
4. Find the maximum, by setting the derivative to zero or using gradient descent.

The maths

k heads in n tosses, with chance of heads p.

Likelihood:     L(p) = pᵏ (1−p)ⁿ⁻ᵏ
Log-likelihood: ℓ(p) = k ln p + (n−k) ln(1−p)
Derivative:     k/p − (n−k)/(1−p) = 0
Solving:        p̂ = k/n

With 7 heads in 10 tosses: p̂ = 0.7

In code

import numpy as np

k, n = 7, 10
candidates = np.linspace(0.01, 0.99, 99)
log_likelihood = k * np.log(candidates) + (n - k) * np.log(1 - candidates)

best = candidates[np.argmax(log_likelihood)]
print("Best p by search:", round(best, 2))
print("Formula k/n:", k / n)

for p in (0.3, 0.5, 0.7, 0.9):
    print(f"Likelihood at p={p}: {p**k * (1 - p)**(n - k):.5f}")

Best p by search: 0.7
Formula k/n: 0.7
Likelihood at p=0.3: 0.00008
Likelihood at p=0.5: 0.00098
Likelihood at p=0.7: 0.00222
Likelihood at p=0.9: 0.00048

Common mistakes

• Reading likelihood as the probability of the parameters. It's the probability of the data, given the parameters.
• Multiplying many small probabilities in code. The product underflows to 0, so sum the logs instead.

Check yourself

Summary

• MLE picks the parameters that make the data most probable.
• Maximise the sum of log probabilities.
• Squared error and cross-entropy both come from MLE.