Probability Distributions and the Normal Curve

Definition

A continuous distribution describes a variable that can take any value in a range, using a probability density function (PDF). The chance of landing in a range is the area under the curve there. The most important one is the normal (Gaussian) distribution, the symmetric bell curve set by its mean and standard deviation.

In simple words

Measure the height of thousands of adults. Most are near the average, and fewer are very short or very tall. Draw it and you get a bell. Exact values have no chance of their own: nobody is exactly 170.000000 cm. So we ask for ranges, like "between 165 and 175".

Why it matters

Weights are initialised from these distributions. Linear regression assumes normal errors, and squared error comes from that. VAEs and diffusion models work with Gaussian noise. Standardising features turns them towards the standard normal scale.

How it works

Distribution	Shape and use
Uniform	Every value in a range is equally likely. Random initialisation.
Normal	Bell curve. Errors, measurements, and noise in diffusion models.
Exponential	Waiting time until an event, such as the next customer.
Beta	Values between 0 and 1. Beliefs about a probability, and A/B tests.
Student's t	Like the normal with heavier tails. Small samples.
Chi-square	Sums of squared normals. Statistical tests.

The 68-95-99.7 rule

For a normal distribution, about 68% of values fall within 1 standard deviation of the mean, 95% within 2, and 99.7% within 3.

The maths

Normal distribution

f (x) = \frac{1}{σ 2 π} e^{- \frac{( x - μ ) ^{2}}{2 σ ^{2}}}

$μ$: The mean: the centre of the bell
$σ$: The standard deviation: how wide the bell is
$e, π$: Fixed constants, about 2.718 and 3.142

Z-score (standardising)

z = \frac{x - μ}{σ}

Exam marks have mean 60 and standard deviation 10. You scored 80.

z = (80 − 60) / 10 = 2
You're 2 standard deviations above the mean.
About 95% of marks lie within ±2σ, so about 2.5% lie above you.

In code

normal.py

import numpy as np

rng = np.random.default_rng(0)
marks = rng.normal(loc=60, scale=10, size=100_000)

print("Mean:", round(marks.mean(), 1), "Std:", round(marks.std(), 1))
for k in (1, 2, 3):
    inside = np.mean(np.abs(marks - 60) < k * 10)
    print(f"Within {k} std: {inside:.1%}")
print("Scored above 80:", f"{np.mean(marks > 80):.1%}")

Output

Mean: 60.0 Std: 10.0
Within 1 std: 68.4%
Within 2 std: 95.5%
Within 3 std: 99.7%
Scored above 80: 2.3%

Common mistakes

Reading the height of the PDF as a probability. Only the area under it is. A narrow PDF can even rise above 1.
Assuming all data is normal. Incomes, for example, are heavily skewed. Plot your data first.

Check yourself

1. About what share of values lie within 2 standard deviations of the mean in a normal distribution?

2. Mean 50 and σ = 5. What is the z-score of 40?

Summary

Continuous variables use densities, and probabilities are areas.
The normal distribution is set by μ and σ.
A z-score says how many σ a value is from the mean.

Finished reading?

Sources: Deisenroth, Faisal & Ong, Mathematics for Machine Learning