Definition
An activation function is a non-linear function applied to a neuron's weighted sum. It lets a network learn curved, complex patterns. Common ones are ReLU in hidden layers, sigmoid for yes-or-no outputs, and softmax for multi-class outputs.
In simple words
It's a dimmer switch on each neuron that decides how strongly to pass on its signal. Without it, a network can only draw straight lines. With it, stacks of bends can build any shape, like making a curve out of many short straight pieces.
Why it matters
The choice affects whether a network trains at all. Sigmoid in deep hidden layers caused vanishing gradients for years, and the switch to ReLU was a key step in the deep learning boom.
How it works
| Function | Output range | Used in | Note |
|---|---|---|---|
| ReLU | [0, ∞) | Hidden layers (the default) | Fast, but neurons can "die" |
| Leaky ReLU | (−∞, ∞) | Hidden layers | Small slope for negatives |
| GELU | (≈−0.17, ∞) | Transformers | A smooth version of ReLU |
| Sigmoid | (0, 1) | Binary output | Saturates, avoid in deep hidden layers |
| Tanh | (−1, 1) | RNNs, some hidden layers | Centred on zero |
| Softmax | (0, 1), sums to 1 | Multi-class output | Gives a probability for each class |
How to choose
- Hidden layers: start with ReLU. Use GELU for Transformers.
- Binary classification output: sigmoid.
- Multi-class output: softmax.
- Regression output: no activation at all.
The maths
- The score for class k
Softmax of the scores [2, 1, 0]:
e² = 7.389 e¹ = 2.718 e⁰ = 1
Sum = 11.107
Probabilities: 7.389/11.107 = 0.665
2.718/11.107 = 0.245
1/11.107 = 0.090
They add up to 1.In code
import numpy as np
relu = lambda x: np.maximum(0, x)
sigmoid = lambda x: 1 / (1 + np.exp(-x))
softmax = lambda z: np.exp(z) / np.exp(z).sum()
x = np.array([-2.0, 0.0, 2.0])
print("ReLU:", relu(x))
print("Sigmoid:", sigmoid(x).round(3))
print("Tanh:", np.tanh(x).round(3))
print("Softmax of [2, 1, 0]:", softmax(np.array([2.0, 1.0, 0.0])).round(3))Output
ReLU: [0. 0. 2.] Sigmoid: [0.119 0.5 0.881] Tanh: [-0.964 0. 0.964] Softmax of [2, 1, 0]: [0.665 0.245 0.09 ]
Common mistakes
- Using sigmoid throughout a deep network. Gradients shrink towards zero as they pass back.
- Adding softmax before a loss that already applies it, such as PyTorch's CrossEntropyLoss.
- Using softmax for multi-label tasks, where several labels can be true. Use a sigmoid per label.
Check yourself
Summary
- Activations add the non-linearity networks need.
- ReLU for hidden layers. Sigmoid or softmax for outputs.
- A poor choice can stop a network from training.
Finished reading?