The Perceptron

Definition

In simple words

Should you go to the cricket match? You weigh a few things: is it a weekend, are friends going, is it raining? Good reasons add points, and bad ones take them away. If the total passes your limit, you go. That's a perceptron.

Why it matters

It's where neural networks began. Its best-known limit, the inability to solve XOR, was highlighted by Minsky and Papert in 1969 and led to the idea of hidden layers. Every neuron in a modern network is a descendant of this one.

How it works

1. Start with zero or random weights.
2. For each example, make a prediction.
3. If it's wrong, move the weights towards the right answer. If it's right, change nothing.
4. Repeat until it stops making mistakes.

A straight line can separate AND and OR, so a perceptron learns them. No single line can separate XOR, and so it fails. Add one hidden layer and the problem goes away.

The maths

In code

def train(data, epochs=10, rate=0.1):
    w = [0.0, 0.0]
    b = 0.0
    for _ in range(epochs):
        for (x1, x2), y in data:
            prediction = 1 if w[0]*x1 + w[1]*x2 + b > 0 else 0
            error = y - prediction
            w[0] += rate * error * x1
            w[1] += rate * error * x2
            b += rate * error
    return w, b

def accuracy(data, w, b):
    right = sum((1 if w[0]*x1 + w[1]*x2 + b > 0 else 0) == y for (x1, x2), y in data)
    return right / len(data)

AND = [((0,0),0), ((0,1),0), ((1,0),0), ((1,1),1)]
XOR = [((0,0),0), ((0,1),1), ((1,0),1), ((1,1),0)]

print("AND accuracy:", accuracy(AND, *train(AND)))
print("XOR accuracy:", accuracy(XOR, *train(XOR, epochs=100)))

AND accuracy: 1.0
XOR accuracy: 0.5

Common mistakes

• Expecting it to learn any pattern. One perceptron only draws a straight line.
• Thinking XOR killed neural networks for good. It led to multi-layer networks, which solve it easily.

Check yourself

Summary

• One neuron with a threshold.
• It learns by correcting its mistakes.
• It can't solve XOR, which is why hidden layers exist.