What is a neural network?

Definition

A neural network is a model made of simple connected units, called neurons, arranged in layers. Each neuron computes a weighted sum of its inputs and passes it through a non-linear function. Training adjusts the weights so the whole network maps inputs to the right outputs.

In simple words

Picture a team deciding whether to buy a house. Each person weighs a few facts and gives a small opinion. Those opinions go to a second group, who form a bigger view, and so on until a final answer comes out. Training means slowly learning how much to trust each opinion.

Why it matters

Neural networks can learn very complex patterns from raw data, which is why they lead in vision, language and speech. Every LLM and image generator is one. They're the reason for most recent progress in AI.

How it works

Part	What it does
Input layer	Receives the features
Hidden layers	Learn useful combinations
Output layer	Gives the prediction
Weights	How much each connection counts
Bias	Shifts the neuron's output
Activation	Adds non-linearity

Why non-linearity matters

Stacking only linear layers gives another linear function, however many layers you use. The activation function is what lets networks learn curves.

The maths

One neuron

\overset{y}{^} = σ (i \sum w_{i} x_{i} + b) = σ (w^{⊤} x + b)

$x_{i}$: An input
$w_{i}$: The weight of that input
$b$: The bias
$σ$: The activation function

A whole layer, as matrices

h = σ (W x + b)

One neuron with weights [0.5, −1], bias 0.5, inputs [2, 1] and ReLU activation:

z = 0.5×2 + (−1)×1 + 0.5 = 1 − 1 + 0.5 = 0.5
ReLU(0.5) = max(0, 0.5) = 0.5

In code

neuron.py

import numpy as np

def relu(z):
    return np.maximum(0, z)

x = np.array([2.0, 1.0])
W = np.array([[0.5, -1.0], [1.0, 1.0]])     # 2 neurons in this layer
b = np.array([0.5, -2.0])

hidden = relu(W @ x + b)
print("Hidden layer output:", hidden)

output_weights = np.array([1.5, 2.0])
print("Final output:", output_weights @ hidden)

Output

Hidden layer output: [0.5 1. ]
Final output: 2.75

Common mistakes

Forgetting the activation function, which leaves you with a plain linear model.
Starting with a huge network. Begin small, and grow only if it underfits.

Check yourself

1. What does training change in a neural network?

2. Without activation functions, a deep network behaves like:

Summary

A network is layers of neurons, each a weighted sum plus an activation.
Training adjusts the weights and biases.
Non-linearity is what makes depth useful.

Finished reading?

Sources: Goodfellow, Bengio & Courville, Deep Learning