What is reinforcement learning?

Definition

Reinforcement Learning (RL) is a kind of machine learning in which an agent learns by acting in an environment and receiving rewards. Its goal is to find a policy, a way of choosing actions, that gets the most total reward over time.

In simple words

Teaching a dog a trick: it tries things, gets a treat for the right one, and slowly learns what earns treats. Nobody shows it the right answer. It finds out by trying. It may also need to accept a small loss now for a bigger reward later, such as giving up a pawn in chess to win a queen.

Why it matters

RL suits problems of decisions over time, when there's no list of right answers. AlphaGo beat the world's best at Go using it, it's used in robot control, and RLHF uses related ideas to tune how chatbots respond.

How it works

Term	Meaning	Game example
Agent	The learner and decision-maker	The player
Environment	Everything the agent interacts with	The game
State (s)	The current situation	The board position
Action (a)	A choice the agent can make	A move
Reward (r)	The score signal after an action	+1 for a win
Policy (π)	The agent's rule for choosing actions	Its strategy
Discount (γ)	How much future rewards count	0.99 means it plans far ahead

The learning loop

The agent sees the state.
It chooses an action.
The environment returns a reward and a new state.
The agent updates what it knows, and repeats.

Exploration vs exploitation

Should you go to your favourite restaurant, or try a new one that might be better? Exploit what you know, or explore for something better. Every RL agent must balance this.

The maths

Return: what the agent maximises

G_{t} = r_{t + 1} + γ r_{t + 2} + γ^{2} r_{t + 3} + \dots = k = 0 \sum \infty γ^{k} r_{t + k + 1}

$G_{t}$: The total future reward from time t
$γ$: The discount factor, between 0 and 1
$r_{t}$: The reward at time t

Rewards of 1, 1 and 10 over three steps, with γ = 0.9:

G = 1 + 0.9×1 + 0.9²×10 = 1 + 0.9 + 8.1 = 10

In code

Epsilon-greedy explores 10% of the time, and exploits the best-known machine the rest of the time.

bandit.py

import random

random.seed(0)
true_win_rates = [0.2, 0.5, 0.8]     # the agent doesn't know these
estimates = [0.0, 0.0, 0.0]
pulls = [0, 0, 0]
total = 0

for step in range(1000):
    if random.random() < 0.1:
        arm = random.randrange(3)                 # explore
    else:
        arm = estimates.index(max(estimates))     # exploit
    reward = 1 if random.random() < true_win_rates[arm] else 0
    pulls[arm] += 1
    estimates[arm] += (reward - estimates[arm]) / pulls[arm]
    total += reward

print("Estimates:", [round(e, 2) for e in estimates])
print("Pulls per machine:", pulls)
print("Total reward:", total)

Output

Estimates: [0.22, 0.38, 0.8]
Pulls per machine: [67, 29, 904]
Total reward: 752

Common mistakes

Designing a reward that can be gamed. Agents find loopholes, which is called reward hacking.
Using RL when you already have labelled examples. Supervised learning is simpler and needs far less trial and error.

Check yourself

1. How does RL differ from supervised learning?

2. If γ = 0, the agent cares about:

Summary

An agent learns by acting and receiving rewards.
It maximises the total discounted reward, not the next one alone.
It must balance exploring and exploiting.

Finished reading?

Sources: Sutton & Barto, Reinforcement Learning: An Introduction