All courses

Advanced · module A12

Reinforcement Learning

By the end: Formulate problems as MDPs and implement tabular and deep RL algorithms

Start here
Covers
Agents that learn by acting, from Markov decision processes to deep RL
Contents
3 chapters, 12 topics
Time
About 3 hours
Level
Advanced
Your progress
0 / 12

This module builds on Deep Learning with PyTorch. We suggest looking at it first, but the choice is yours.

Chapter 1 Foundations

  1. 1.1

    Markov Decision Processes

    States, actions, rewards, returns and discounting

    Interactive: a grid world

    Gₜ = Σ γᵏ rₜ₊ₖ₊₁

    15 min Activity
  2. 1.2

    Bellman Equations and Dynamic Programming

    Value functions, policy iteration and value iteration

    Code: value iteration

    V(s) = maxₐ Σ P(s′|s,a)[r + γV(s′)]

    15 min Read
  3. 1.3

    Multi-Armed Bandits

    Exploration and exploitation, UCB and Thompson sampling

    Code: compare bandit strategies

    12 minBeing written

Chapter 2 Model-Free Learning

  1. 2.1

    Monte Carlo and Temporal-Difference Learning

    Learning from complete and partial episodes

    Code: TD(0)

    12 minBeing written
  2. 2.2

    Q-Learning and SARSA

    Off-policy and on-policy control

    Code: Q-learning on a grid

    Q ← Q + α[r + γ maxQ′ − Q]

    15 min Read Activity
  3. 2.3

    Deep Q-Networks

    Replay buffers, target networks and Atari

    Code: DQN on a control task

    20 minBeing written
  4. 2.4

    Policy Gradients and Actor-Critic

    REINFORCE, baselines, A2C and PPO

    Code: PPO on a control task

    20 minBeing written

Chapter 3 Beyond the Basics

  1. 3.1

    Model-Based RL and World Models

    Learning a model of the environment to plan

    Reading and demo

    12 minBeing written
  2. 3.2

    Multi-Agent Reinforcement Learning

    Self-play, cooperation and competition

    Lab: self-play

    10 minBeing written
  3. 3.3

    Case Studies: AlphaGo to MuZero

    How search and learning beat world champions

    Case study

    10 minBeing written
  4. 3.4

    Reinforcement Learning in Robotics

    Simulation to reality and domain randomisation

    Lab: a robot arm in simulation

    12 minBeing written
  5. 3.5

    Module Review and Assessment

    Coding tasks and a scored quiz

    Graded project

    20 minBeing written