All courses

Advanced · module A7

Sequence Models, Transformers and Modern NLP

By the end: Implement a Transformer, fine-tune pre-trained language models, and build NLP and speech systems

Start here
Covers
From word embeddings to Transformers built from scratch, with speech
Contents
3 chapters, 14 topics
Time
About 4 hours
Level
Advanced
Your progress
0 / 14

This module builds on Deep Learning with PyTorch. We suggest looking at it first, but the choice is yours.

Chapter 1 Sequence Models

  1. 1.1

    Word Embeddings

    Word2Vec, GloVe and fastText

    Code: train and explore embeddings

    15 minBeing written
  2. 1.2

    Recurrent Networks, LSTMs and GRUs

    Hidden state, gates and backpropagation through time

    Code: an LSTM text generator

    hₜ = tanh(Wₕhₜ₋₁ + Wₓxₜ)

    18 minBeing written
  3. 1.3

    Sequence-to-Sequence Models

    Encoders, decoders, teacher forcing and beam search

    Code: a translator

    15 minBeing written

Chapter 2 The Transformer

  1. 2.1

    The Attention Mechanism

    Why attention replaced fixed memory

    Interactive: attention weights

    12 minBeing written
  2. 2.2

    Self-Attention Step by Step

    Queries, keys, values and scaled dot-product attention

    Code: self-attention in NumPy

    softmax(QKᵀ / √dₖ)V

    20 min Read
  3. 2.3

    Multi-Head Attention and Positional Encoding

    Parallel heads and how word order is added

    Code: positional encodings

    15 minBeing written
  4. 2.4

    The Transformer Architecture

    Encoder and decoder blocks, residual connections and layer normalisation

    Code: build a Transformer from scratch

    30 minBeing written
  5. 2.5

    Encoder, Decoder and Encoder-Decoder Models

    BERT, GPT and T5 families

    Code: compare the three

    12 minBeing written
  6. 2.6

    Tokenisation

    Byte-pair encoding, WordPiece and SentencePiece

    Code: train a tokeniser

    12 minBeing written

Chapter 3 Applied NLP

  1. 3.1

    The Hugging Face Ecosystem

    Models, datasets, tokenisers and the Trainer

    Code: fine-tune a classifier

    18 minBeing written
  2. 3.2

    NLP Tasks with Transformers

    Named entities, question answering, summarisation and translation

    Code: four task pipelines

    15 minBeing written
  3. 3.3

    Multilingual and Indian-Language NLP

    Code-mixing, low-resource languages and Indian language models

    Code: a Hindi classifier

    12 minBeing written
  4. 3.4

    Speech Recognition and Synthesis

    Audio features, Whisper-style recognition and text-to-speech

    Code: transcribe and synthesise

    15 minBeing written
  5. 3.5

    Module Review and Assessment

    Coding tasks and a scored quiz

    Graded project

    20 minBeing written