All courses

Advanced · module A11

Generative Models and Multimodal AI

By the end: Explain and implement the main families of generative models, and build a multimodal application

Covers
How machines generate images, audio, video and 3D, and how models combine senses
Contents
2 chapters, 12 topics
Time
About 3 hours
Level
Advanced
Your progress
0 / 12

This module builds on Deep Learning with PyTorch. We suggest looking at it first, but the choice is yours.

Chapter 1 Generative Model Families

  1. 1.1

    Autoencoders and Variational Autoencoders

    Latent spaces, the ELBO and the reparameterisation trick

    Code: a VAE on digits

    ELBO = E[log p(x|z)] − KL(q(z|x) ‖ p(z))

    18 minBeing written
  2. 1.2

    Generative Adversarial Networks

    Generator, discriminator and the training game

    Code: a simple GAN

    18 minBeing written
  3. 1.3

    Normalising Flows

    Invertible transformations and exact likelihood

    Reading and demo

    10 minBeing written
  4. 1.4

    Diffusion Models

    Adding noise, learning to remove it, and sampling

    Code: a small diffusion model

    25 minBeing written
  5. 1.5

    Latent Diffusion and Guidance

    Text-to-image systems and classifier-free guidance

    Lab: controlled image generation

    15 minBeing written
  6. 1.6

    Flow Matching and Fast Sampling

    Newer, faster generative training

    Reading and demo

    10 minBeing written

Chapter 2 Multimodal AI

  1. 2.1

    Contrastive Image-Text Models

    CLIP-style training and zero-shot classification

    Code: zero-shot classification

    12 minBeing written
  2. 2.2

    Vision-Language Models

    Models that answer questions about images and documents

    Lab: visual question answering

    12 minBeing written
  3. 2.3

    Audio, Music and Video Generation

    How sound and video generators work

    Lab: compare generators

    10 minBeing written
  4. 2.4

    Evaluating Generative Models

    FID, CLIP score and human evaluation

    Code: compute FID

    10 minBeing written
  5. 2.5

    Provenance and Watermarking

    Detecting and labelling generated content

    Case study discussion

    8 minBeing written
  6. 2.6

    Module Review and Assessment

    Coding tasks and a scored quiz

    Graded project

    20 minBeing written