Introduction

Who is this course for?

This course is the material I would have liked to find when I started getting interested in generative models (back in 2017).

For people with an engineering background, exploring the Machine Learning literature can feel intimidating. Algebra, calculus, statistics, etc., are subjects taught in the first years of a bachelor's degree, and their details tend to be forgotten if you don't keep practicing them in later years. As a result, concepts like ELBO optimization, density transformation, or optimal transport can seem harder than they actually are.

This course is aimed at anyone who identifies with this situation, has at least a minimal technical background, and, above all, is interested in understanding at a lower level how these models work. My goal is to walk you through learning the underlying concepts while also enabling you use them from a practical standpoint. That's why, in every chapter, you'll find code to reinforce your understanding of the explanations and to let you experiment on your own.

Before we start, two things to clarify:

1. If you're looking for material that goes deep into statistics, calculus, or probability, this isn't the place. On the internet you can find plenty of high-quality content better than these notes.

2. I've tried to keep a visually appealing aesthetic so this is a resource you'll want to come back to, but it's important to note that understanding and consolidating knowledge takes time.

"Learning is not supposed to be fun. It doesn't have to be actively not fun either, but the primary feeling should be that of effort." — Andrej Karpathy

Generative AI

As you can see in the figure, within the broader landscape of artificial intelligence, generative AI covers methods and techniques capable of generating new data samples.

From a probabilistic point of view, we fit a model to the training distribution $p_{\mathrm{data}}(x)$ and then sample from the learned distribution $p_\theta(x)$.

This process applies to a variety of domains. Below are some examples you're probably already familiar with.

Image generation

Nano Banana (Google), GPT Image 2 (OpenAI), DALL·E, Stable Diffusion, Adobe Firefly, Midjourney, FLUX…

Text / code generation

ChatGPT, Claude, Gemini, Grok, Mixtral…

Audio generation

Suno, ElevenLabs, WaveNet, VALL-E, Stable Audio, Udio…

Conditional generative models

We generally want to control what gets generated. That's why conditional generative models are used, in the form $p(x \mid c)$. Some examples:

• $c$ = text, $x$ = image: a text-to-image model. Example: Nano Banana.
• $c$ = image, $x$ = text: an image-to-text model. Example: give ChatGPT an image and ask it to describe it.
• $c$ = image, $x$ = image: an image-to-image model. Example: image editing in Adobe Firefly.
• $c$ = audio, $x$ = text: a speech-to-text model. Example: Whisper, or any voice dictation.
• $c$ = text, $x$ = audio: a text-to-speech model. Example: ElevenLabs TTS.
• $c$ = text, $x$ = text: a sequence-to-sequence model. Example: translators (text generation models would also fall under this category).

Other applications

The nice thing about modeling a distribution is that it doesn't just let you generate data, but a lot of other things too.

Density estimation

This consists of evaluating the density that the model assigns to a data sample, i.e., computing $p(x)$. This is useful for problems such as data compression, anomaly detection, classification, model comparison, etc.

The simplest approach to density estimation is to use kernel density estimation or KDE. However, KDE runs into limitations in high-dimensional spaces (the curse of dimensionality), where the density of the data becomes extremely sparse.

Modern generative networks mitigate this limitation by learning compact, complex representations that make it possible to estimate or model density in a much more efficient and scalable way for high-dimensional data such as images, audio, or text.

Data imputation

This consists of "filling in" missing values in a data vector or matrix. For example, a simple way to fill in missing data in tabular data is to use the mean value of each column (mean imputation).

We can generalize this by learning a joint model over all the variables and, for each incomplete record, sampling the missing values $X_m$ conditioned on the observed ones $X_o$: $p_\theta(X_m\mid X_o)$. This is called multiple imputation.

A generative model can be used to fill in more complex types of data, such as masked pixels in an image.

Representation learning

Some types of generative models have latent variables $z$, which can be understood as the causes that generated the observed data $x$. These variables generally reduce the dimensionality of the data and make it possible to discover patterns / useful representations.

Interpolation in the latent space

The latent space is where the latent variables live. It can be understood as the space into which models compress the data.

One of the most interesting capabilities of certain latent-variable models is the ability to generate samples with certain desired properties by interpolating between existing data points in the latent space.

In the following example we reduce the latent space to two dimensions: each chair is a point on a plane. We pick four samples and interpolate their coordinates. Decoding the intermediate points gives us representations that combine properties of the selected samples.

Data compression

Models that can assign high probability to frequently occurring data and low probability to rare data can be used for data compression, since they can assign shorter codes to the most common elements. Compression is a beautiful field that overlaps directly with Information Theory. We'll cover some of this in the Variational Autoencoders chapter.

Timeline

This timeline brings together a selection of particularly influential works in the evolution of generative models. Don't worry if you don't understand what they're about right now — it's a reference you can come back to later.

We'll also cover foundational models. However, the literature is much broader than that, so I've decided to include here works related to generative models, along with some architectures and training techniques (such as CLIP, LoRA, RLHF, or DPO) that, while not generative themselves, have been key to their evolution.

Evolution of generative models

AllLLMImageMultimodalDiffusionGANAlignmentReasoningEfficiencyArchitecture

26 papers visible.

2013

2014

2015

2016

2017

2018

2019

2020

2021

2022

2023

2024

2025

2026

Variational autoencoderAuto-Encoding Variational BayesFormulates learning latent-variable models via variational inference and the reparameterization trick.GANGenerative Adversarial NetsIntroduces a minimax training game between a generator and a discriminator: one produces samples and the other tries to tell them apart from real data.Diffusion modelsDeep Unsupervised Learning using Nonequilibrium ThermodynamicsIntroduces a forward process that gradually destroys the structure of the data by adding noise, and learns a reverse dynamics that recovers it.PixelRNNPixel Recurrent Neural NetworksDecomposes an image's distribution into pixel-wise conditional probabilities and models them sequentially.TransformerAttention Is All You NeedReplaces recurrence and convolutions with attention blocks in a sequence-to-sequence model.VQ-VAENeural Discrete Representation LearningIntroduces an autoencoder with a discrete latent code learned via vector quantization.GPTImproving Language Understanding by Generative Pre-TrainingCombines autoregressive generative pretraining on unlabeled text with supervised fine-tuning for downstream tasks.StyleGANA Style-Based Generator Architecture for Generative Adversarial NetworksReformulates a GAN's generator with a mapping network and style controls injected at different scales.GPT-2Language Models are Unsupervised Multitask LearnersScales up GPT and evaluates it zero-shot.Scaling lawsScaling Laws for Neural Language ModelsEmpirically studies how language-model loss changes with the number of parameters, data, and compute..GPT-3Language Models are Few-Shot LearnersPresents GPT-3, a 175-billion-parameter model, and its in-context learning capabilities.RAGRetrieval-Augmented Generation for Knowledge-Intensive NLP TasksCombines a retrieval model with a sequence-to-sequence generator and marginalizes over the retrieved documents.DDPMDenoising Diffusion Probabilistic ModelsTrains a model to gradually invert a fixed process that adds noise to the data.DALL·EZero-Shot Text-to-Image GenerationJointly models text and image tokens with an autoregressive Transformer, using a discrete VAE to compress images.CLIPLearning Transferable Visual Models From Natural Language SupervisionLearns aligned representations via a contrastive objective over text-image pairs.LoRALoRA: Low-Rank Adaptation of Large Language ModelsKeeps the pretrained weights frozen and introduces low-rank updates in new matrices.Latent DiffusionHigh-Resolution Image Synthesis with Latent Diffusion ModelsMoves the diffusion process from pixel space to a latent space learned by an autoencoder.Chain-of-ThoughtChain-of-Thought Prompting Elicits Reasoning in Large Language ModelsIncludes examples with intermediate reasoning steps inside the prompt before asking for an answer.InstructGPT / RLHFTraining Language Models to Follow Instructions with Human FeedbackCombines supervised fine-tuning on demonstrations, a reward model learned from human preferences, and reinforcement-learning optimization.Chinchilla scalingTraining Compute-Optimal Large Language ModelsRe-examines the optimal split, given a compute budget, between parameters and training tokens.Flow MatchingFlow Matching for Generative ModelingProposes training the vector fields of continuous normalizing flows by regressing on conditional probability paths.Diffusion TransformerScalable Diffusion Models with TransformersReplaces the usual diffusion U-Net with a Transformer that processes patches of the latent space.DPODirect Preference Optimization: Your Language Model is Secretly a Reward ModelDerives a classification objective that directly optimizes a language model using pairs of preferred and rejected responses.MambaMamba: Linear-Time Sequence Modeling with Selective State SpacesIntroduces input-dependent selection mechanisms in state-space models along with a hardware-efficient implementation.Llama 3The Llama 3 Herd of ModelsDocuments a family of autoregressive models and its data, scaling, evaluation, safety, and post-training pipeline.DeepSeek-R1DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement LearningStudies how to train language models for reasoning tasks using reinforcement learning with verifiable rewards, followed by stages of fine-tuning and RL.