GAN Networks

"The most interesting idea in the last 20 years in Machine Learning" — [Yann LeCun, 2016]

Until the arrival of Transformers and diffusion models, Generative Adversarial Networks^{[goodfellow2014generative]} had broadly impacted the generative landscape, showing great advances in many tasks such as image generation, natural language, and music.

Back in the day, websites like thispersondoesnotexist and its variants became famous. Today we are used to seeing realistic AI-generated images, but back then it was surprising to think that those faces did not correspond to real people, but had instead been generated by a model.

Nowadays, GANs are no longer the default option for many generation tasks, but they remain useful in specific niches, when fast sampling or translation between domains is needed.

They are inspired by game theory: two models, a generator and a discriminator, compete while also improving each other.

Generative adversarial networks — Source: miamiamia0103

Architecture

Generator network: receives as input a vector of random numbers or Gaussian noise (which is why it is sometimes denoted $z$ ), from which it generates data.
Discriminator network: it has to identify whether the data it receives is real (part of the training set) or fake (generated by the generator network).

Playground

Explanation

The discriminator's task is to differentiate whether the samples it receives correspond to the training data distribution ('real' data) or whether, on the contrary, they are samples generated by the generator ('fake' data). The generator's objective, therefore, is to produce samples that are as realistic as possible in order to try to fool the discriminator.

At the beginning of training, neither network is good at its task. The generator produces blurry, incoherent images, and the discriminator is not good at telling real data from fake data apart.

However, as training progresses, the generator produces better examples to try to fool the discriminator, and the discriminator, by seeing more real data, learns to tell it apart from generated data, so the two enter a zero-sum game.

Normally, after training the discriminator is discarded, since the main interest lies in data generation. However, the discriminator is a good binary classifier that could be used for other tasks.

Applications

The most studied and popularly known application area is image generation. Here are some applications:

Conditional image generation: for example, BigGAN^{[brock2018large]} can generate ImageNet samples conditioned on a specific class. StyleGAN^{[karras2019style]} is capable of generating high-resolution face images and introduces a style-based architecture that allows interpolation in the latent space and control of characteristics at different scales, such as overall face geometry, hairstyle, or texture.
Paired image-to-image generation: data in the form $(x_n, y_n)$ can be used to build conditional generative models $p(x \mid y)$ . In some cases, the conditioning variable $y$ has the same dimension as the output variable $x$ . Thus, a model can be used to perform image-to-image translation, a process where an image is generated conditioned on the style of another. An example of this would be colorization, where the model takes a black-and-white photo and "fills in" the colors based on the patterns it has learned, or segmentation, where the model converts a color-label scheme (like a map where each color is an object) into a realistic photograph that exactly respects those shapes.
Unpaired image-to-image generation: one limitation of conditional GANs is the difficulty of collecting paired data. It is much easier to collect unpaired data, for example, a set of daytime images $D_x$ and a set of nighttime images $D_y$ . We assume that the sets $D_x$ and $D_y$ come from the marginal distributions $p(x)$ and $p(y)$ respectively. Since these marginal distributions do not by themselves determine a unique correspondence between the two domains, models like CycleGAN learn transformations in both directions and introduce additional constraints, such as cycle consistency. This is called unsupervised domain translation and has applications such as style transfer.

Evolution of GANs in image generation — From left to right: original GAN (2014), DCGAN (2015), CoupledGAN (2016), ProgressiveGAN (2017), BigGAN (2018). Source.

Applications in other domains

Video generation: spatio-temporal coherence is obtained by ensuring that the discriminator has access to real data and to the generated sequences in order, thereby penalizing the generator when it generates individually realistic frames without respecting temporal order.
Audio generation: GAN architectures have been developed to replicate instruments with GANSynth^{[engel2019gansynth]}, to convert voices^{[kaneko2020cyclegan]}, and to directly generate waveforms with WaveGAN^{[donahue2018adversarial]}.
Text generation: there are several text-data tasks for which GAN-based approaches have been developed, such as conditional text generation and text style transfer. Text data is usually represented as discrete values (at the character or word level), indicating membership in a set of a given vocabulary.
Domain adaptation: an important task in machine learning is correcting shifts in data distribution, since in many cases the distribution on which the model is evaluated at inference time is not exactly the same as the training data distribution. In general, the image generation approaches mentioned focus on pixel-level adaptations, such as pix2pix^{[isola2017image]}. However, extensions of these approaches to the general domain adaptation problem seek to do so not only in the observed data space, but also at the feature level.

Loss function

On one hand, we want the discriminator to become an expert at identifying what is real and what is fake:

Identifying the real: We maximize $\mathbb{E}_{x \sim p_r}[\log D(x)]$ , forcing the discriminator to assign a probability close to 1 to any data coming from the real set.
Identifying the fake: We maximize $\mathbb{E}_{z \sim p_z}[\log(1 - D(G(z)))]$ , forcing the discriminator to assign a probability close to 0 to any data generated by G.

On the other hand, the generator is trained to increase its chances of producing a high probability for a fake example, thus minimizing $\mathbb{E}_{z \sim p_z}[\log(1 - D(G(z)))]$ .

What is E and why use log?

The symbol $\mathbb{E}$ means Mathematical Expectation (or expected value). It is simply an average. When we write $\mathbb{E}_{x \sim p_r}[\dots]$ , we are just saying: "Calculate the average of $[\dots]$ for the data $x$ that comes from distribution $r$ ".
Small probabilities are often multiplied together, and multiplying very small numbers (e.g. $0.0001 \times 0.0001$ ) makes the result tend toward zero. The logarithm turns those multiplications into sums (e.g.: $\log(a \cdot b) = \log(a) + \log(b)$ ), which makes the calculations much more stable.
In the case of GANs, if the discriminator is very confident about something but is wrong (for example, it says a fake datum is 99% real), the logarithm of a number close to zero ( $\log(1 - 0.99)$ in this case) is a very large negative number. This forces the network to correct its error immediately.

By combining both parts, we are playing a 'minimax' game in which we must optimize the following loss function:

\min_G \max_D L(D, G) = \mathbb{E}_{x \sim p_r(x)}[\log D(x)] + \mathbb{E}_{z \sim p_z(z)}[\log(1 - D(G(z)))] = \mathbb{E}_{x \sim p_r(x)}[\log D(x)] + \mathbb{E}_{x \sim p_g(x)}[\log(1 - D(x))]

Limitations

Although GANs have shown great success in generating realistic images, their training can be slow and unstable.

Each model updates its cost independently without taking the other "player" into account. The simultaneous gradient update of both models does not guarantee convergence. This can sometimes lead to mode collapse. During training, the generator can collapse into a configuration in which it always produces the same outputs. Even though the generator may be able to fool the discriminator, it fails to learn to represent the complex real-world data distribution and gets stuck in a small space with extremely low variety.

Training a GAN faces a dilemma:

If the discriminator is not good, the feedback the generator receives is not good either, and as a result the loss function does not represent reality (causing the generator to fail to properly model the real data distribution).
If the discriminator is very good, the gradient of the loss function drops close to zero and learning becomes slow or even gets stuck (vanishing gradient).