2014
GAN Goodfellow et al.
Trained on MNIST and faces. The samples were small and blurry.
探索 · an exploration
Train two neural networks in your browser. One generates samples while the other learns to distinguish them from real data.
the game
Scroll and the diagram fills in step by step.
01
A generator turns random numbers into a sample, such as a point or an image. It learns from the discriminator rather than receiving real examples directly.
02
A discriminator receives real and generated samples. For each one, it estimates the probability that the sample is real.
03
The discriminator trains as a binary classifier. Its loss penalizes labeling real samples as generated and generated samples as real.
04
The generator learns to increase the probability that the discriminator assigns to its samples being real. Backpropagation through the discriminator computes how to change the generator weights.
05
At the theoretical optimum, the generated distribution matches the real distribution. The discriminator cannot distinguish them and assigns a probability of one half to either.
the lab
Train a generator and discriminator on a one-dimensional distribution with one or two peaks. Each network has 24 hidden units.
0
steps
—
D on real samples
—
D on generated samples
loss · discriminator · generator
The ink curve shows the real distribution. The vermilion curve shows the generated distribution. The gold dashes show the discriminator output, which is high where it classifies samples as real. When the distributions match, that output approaches one half.
Switch to two bumps and reset a few times. The generator sometimes learns both peaks and sometimes concentrates on just one. Failing to represent parts of the real distribution is called mode collapse.
the objective
The discriminator maximizes this objective while the generator minimizes it.
The discriminator, D
Maximizes this objective by making log D(x) large on real samples and log(1 − D(G(z))) large on generated samples.
The generator, G
Changes only the second term. In practice, the generator maximizes log D(G(z)) instead. This provides a stronger gradient when the discriminator easily rejects generated samples.
For a fixed generator, the best possible discriminator is
D*(x) = pdata(x)pdata(x) + pg(x)
The discriminator output is high where real samples are more likely than generated ones. It equals one half where the two densities match. Goodfellow et al. showed that the objective reaches its global optimum when the generated distribution equals the real one.
the trouble
The generator produces samples from only part of the real distribution. With two peaks, it may learn one and miss the other.
If the discriminator becomes too accurate early in training, the generator gradient can approach zero. The non-saturating loss reduces this problem.
The networks optimize competing objectives, so their losses can oscillate rather than settle. Later work used the Wasserstein distance to improve training stability.
the lineage
Selected developments in image generation between 2014 and 2020.
2014
Trained on MNIST and faces. The samples were small and blurry.
2015
Convolutional generators and discriminators, with architectural guidelines that improved training stability.
2016
Condition the generator on an input image to learn mappings such as edges to photographs or maps to satellite views.
2017
Use a critic that estimates the Wasserstein distance between distributions to provide more useful gradients during training.
2017
Learn image translation without paired examples. Translating a horse to a zebra and back should reproduce the original image.
2017
Increase both networks from 4×4 to 1024×1024 resolution during training to generate high-resolution faces.
2018
Larger models and batches improved the quality of images generated from ImageNet.
2018
Inject style information at each layer to control image attributes, such as hairstyle, separately.
2020
Diffusion models learn to remove noise one step at a time. They later surpassed GANs on image-generation benchmarks.