2014
GAN Goodfellow et al.
Two networks, one game. Trained on MNIST and faces, the samples were small and blurry and the idea was enormous.
探索 — an exploration
A forger and a critic, trained against each other until the forgeries pass. Below, a real one learns, live, in the page.
After Goodfellow et al., “Generative Adversarial Nets”, 2014. The idea is theirs. The small networks on this page are mine, and they run in your browser.
the game
A generator and a discriminator, and the gradient that ties them. Scroll, and the diagram fills in as the game is played.
01
A generator takes a handful of random numbers and turns them into a candidate: a point, an image, a sentence. At first it has no idea what real ones look like. It never will, directly. Its output is a forgery, and it starts out a bad one.
02
A discriminator is shown a mix of real examples and forgeries and asked one question about each: real, or fake? It answers with a probability. Early on the forgeries are so poor that this is easy.
03
The critic is trained like any classifier. It is rewarded for calling real things real and forgeries fake, and it sharpens its judgement with every batch.
04
This is the trick. The forger’s reward is the critic’s verdict on its work, and because the critic is differentiable, the forger can follow the gradient of that verdict back through the critic and into its own weights. It learns which way to move to be believed.
05
The two improve against each other. If it works, the forgeries become indistinguishable from the real thing and the critic is reduced to a coin flip: one half, everywhere. That is the whole idea, in two networks and a game.
the lab
The world is a line, and everything real on it is drawn from one or two bumps. A forger with 24 hidden units learns to draw from it, and a critic of the same size learns to catch it. Nothing here is recorded. It trains as you watch.
0
steps
—
critic on real
—
critic on forgeries
loss · critic · forger
What to watch. The ink curve is the world: everything real lives under it. The vermilion curve is the forger’s output, and at the start it is nowhere near. The gold dashes are the critic’s verdict along the line, high where it believes things are real. As the forger moves under the ink, the gold line flattens towards one half: the critic can no longer tell.
Switch the world to two bumps and reset a few times. Sometimes the forger learns both. Sometimes it paints only one and the critic, satisfied there, never pushes it to the other. That failure has a name, and it is the next section.
the objective
The whole game is a single minimax expression. The critic pushes the value up, the forger pushes it down, and the point where neither can move is the point where the forgeries are perfect.
The critic, D
Pushes the value up. It wants log D(x) large on real things and log(1 − D(G(z))) large on forgeries: confident on both.
The forger, G
Pushes it down, and can only touch the second term. In practice it maximises log D(G(z)) instead, which points the same way but gives a usable gradient when the critic is winning.
For a fixed forger, the best possible critic is
D*(x) = pdata(x)pdata(x) + pg(x)
which is why the gold curve above rides high wherever the ink is and the vermilion is not, and sinks to one half only when the two densities agree everywhere. At that point, and only there, neither player can improve. Goodfellow et al. showed the game has exactly one such point: the forger’s distribution is the world’s.
the trouble
The game is elegant and the training is not. Three ways it goes wrong, all of which you can provoke in the lab above.
The forger finds one thing the critic accepts and makes only that. With two bumps in the world it may paint one, perfectly. Nothing in its reward asks for variety; the critic can only object to what it is shown.
If the critic becomes too good too early, its verdicts saturate and the forger’s gradient goes to zero. It is told it is wrong, but not which way is right. The non-saturating loss softens this; it does not remove it.
There is no single loss to minimise, only a game. The players chase each other, and the curves wobble instead of settling. Later work changed the distance being measured, most famously to the Wasserstein distance, to make the chase better behaved.
the lineage
Six years from a blurry digit to a face you would swear was photographed, and then a different idea took the field.
2014
Two networks, one game. Trained on MNIST and faces, the samples were small and blurry and the idea was enormous.
2015
Convolutional forger and critic, and a set of training rules that made the game stable enough to be practical.
2016
Condition the forger on an input image and it learns translation: edges to photographs, maps to satellite views.
2017
Swap the critic’s objective for an estimate of the Wasserstein distance. Losses that mean something and gradients that do not vanish.
2017
Translation without paired examples, held together by a cycle: horse to zebra and back must return the horse.
2017
Grow both networks from 4×4 to 1024×1024 during training. The first convincingly high-resolution faces.
2018
Scale, mostly. Larger batches and larger models on ImageNet, and a sharp jump in fidelity.
2018
Feed the noise in as style at every layer. Faces that did not exist, and controls that could change a hairstyle without touching a face.
2020
Not a GAN. Diffusion models learn to undo noise one step at a time, and within a year they were beating GANs on the benchmarks GANs had defined.