1 Original GAN

With data samples $ẋ$ that follow a certain unknown distribution $p\left(x\right)$, the idea is to have a neural network generator $G_{𝜃}$, parameterized by $𝜃$, which transforms samples $z\sim N(0,I)$ to $\tilde{x}$ that follows $p\left(x\right)$. Figure 1 depicts the architecture.

In order to train $G_{𝜃}$, some optimization objective is needed to give $G_{𝜃}$ feedback about whether $\tilde{x}$ really looks like it is drawn from $p\left(x\right)$. A binary descriminator/classifier $D_{\varphi }$ that returns the probability that $\tilde{x}$ follows $p\left(x\right)$ would serve the purpose. Figure 2 shows the architecture we have so far.

The introduction of $D_{\varphi }$ merely defers the responsibility of guiding $G_{𝜃}$. How do we train $D_{\varphi }$? We could feed it both real samples $(x,y)=(ẋ,1)$ and artificial samples $(x,y)=(\tilde{x},0)$ created by $G_{𝜃}$. Figure 3 shows the complete GAN architecture.

The loss function $ℒ$ of $D_{\varphi }$ is the common binary cross entropy function, shown below (see amaires.github.io/OptimizationObjective/ for a refresher):

Remove the negation in (1) and rewrite it in conditional expectation form:

If the same number of real samples $ẋ$ and artificial samples $\tilde{x}$ are fed to $D_{\varphi }$ in each batch/mini-batch, $Pr(y=1)=Pr(y=0)=\frac{1}{2}$. Also note that $x\sim p\left(x\right|y=1)$ is the same as $ẋ\sim p\left(ẋ\right)$ and $x\sim p\left(x\right|y=0)$ is the same as $\tilde{x}\sim p_{𝜃}\left(\tilde{x}\right|z)$, $ℒ$ can be further written:

Note that the objective is parameterized by both $𝜃$ and $\varphi$, but it is maximized only in terms of $\varphi$, the paramters for the discriminator. It is also worth noting that $D_{\varphi }$ gives the probability if a sample follows $p\left(x\right)$, but not $p\left(x\right)$ itself. Also, $G_{𝜃}$ generates a sample $\tilde{x}$, not $p_{𝜃}\left(\tilde{x}\right)$.

The process so far optimizes $\varphi$ to make $D_{\varphi }$ better at telling real samples apart from samples created by a fixed $G_{𝜃}$. For each batc/mini-batch of samples, $\varphi$ takes a gradient ascent step. This process does not seem to improve $G_{𝜃}$ whatsoever. If $𝜃$ also takes a gradient ascent step using ${\nabla <!--FUNCTION\; APPLICATION-->}_{𝜃}ℒ$, it also makes $D_{\varphi }$ better. In this case, it is probably equivalent to making samples created by $G_{𝜃}$ more obviously fake, the opposite of what we want. The solution to this last piece of the GAN puzzle is to update $𝜃$ with a gradient descent step. More formally, our objective function is

Intuitively, $D_{𝜃}$ tries to tell real samples apart from artificial samples, and $G_{𝜃}$ tries to create artificial samples that are hard to distinguish from real samples, hence the name generative adversarial network.

In GAN, $p\left(x\right)$ and $p_{𝜃}$ are never explicitly modeled. $G_{𝜃}$ can produce good samples following $p_{𝜃}\left(x\right)$, but it does not know the form of $p_{𝜃}\left(x\right)$. In other words, $G_{𝜃}$ defines a sampling process without knowing its distribution.

1.1 Jenson-Shannon Divergence

For a given $G_{𝜃}$, $D_{\varphi }$’s objective is to maximize

Let $l_{𝜃,\varphi }\left(x\right)$ be the function under the integral in (2). Assuming $D_{\varphi }$ is flexible and powerful enough, when $L_{𝜃,\varphi }$ is maximized with parameter ${\varphi }^{\ast }$, $l_{𝜃,{\varphi }^{\ast }}\left(x\right)$ is also maximized everywhere. Take $l_{𝜃,\varphi }\left(x\right)$’s derivative against $D_{\varphi }\left(x\right)$, we have

Set the expression to 0, you derive $D_{{\varphi }^{\ast }}\left(x\right)$:

Substitute $D_{\varphi }\left(x\right)$ with (3) in (2), we have:

The first two terms is actually twice the Jenson-Shannon Divergence (JSD) defined as:

Using JSD, $L_{𝜃}$ is further simplified as:

Now it becomes clear that the generator $G_{𝜃}$ is really optimizing the J-S divergence between $p_{𝜃}\left(x\right)$ and $p\left(x\right)$ given the optimal $D_{\varphi \ast }$.

The Jenson-Shannon Divergence has a few properties as well.

• $JSD(p,q)\ge 0$
• $JSD(p,q)\ge 0$ iff $p=q$
• Unlike the K-L divergence, $JSD(p,q)$ is symmetric. That is $JSD(p,q)=JSD(q,p)$.

1.2 Contrast Maximization

When $D_{\varphi }$ is trained reasonably well, which means $D_{\varphi }\left(x\right)$ is close to 1 for real samples, and is close to 0 for artifical samples, $L_{𝜃,\varphi }$can be shown to maximize the contrast of $D_{\varphi }\left(x\right)$ between real and artifical samples, as shown below.

(4) uses the first derivative of $\log <!--FUNCTION\; APPLICATION-->(\cdot )$ at $\log <!--FUNCTION\; APPLICATION-->\left(1\right)$ for approximation. Of course, when $D_{𝜃}$ is not a very good discriminator yet, the above does not hold.

2 f-GAN

too much math and too little practical impact to write about... will pick up later

3 WGAN

3.1 Problems with GAN

GAN is known to generate very impressive photorealistic images, but it also has a few well documented drawbacks.

The first is a problem known as mode collapse. The discriminator $D_{\varphi }$ only cares about distinguishing real samples from artificial samples. It does not care about whether those artificial samples have broad coverage or not. For example, suppose the real samples include images of different animals such as cats, dogs, and horses. $G_{𝜃}$ is happy to generate pictures of only dogs as long as these pictures become more real each time $𝜃$ is updated. $D_{\varphi }$ is also perfectly happy with $G_{𝜃}$’s behavior as long as each time $\varphi$ is updated, $D_{\varphi }$ can tell real animal pictures from these artifical dog picutres a little better.

The second notorious problem with GAN is its difficulty to train. Unlike other deep learning problems which see their loss function decreases gradually until converging to 0 in training, GAN’s minimax objective does not offer any such guarantee. In practice, GAN’s objective keeps oscillating during training. Deciding when to stop training is often a manual process. Another reason that prevents GAN’s effective training has to do with $D_{\varphi }$’s final activation function, typically sigmoid. When real samples and artificial samples are far apart, it is easy for $D_{\varphi }$ to distinguish them. In this case, sigmoid’s outputs are very close to 1 for real samples, and 0 for artificial samples. Its derivative is very close to 0, a problem known as vanishing gradient. These 0 gradients cannot provide effective back-propagation for $G_{𝜃}$ to improve its sample generating process.

3.2 Intuition of WGAN

The inspiration of Wasserstein GAN (WGAN) comes from contrast maximization described in Section 1.2 and sigmoid’s vanishing gradient problem described in Section 3.1. WGAN’s objective is

Basically, $D_{\varphi }$ tries to maximize its output for real samples and minimize its output for artificial samples. $G_{𝜃}$ on the other hand tries to generate artifical samples that also get large outputs.

This objective looks just like the contrast maximization loss in Section (1.2), but there are two main differences: it is no longer an approximation in a narrow range and sigmoid is not used to compress $D_{\varphi }$’s output to between 0 and 1. Completely removing $D_{\varphi }$’s output value constraint may, however, pose another problem; $L_{𝜃,\varphi }$ may grow very rapidly out of bound and may still take the form of a sigmoid function and stifle back-propagation. Ideally, we’d like $D_{\varphi }$ to behave roughly like a linear function of $x$. Given deep neural networks are differentiable for almost all input, we could force the norm of $D_{\varphi }$’s gradient to be close to 1 everywhere:

This constraint can be added to $L_{𝜃,\varphi }$ as a penalty term:

This penalty term needs to be numerically computable. Averaging over all possible value of $x$ is out of the question, but one possbility is to average it over both the real samples and the artificial samples, as shown below:

where $\lambda$ is the knob adjusting the relative importance between maximizing contrast and making $D_{\varphi }$ roughly a linear function.

A variant of the above formulation creates samples by randomly and linearly interpolating between real and artificial samples, and averages the penalty term over these samples instead. The final objective function, expressed in numerical computation form, is the following:

where ${𝜀}_i\sim U(0,1)$. GAN with this gradient norm penalty is called Wasserstein Generative Adversarial Network - Gradient Penalty (WGAN-GP).

Both $D_{\varphi }$ and $G_{𝜃}$ affects the penalty term. Though it is not mentioned in the original WGAN-GP paper, I don’t think it is desirable to add the penalty term to $G_{𝜃}$’s loss function. WGAN-GP’s objective should instead be

There are a couple more things worth noting.

• ${\nabla <!--FUNCTION\; APPLICATION-->}_x{D}_{\varphi }\left(x\right)$ is the derivative against $x$, not against $\varphi$. What is used in back-propagation to update $\varphi$ includes ${(\parallel {\nabla <!--FUNCTION\; APPLICATION-->}_x{D}_{\varphi }\left(x\right)\parallel -1)}^2$’s derivative against $\varphi$.
• Obviously, with WGAN, $G_{𝜃}$ no longer minimizes the Jenson-Shannon divergence even given an optimal $D_{{\varphi }^{\ast }}$.

3.3 Math of WGAN

3.3.1 Infimum and supremum

Most people are familiar with the concept of maximum and minimum. Explicitly, if $X$ is a (partially) ordered set and $S$ a subset, then $\overline{s}$ is the maximum of $S$ iff $\overline{s}\in S$ and $s\le \overline{s}$ for all $s\in S$. Similarly, $\underline{s}$ is the minimum of $S$ iff $\underline{s}\in S$ and $s\ge \underline{s}$ for all $s\in S$.

The supremum (sup) of $S$ can be defined like this. Let $T=\{t\in X|s\le t\forall <!--FUNCTION\; APPLICATION-->s\in S\}$, which defines the set of elements greater than all members of $S$. If $T$ is empty, $S$’s supremum does not exist, otherwise it is the mininum of $T$. If S has a maximum, it must be the same as $S$’s supremum. Even if $S$ does not have a maximum, it may still have a supremum. Below are three examples comparing maximum and supremum.

1. $S=\left\{x\right|x\le 2\}$: $S$’s maximum is 2, and its supremum is 2 as well.
2. $S=\left\{x\right|x<2\}$: $S$ does not have a maximum, but its supremum is 2.
3. $S=\left\{x\right|x>2\}$: $S$ has neither a maximum nor a supremum.

Similar comparisons can be made between minimum and infimum. Informally, if one uses supremum and maximum interexchangeably, little is lost. The same goes for infimum and minimum.

3.3.2 Wasserstein Distance

K-L divergence and J-S divergence are often used to measure the closeness between two distributions. In fact, VAE (amaires.github.io/VAE) uses K-L divergence for optimization and GAN’s $G_{𝜃}$ minimizes the J-S divergence given an optimal $D_{{\varphi }^{\ast }}$. Unfortunately, these two measures have discontinuity when two distributions have disjoint supports (the support of a function is the subset of the function domain not mapped to 0). For example, given two distributions defined below:

It is not hard to figure out their K-L and J-S divergence:

Ideally, we’d like a measure that is smoother. Wasserstein distance is exactly such a function defined as:

where $\prod <!--FUNCTION\; APPLICATION-->(p_1,p_2)$ contains all joint distribution of $(x,y)$ such that $p_1\left(x\right)=\mathrm{\int \; <!--nolimits-->}<!--FUNCTION\; APPLICATION-->\gamma (x,y)dy$ and $p_2\left(y\right)=\mathrm{\int \; <!--nolimits-->}<!--FUNCTION\; APPLICATION-->\gamma (x,y)dx$. Wasserstein distance is also called earth-mover distance. It informally captures the minimal amount of mass/dirt needs to be moved to turn the shape of $p_1$ into that of $p_2$. Using this definition, the Wasserstain distance between $p\left(x\right)$ and $p_{𝜃}\left(x\right)$ above can be calculated to be $\left|𝜃\right|$, which is a continuous function of $𝜃$.

In general, however, Wasserstein distance is intractable to calculate. Fortunately, Wasserstein distance has another definition, based on the Kantorovich-Rubinstein duality, which is easier to handle:

where $f(\cdot )$ is any real valued function. ${\parallel f\parallel }_L\le 1$ means $f$’s Lipschitz constant is 1. Technically, it means

Intuitively, it is equivalent to saying $f\left(x\right)$’s value should not change too much as $x$ changes.

Wasserstein distance (5) really captures the contrast maximization idea in Section 1.2 well. The Lipschitz continuity constraint can be approximated by the gradient penalty term introduced in 3.2.

3.3.3 GAN and WGAN

GAN’s $D_{\varphi }$ optimizes maximum likelihood of observing samples $ẋ$ and $\tilde{x}$. Given an optional $D_{{\varphi }^{\ast }}$, $G_{𝜃}$ minimizes the J-S divergence between $p_{𝜃}\left(x\right)$ and $p\left(x\right)$.

In WGAN, $D_{\varphi }$ maximizes the Wasserstein distance between $ẋ$ and $\tilde{x}$ while $G_{𝜃}$ tries to reduce it. Wasserstein distance is a smoother and more effective measure of closeness between two distributions, resulting in more stable training and less mode collapse in WGAN than GAN.

4 Latent Representation and BiGAN

The training of Variational Autoencoders produces both a generator and a encoder. The latter is capable of extracting features or latent representations of data. GAN only has a generator $G_{𝜃}$. It is conceivable that the pre-final layers of the discriminator $D_{\varphi }$ may be used for feature representations. The intuition is that $D_{\varphi }$ would have learned useful high-level representations of both real and artificial samples.

Bidirectional Generative Adversarial Networks (BiGAN), depicted in Figure 4, takes a much more direct approach to latent representation. It introduces an encoder $E_{\gamma }$ that maps real samples $ẋ$ to $\tilde{z}$ in the latent space. $D_{\varphi }$ works on the joint distribution of $(x,z)$ and tries to maximize the contrast between real samples $(ẋ,\tilde{z})$ and artificial samples $(\tilde{x},z)$. The loss function introduced in WGAN-GP can be used here. After training is done, new samples can be generated by $G_{𝜃}$, and latent representations can be inferred via $E_{\gamma }$.

5 Image-to-image translation and CycleGAN

A GAN’s generator maps samples drawn from $N(0,I)$ to meaningful images. Can GAN map images in one domain to ones in another domain, for example from summer pictures to winter pictures of the same place, from horse pictures to zebra pictures, and from photos to Van Gogh’s paintings?

It is not hard to design a GAN for that purpose. For example, suppose our goal is to add black/white stripes to a horse and make it look like a zebra, we could replace $z\sim N(0,I)$ with a bunch of horse pictures, and the real samples will be drawn from zebra pictures. Unfortunately, this architecture does not ensure that the generated zebra picture will be much like the input horse picture. CycleGAN introduced two innovations to address this problem:

1. Create two GANs. The first GAN translates horse pictures to zebra pictures, and the second GAN translates zebra pictures to horse pictures.
2. Each generated zebra picture is then fed to the second GAN to translate to a horse picture. This generated horse picture should look similar to the original horse picture. A similar process happens with the reverse translation direction.

For simplicity, we’ll use the following notation.

CycleGAN’s optimization objective is