Pros and Cons of GAN Evaluation Measures:
New Developments

Evaluation of generative models is an active area of research. Several quantitative and qualitative measures have been proposed so far. In this section, I review some of the influential measures that have inspired the researchers.

A classic approach is to compare the log-likelihoods of models. This approach, however, has several shortcomings. A model can achieve high likelihood, but low image quality, and conversely, low likelihood and high image quality. Also, kernel density estimation in high-dimensional spaces is very challenging (Theis et al., 2015).

The two most common GAN evaluation measures are Inception Score (IS) and Fréchet Inception Distance (FID). They rely on a pre-existing classifier (InceptionNet) trained on ImageNet. IS (Salimans et al., 2016) computes the KL divergence between the conditional class distribution and the marginal class distribution over the generated data. FID (Heusel et al., 2017) calculates the Wasserstein-2 (a.k.a Fréchet) distance between multivariate Gaussians fitted to the embedding space of the Inception-v3 network of generated and real images³³3To learn how to implement FID, please refer to here..

IS does not capture intra-class diversity, is insensitive to the prior distribution over labels (hence is biased towards ImageNet dataset and Inception model⁴⁴4See also here.), and is very sensitive to model parameters and implementations (Barratt and Sharma, 2018; Odena, 2019). It requires a large sample size to be reliable. A simple class-conditional model that memorizes one example per ImageNet class achieves a high IS. Using a 1D example, Barratt and Sharma (2018) show that the true underlying data distribution may achieve a lower IS than other distributions.

FID has been widely adopted because of its consistency with human inspection and sensitivity to small changes in the real distribution (e.g. slight blurring or small artifacts in synthesized images). FID, unlike IS, can detect intra-class mode collapse⁵⁵5Mode collapse happens when a generator produces data from only a few modes of the target distribution, thus failing to generate diverse enough outputs. Please see here.. The Gaussian assumption in FID calculation, however, might not hold in practice. A major drawback with FID is its high bias. The sample size to calculate FID has to be large enough (usually above 50K). Smaller sample sizes can lead to over-estimation of the actual FID (Chong and Forsyth, 2020).

While single-value metrics like IS and FID successfully capture many aspects of the generator, they lump quality and diversity evaluation and are therefore not ideal for diagnostic purposes (Precision-Recall addresses this shortcoming). Further, they continue to have a blind spot for image quality (Karras et al., 2019).

Another popular approach to evaluate GANs is manual inspection (e.g.  Denton et al. (2015); Zhou et al. (2019)). While this approach can directly tell us about the image quality, it is limited in assessing sample diversity. It is also subjective as the reviewers may incorporate their own biases and opinions. It also requires knowledge of what is realistic and what is not for the target domain which might be hard to learn in some domains (e.g. medical domain⁶⁶6For example, to tell whether a GAN can generate X-rays with pneumonia in them, it is better to ask doctors rather than Amazon Mechanical Turk workers!). Further, it is constrained by the number of images that can be reviewed in a reasonable time prohibiting its usage during model development. Finally, due to the subtle differences in experimental protocols (e.g. user interface design, fees, duration), it is often difficult to replicate the results across different publications.

This paper is organized as follows. Recently proposed evaluation measures are categorized under quantitative and qualitative measures and are covered in sections 2 and 3, respectively⁷⁷7Some methods may fall in both categories.. In section 4, I then discuss some benchmark and model comparison studies as well as analysis works. Finally, section 5 highlights some major points and offers suggestions for future research in GAN evaluation. Note that some of the works reviewed here have not been accepted in prior published work and may have not been peer reviewed. Therefore, they should be treated with additional caution in the absence of widespread usage.

FID scores computed with classification-pretrained embeddings have been shown to correlate well with human evaluations (Heusel et al., 2017). Meanwhile, they can be misleading as they are biased towards ImageNet. On non-ImageNet datasets, FID can result in inadequate evaluation. Morozov et al. (2020) advocate for using self-supervised representations to evaluate GANs on the established non-ImageNet benchmarks¹¹¹¹11https://github.com/stanis-morozov/self-supervised-gan-eval. They show that representations, typically obtained via contrastive or clustering-based approaches, provide better transfer to new tasks and domains, and thus can serve as more universal embeddings of natural images. Further, they demonstrate that self-supervised representations produce a more reasonable ranking of models and often improve the sample efficiency of FID.

These methods measure disentanglement in the representations learned by generative models, and are useful for improving the generalization, robustness, and interpretability of models.

Ravuri and Vinyals (2019) propose a measure that is in essence similar to the “Classification Performance” discussed in Borji (2019). They argue that if a generative model is learning the data distribution in a perceptually meaningful space then it should perform well in downstream tasks. They use class-conditional generative models from a number of generative models such as GANs and VAEs to infer the class labels of real data. They then train an image classifier using only synthetic data and use it to predict labels of real images in the test set. Their findings are as follows: a) when using a state-of-the-art GAN (BigGAN-deep by Brock et al. (2018)), Top-1 and Top-5 accuracies decrease by 27.9% and 41.6%, respectively, compared to the original data, b) CAS automatically identifies particular classes for which generative models fail to capture the data distribution (Fig. 6), and c) IS and FID are neither predictive of CAS, nor useful when evaluating non-GAN models.

Meehan et al. (2020) formalize a notion of overfitting called data-copying where a generative model memorizes the training samples or their small variations (Fig. 7). They provide a three sample test for detecting data-copying that uses the training set, a separate held-out sample from the target distribution, and a generated sample from the model. The key insight is that an overfitted GAN generates samples that are on average closer to the training samples, therefore the average distance of generated samples to training samples is smaller than the corresponding distances between held-out test samples and the training samples. They also divide the instance space into cells and conduct their test separately in each cell. This is because, as they argue, generative models tend to behave differently in different regions of space.

Zhao et al. (2018)¹⁶¹⁶16Please see also the subsection on “Evaluating Mode Drop and Mode Collapse” in Borji (2019). utilize carefully designed training datasets to characterize how existing models generate novel attributes and their combinations. Some of their findings are as follows. When presented with a training set with all images having exactly 3 objects, both GANs and VAEs typically generate 2-5 objects (Fig. 8.B). Over a multi-modal training distribution (e.g. images with either 2 or 10 objects), the model acts as if it is trained separately on each mode, and it averages the two modes. When the modes are close to each other (e.g. 2 and 4 objects), the learned distribution assigns higher probability to the mean number of objects (3 in this example), even though there was no image with 3 objects in the training set (Fig. 8.C). When the training set contains certain combinations (e.g. red cubes but not yellow cubes; Fig. 8.D), the model memorizes the combinations in the training set when it contains a small number of them (e.g. 20), and generates novel combinations when there is more variety (e.g. 80). Xuan et al. (2019) made a similar observation shown in Fig. 9. On a geometric-object training dataset where the number of objects for each training image is fixed, they found a variable number of objects in generated images. Some other studies that have investigated GAN generalization include O’Brien et al. (2018); van Steenkiste et al. (2020).

Sajjadi et al. (2018) propose to use precision and recall to explicitly quantify the trade off between quality (precision) and coverage (recall). Precision measures the similarity of generated instances to the real ones and recall measures the ability of a generator to synthesize all instances found in the training set (Fig. 10). P&R curves can distinguish mode-collapse (poor recall) and bad quality (poor precision). Some new ideas based on P&R are summarized below.

Grnarova et al. (2019) leverage the notion of duality gap from game theory to propose a domain-agnostic and computationally-efficient measure that can be used to assess different models as well as monitoring the progress of a single model throughout training. Intuitively, duality gap measures the sub-optimality (w.r.t. an equilibrium) of a given solution (G,D) where G and D and the generator and the discriminator, respectively. They also show that their measure highly correlates with FID on natural image datasets, and can also be used in other modalities such as text and sound. Further, their measure requires no labels or a pretrained classifier, making it domain agnostic. In a follow-up work, Sidheekh et al. (2021) extend the notion of duality gap to proximal duality gap that is applicable to the general context of training GANs.

A number of studies (e.g.  Durall et al. (2020); Frank et al. (2020); Dzanic et al. (2019); Zeng et al. (2017)) have shown that current generators are unable to correctly approximate the spectral distributions of real data. Durall et al. (2020) showed that up-scaling operations commonly used in GANs (e.g. up-convolutions) alter the spectral properties of the images causing high frequency distortions in the output. This can be seen in the average azimuthal integration of the power spectrum shown in the top panel of Fig. 13. Based on this observation, they proposed a novel spectral regularization term to compensate spectral distortions. They also proposed a very simple but highly accurate detector for generated images and videos, i.e. a DeepFake detector.

Almost all existing metrics are solely for evaluating models that generative image. They do not take into account the corresponding text in the context of text-to-image generation. Thus, they may be fooled by a network that ignores the textual input and only focuses on generating realistic looking images. To remedy this shortcoming, Ding et al. (2021) introduce the caption score to evaluate the correspondence between images and text. This score measures the quality and accuracy for text-image generation at a finer granularity than FID and Inception Score (IS), and is defined as:

where $t$ is a sequence of text tokens and $x$ is the image. The higher the CapS, the better. In addition to CapS, they also report results using other measures such as FID and IS.

A commonly used metric to evaluate generative models of text is the perplexity. It measures the probability for a sentence to be produced by a language model that has been trained on a dataset:

$\text{perplexity }=\prod_{t=1}^{T}\left(\frac{1}{P_{\mathrm{LM}}\left(\bm{x}^{(t+1)}\mid\bm{x}^{(t)},\ldots,\bm{x}^{(1)}\right)}\right)^{1/T}$

Zhou et al. (2019) propose a human-in-the-loop evaluation approach that is grounded in psychophysics research on visual perception (Fig. 14). They introduce two variants. The first one measures visual perception under adaptive time constraints to determine the threshold at which a model’s outputs appear real (e.g. 250 ms). The second variant is less expensive and measures human error rate on fake and real images without time constraints¹⁸¹⁸18An important point when conducting experiments of this sort (e.g.  Denton et al. (2015)) is to make sure that real and generated images are shuffled during the presentation. Otherwise, subjects maybe able to guess what type of images are shown in a session from the first few images, if all the images in the session in either real or fake.. Kolchinski et al. (2019) propose an approach to approximate HYPE and report 66% accuracy in predicting human scores of image realism, matching the human inter-rater agreement rate. A major drawback with human evaluation approaches such as HYPE is scaling them. One way to remedy this is to train a model from human judgments and interact with a person only when the model is not certain. HYPE is more reliable compared to automated ones but cannot be used for monitoring the training process.

Wang et al. (2020b) outline a method called Neuroscore using neural signals and rapid serial visual presentation (RSVP) to directly measure human perceptual response to generated stimuli. Participants are instructed to attend to target images (real and generated) amongst a larger set of non-target images (left panel in Fig. 15). This paradigm, known as the oddball paradigm, is commonly used to elicit the P300 event-related potential (ERP), which is a positive voltage deflection typically occurring between 300 ms and 600 ms after the appearance of a rare visual target. Fig. 15 shows a depiction of this approach. Wang et al. show that Neuroscore is more consistent with human judgments compared to the conventional metrics. They also trained a convolutional neural network to predict Neuroscore from GAN-generated images directly without the need for neural responses.

Bau et al. (2019) visualize mode collapse at both distribution level and instance level. They employ a semantic segmentation network to compare the distribution of segmented objects in the generated images versus the real images. Differences in statistics reveal object classes that are omitted by a GAN. Their approach allows to visualize the GAN’s omissions for an omitted class and to compare differences between individual photos and their approximate inversions by a GAN. Fig. 16 illustrates this approach.

Jahanian et al. (2019) propose a method to quantify the degree to which basic visual transformations are achieved by navigating the latent space of a GAN (See Fig. 17). They first learn a $N$-dimensional vector representing the optimal path in the latent space for a given transformation. Formally, the task it to learn the walk $w$ by minimizing the following objective function:

where $\alpha$ is the step size, and $\mathcal{L}$ is the distance between the generated image $G(z+\alpha w)$ after taking an $\alpha$-step in the latent direction and the target image edit($G(z)$, $\alpha$). The latter is the new image derived from the source image $G(z)$. To quantify how well a desired image manipulation under each transformation is achieved, the distributions of a given attribute (e.g. “luminance”) in the real data and generated images (after walking in latent space) are compared.

Bau et al. (2018) propose a technique to dissect and visualize the inner workings of an image generator¹⁹¹⁹19A demo of this work is available at here.. The main idea is to identify GAN units (i.e. generator neurons) that are responsible for semantic concepts and objects (such as tree, sky, and clouds) in the generated images. Having this level of granularity into the neurons allows editing existing images (e.g. to add or remove trees as shown in Fig. 18) by forcefully activating and deactivating (ablating) the corresponding units for the desired objects. Their technique also allows finding artifacts in the generated images and hence can be used to evaluate and improve GANs. A similar approach has been proposed in Park et al. (2019) for semantic manipulation and editing of GAN generated images²⁰²⁰20See here for an illustration..

Wang et al. (2020a) ask whether it is possible to create a “universal” detector to distinguish real images from synthetic ones using a dataset of synthetic images generated by 11 CNN-based generative models. See Fig. 19. With careful pre- and post-processing and data augmentation, they show that an image classifier trained on only one specific CNN generator is able to generalize well to unseen architectures, datasets, and training methods. They highlight that today’s CNN-generated images share common systematic flaws, preventing them from achieving realistic image generation. Similar works have been reported in Chai et al. (2020); Yu et al. (2019). In alignment with these results, Gragnaniello et al. (2021) also conclude that we are still far from having reliable tools for GAN image detection.

Following previous works (e.g.  Lucic et al. (2017); Shmelkov et al. (2018)), new studies have investigated formulating good criteria for GAN evaluation, or have conducted systematic GAN benchmarks. Gulrajani et al. (2020) argue that a good evaluation measure should not have a trivial solution (e.g. memorizing the dataset) and show that many scores such as IS and FID can be won by simply memorizing the training data (Fig. 20). They suggest that a necessary condition for a metric not to behave this way is to have a large number of samples. They also propose a measure based on neural network divergences (NND). NND works by training a discriminative model to discriminate between samples of the generative model and samples from a held-out test set. The poorer the discriminative model performs, the better the generative model is. Through experimental validation, they show that NND can effectively measure diversity, sample quality, and generalization. Lee and Town (2020) introduce Mimicry²¹²¹21https://github.com/kwotsin/mimicry, a lightweight PyTorch library that provides implementations of popular GANs and evaluation metrics to closely reproduce reported scores in the literature. They also compare several GANs on seven widely-used datasets by training them under the same conditions, and evaluating them using three popular GAN metrics. Some tutorials and interaction tools have also been developed to understand, visualize, and evaluate GANs²²²²22See this link and https://poloclub.github.io/ganlab/..

Also mention that deepfakes might be detected because they have different spectral properties than real images. See for example Durall et al. (2020)

Fairness and bias in ML algorithms, datasets, and commercial products have become growing concerns recently (e.g.  Buolamwini and Gebru (2018)) and have attracted widespread attention from public and media²³²³23https://thegradient.pub/pulse-lessons/. Even without a single precise definition for fairness (Verma and Rubin, 2018), it is still possible to observe the lack of fairness across many domains and models, with GANs being no exception. Some recent works (e.g.  Xu et al. (2018); Yu et al. (2020) have tried to mitigate the bias in GANs or use GANs to reduce bias in other ML algorithms (e.g.  Sattigeri et al. (2019); McDuff et al. (2019)). Bias can enter a model during training (through data, labeling, architecture), evaluation (such as who created the evaluation measure), as well as deployment (how is the model being distributed and for what purposes). Therefore it is critical to address these aspects when evaluating and comparing generative models.

A alarming application of generative models is fabricating fake content²⁴²⁴24Some concerns can be found at these links 1, 2, 3, and 4. It is thus crucial to develop tools and techniques to detect and limit their use (See Tolosana et al. (2020) for a survey on deepfakes). There is a natural connection between deepfake detection and GAN evaluation. Obviously, as generative models improve, it becomes increasingly harder for a human to distinguish between what is real and what is fake (manipulated images, videos, text, and audio). Telling the degree of fakeness of an image or video directly tells us about the performance of a generator. Even over faces, where generative models excel, it is still possible to detect fake images (Fig. 21), although in some cases it can be very daunting (Fig. 22). Difficulty of deepfake detection for humans is also category dependent, with some categories such as faces, cats, and dogs being easier than bedrooms or cluttered scenes (Fig. 23). Fortunately, or unfortunately, it is still possible to build deep networks that can detect the subtle artifacts in the doctored images (e.g. using the universal detectors mentioned above (Wang et al., 2020a; Chai et al., 2020)). Moving forward, it is important to study whether and how GAN evaluation measures can help us mitigate the threat from deepfakes.