Memory-Driven Text-to-Image Generation

The memory bank $M$ is a set of image features $v_{i}$ extracted from training set images, and each image features $v_{i}$ is associated with matched text descriptions that are provided in the dataset, e.g., in COCO, each image has five matched text descriptions. These descriptions are used in the matching algorithms, allowing a given text to find the most compatible image features at inference time.

Given a new text description, in order to effectively retrieve the most compatible image features from the memory bank $M$, we have designed several matching algorithms and also explored the effectiveness of each algorithms. A detailed comparison between different algorithms is shown in the supplementary material.

To avoid the identity mapping and also to make full use of image features $v$ in the generator, we first average $v$ on each channel to filter potential content details (e.g., overall spatial structure) contained in $v$, getting a global image feature $v_{G}$, where $v_{G}$ only keeps basic information of the corresponding real image $I$, serving as basic image priors. By doing this, the model can effectively avoid copying and pasting from $I$, and greatly ensure the diversity of output results, especially on the first stage. This is because the following stages focus more on refining basic images produced by the first stage, according to adding more details and improving their resolution, shown in Fig. 3.

However, only feeding the global image feature $v_{G}$ at the beginning of the network, the model may fail to fully utilize the cues contained in the image features $v$. Thus, we further incorporate the image features $v$ at each stage of the network. The reason to feed image features $v$ rather than the global feature $v_{G}$ at the following stages is that $v$ contains more information about the desired output image, such as image contents and geometric structure of objects, where these details can work as candidate information for the main generation pipeline to select. To enable this regional selection effect, we adopt the text-image affine combine module (ACM) (Li et al., 2020), which is able to selectively fuse text-required image information within $v$ into the hidden features $h$, where $h$ is generated from the given text description $S$. However, simply fusing image features $v$ into the generation pipeline may introduce constraints on producing diverse and novel synthetic results, because different image information (e.g., objects and visual attributes) in $v$ may be entangled, which means, for example, if the model only wants to generate one object, the corresponding entangled parts (e.g, objects and attributes) may be produced as well. This may cause an additional generation of text-irrelevant objects and attributes. Thus, to avoid these drawbacks, inspired by the study (Karras et al., 2019), we use several fully connected layers to disentangle the image features $v$, getting disentangled image features $v_{D}$, which allows the model to disconnect relations between different objects and also attributes. By doing this, the model is able to prevent the constraints introduced by the image features $v$, and then selectively choose text-required image information within $v_{D}$, where this information is effectively disentangled without a strong connection.

Why does the generator with image features work better? Ideally, the generator produces a sample, e.g., an image, from a latent code, and the distribution of these samples should be indistinguishable from the training distribution, where the training distribution is actually drawn from the real samples in the training dataset. Based on this, incorporating image features from real images in training dataest into the generator allows the generator to directly draw cues of the desired distribution that it eventually needs to generate. Besides, the global feature $v_{G}$ and disentangled image features $v_{D}$ can provide basic information of target results in advance, and also work as candidate information, allowing the model to selectively choose text-required information without generating it by the model itself, and thus easing the whole generation process. To some extent, the global feature $v_{G}$ can be seen as the meta-data of target images, which may contain information about what kinds of objects to generate, e.g., zebra or bus, and $v_{D}$ is able to provides basic information of objects, e.g., the spatial structure like four legs and one head for the zebra and the rectangle shape for the bus.

To further improve the discriminator to make a more reliable prediction, with respect to both visual appearances and geometric structure, we propose to incorporate the content information into it. This is mainly because, in a deep convolution neural network, when the network goes deeper, the less content details are preserved, including the exact shape of objects (Gatys et al., 2016; Johnson et al., 2016). We think the loss of content details may prevent the discriminator to provide fine-grained shape-quality-feedback to the generator, which may cause the difficulty for the generator to produce realistic geometric structure. Also, Zhou et al. (2014) showed that the empirical receptive field of a deep convolution neural network is much smaller than the theoretical one especially on deep layers. This means, using convolution operators with a local receptive field only, the network may fail to capture the spatial structure of objects when the size of objects exceeds the receptive field.

To incorporate the content details, we propose to generate a series of image content features, $\{a_{128},a_{64},a_{32},\ldots,a_{4}\}$, by aggregating different image regions via applying pooling operators on the given real or fake features. The size of these content features is from $a_{128}\in\mathbb{R}^{C\times 128\times 128}$ to $a_{4}\in\mathbb{R}^{C\times 4\times 4}$, where $C$ represents the number of channels, and the width and the height of the next image content features are $1/2$ the previous one. Thus, the given image is pooled into representations for different regions, from fine- ($a_{128}$) to coarse-scale ($a_{4}$), which is able to preserve content information of different subregions, such as the spatial structure of objects. Then, these features are concatenated with the corresponding hidden features on the channel-wise direction, incorporating the content information into the discriminator.

The number of different-scale content features can be modified, which is dependent on the size of given images. These features aggregate different image subregions by repetitively adopting fixed-size pooling kernels with a small stride. Thus, these content features maintain a reasonable small gap for image information. For the type of pooling operation between max and average, we perform comparison studies to show the difference in Sec. 6.2.

Why does the discriminator with content information work better? Basically, the discriminator in a generative adversarial network is simply a classifier (Goodfellow et al., 2014). It tries to distinguish real data from the data created by the generator (note that in our method, we implement the Minmax loss in the loss function, instead of the Wasserstein loss (Arjovsky et al., 2017)). Also, the implementation of content information has shown its great effectiveness on classification (Lazebnik et al., 2006; He et al., 2015) and semantic segmentation (Liu et al., 2015; Zhao et al., 2017). Based on this, incorporating the content information into the discriminator is helpful, allowing the discriminator to make a more reliable prediction on complex datasets, especially for the datasets with complex image scenery settings, such as COCO.

To train the network, we follow (Li et al., 2020) and adopt adversarial training. There are three stages in the model, and each stage has a generator network and a discriminator network. The generator and discriminator are trained alternatively by minimizing the generator loss $\mathcal{L}_{G}$ and discriminator loss $\mathcal{L}_{D}$. Please see the supplementary material for more details about training objectives. We only highlight some training differences compared with Li et al. (2020).

Generator objective. The objective functions to train the generator are similar as in (Li et al., 2020), but, differently, the inputs for the generator are a pair of (S, v) and a noise $z$, denoted by $G_{i}(z,S,v)$, where $i$ indicates the stage number.

Discriminator objective. To improve the convergence of our GAN-based generation model, the $R_{1}$ regularization (Mescheder et al., 2018) is adopted in the discriminator:

where $\psi$ represents parameter values of the discriminator.

Quantitative comparison. Quantitative results are shown in Tables 1 and 2. As we can see, compared to other approaches, our method achieves better FID and R-precision scores on both datasets, and even has a better performance than OP-GAN, where OP-GAN adopts bounding boxes. This indicates that (1) our method can produce more realistic images from given text descriptions, in terms of image quality and diversity, and (2) synthetic results produced by our method are more semantically aligned with the given text descriptions. Besides, in human evaluation, our method achieves better alignment and realism scores, compared with DF-GAN, which indicates that our results are most preferred by workers, which further verifies the better performance of our method, with respect to semantic alignment and image realism.

Qualitative comparison. In Fig. 6, we present synthetic examples produced by our method at $256\times 256$, along with the corresponding retrieved images that provide image features. As we can see, our method is able to produce high-quality results on CUB and COCO, with respect to realistic appearances and geometric structure, and also semantically matching the given text descriptions. Besides, the synthetic results are different from the retrieved image features, which indicates there is no significant copy-and-paste problem in our method.

Diversity evaluation. To further evaluate the diversity of our method, we fix the given text description and the corresponding retrieved image features, and only change the given noise $z$ to generate output images, shown in Fig. 7. When we fix the sentence and image features and only change the noise, our method can generate obviously different images, but they still semantically match the given sentence and also make use information from the image features. More evaluations are shown in the supplementary material.

Effectiveness of the image features. To better understand the effectiveness of image features in the generator, we conduct an ablation study shown in Table 3. Without image features, the model “Ours w/o Feature” achieves worse quantitative results on both FID and R-precision compared with the baseline, which verifies the effectiveness of image features on high-quality image generation. Interestingly, without image features, even our method becomes a pure text-to-image generation method, similar to other baselines, but the FID of “Ours w/o Feature” is still competitive with other baselines. This indicate that even without the image features fed into our method, our method can still generate better synthetic results, with respect to image quality and diversity. We think this is mainly because with the help of content information, our better discriminator is able to make a more reliable prediction on complex datasets, which in turn encourages the generator to produce better synthetic images.

Effectiveness of the disentanglement. Here, we show the effectiveness of the fully connected layers applied on the image features $v$. Interestingly, from Table 3, the “model w/o Disen.” achieves better FID and R-precision compared with the baseline. This is likely because the model may suffer from an identity mapping problem. To verify this identity mapping problem, we conduct another experiment, where we feed mismatched sentence and image pairs into the network without using search algorithms, denoted “model w/o Disen.*”. As we can see, on mismatched pairs, although FID is still low, the R-precision degrades significantly.

Effectiveness of the content information. To verify the effectiveness of the content information adopted in the discriminator, we conduct an ablation study, shown in Table 3. As we can see, FID and R-precision degrade when the discriminator without adopting the content information. This may indicate that the content information can effectively strengthens the differentiation abilities of the discriminator. Then, the improved discriminator is able to provide the generator with fine-grained training feedback, regarding to geometric structure, thus facilitating training a better generator to produce higher-quality synthetic results.

Comparison between different pooling types. Here, we conduct a comparison study on different pooling types (i.e., max and average) in Table 3. As we can see, the model with the average pooling works better than max pooling. We think that this is likely because max pooling fails to capture the contextual information between neighboring pixels, because it only picks the maximum value among a region of pixels, while average pooling calculates the average value between them.

Effectiveness of the regularization. We evaluate the effectiveness of the adopted regularization in the discriminator. From Table 3, the model without the regularization has worse quantitative results, compared with the full model. We think that this is because the regularization effectively improves GAN convergence by preventing the generator from training on junk feedback, once the discriminator cannot easily tell the difference between real and fake.