DR-GAN: Distribution Regularization for Text-to-Image Generation

With the recent successes of GANs [7], a large number of GAN-based T2I methods [37, 55, 11, 50, 25, 36, 3, 39, 9, 31, 42] have boosted the performance of the T2I task. Reed et al. [37] first introduced the adversarial process to generate images from text descriptions. However, they can only generate images with $64\times 64$ resolution. And, the quality of the images is not good. Then, StackGAN/StackGAN++ [55, 11] and HDGAN [56] adopted multi-stage generation patterns to progressively enhance the quality and detail of synthesized images. Thereafter, the multi-stage generation framework has been widely used in GAN-based T2I methods. Based on the multi-stage generation framework, AttnGAN [50], DMGAN [31], and CPGAN [18] adopted the word-level or the object-level attention mechanisms to help the generator enhance local regions’ or objects’ semantics. MirrorGAN [36] combined the Text-to-Image generation and Image-to-Text generation to improve the global semantic consistency between text description and the generated image. SDGAN [9] and SEGAN [39] combined the Siamese Network and the contrastive loss to enhance the semantics of the synthesized image. Like these T2I methods, we also adopt the multi-stage generation framework to build our DR-GAN. But different from them, we help the generator to better distill key information for distribution learning.

The GANs [7], as a latent distribution learning strategy, has been widely adopted in various generation tasks [37, 1, 22, 35, 51, 53]. However, GANs tend to suffer from unstable training, mode collapse, and uncontrollable problem, etc., and they hinder GANs from effectively modeling the real data distribution. Recently, many approaches [33, 11, 4, 29, 32, 55] have been exploited to overcome these issues. LSGANs [29] overcome the vanishing gradient problem by introducing a least square loss to replace the entropy loss. WGAN [30] introduced the Earth Mover (EM) distance to improve the stability of learning distribution and provide meaningful learning curves useful for hyperparameter search and debugging. MDGAN [4] improved the stability of distribution learning by utilizing an encoder $E(x):x\rightarrow z$ to produce the latent variable $z$ for the generator $G$. SN-GANs [32] proposed a novel weight normalization method named spectral normalization to better stabilize the training of the discriminator. F-GAN [33] adopted the Kullback-Leibler (KL) divergence to help align the generated image distribution with the real data distribution. In the T2I task, many GAN-based methods introduce various strategies such as multi-stage generation pattern [11, 55], attention mechanism [50, 31], and cycle consistent mechanism [36] to help match synthesized distribution with the real image distribution. However, diverse visual information, messy background, and other non-key visual information in images usually make the image distribution complicated. It makes distribution learning more difficult. Thus, our idea is to explore an effective distribution normalization strategy to overcome the challenge.

Most recent GAN-based T2I methods [50, 43, 25, 36, 39, 31, 9] adopted a multi-stage generation framework to progressively map the text embedding distribution to the image distribution, to synthesize high-quality images. Like all these methods, we also adopt such a generation pattern from AttnGAN [50] as our baseline to build the DR-GAN.

As shown in Fig. 1, DR-GAN has a Text Encoder [50], a conditioning augmentation module [55] $F^{ca}$, $m$ generation modules $G_{i}^{o}$, ($\emph{i}=0,1,2,\ldots,m-1$), and two new designs: $m$ Semantic Disentangling Modules (SDMs) $SDM_{i}$, and $m$ Distribution Normalization Modules (DNMs) $DNM_{i}$, $\emph{i}=0,1,2,\ldots,m-1$.

The Text Encoder transforms the input text description (a single sentence) into the sentence feature $s^{\prime}$ and word features $W$. The $F^{ca}$ [55] converts a sentence feature $s^{\prime}$ to a conditioning sentence feature $s$. The SDM distills key information from the text or image features, in the intermediate stage of generation, for better approximating the real image distribution. In the testing stage, these SDMs take noise $z\sim N(0,1)$, the sentence feature $s$, and the word features $W$ to produce a series of hidden features $H_{i}$ ($\emph{i}=0,1,2,\cdots,m-1$); while in the training stage, besides $z$, $s$ and $W$, the $i$-th $SDM_{i}$ also takes the $i$-th scale real image, $I^{*}_{i}$, as input to generate $H_{i}$. Then, $G_{i}^{o}$ takes $H_{i}$ to generate $i$-th scaled images $\hat{I}_{i}$. The $DNM_{i}$ contains a Variational Auto-Encoder (VAE) module and a discriminator: the former normalizes image latent distributions, and the latter distinguishes between the real image and the synthesized image. The generation stage information flow is formulated as

Note that the $SDM_{0}$ only contains a series of convolution layers and upsampling modules. The proposed Semantic Disentangling Module (SDM) is adopted in the $SDM_{i}$ $i=0,1,2,\cdots,m-1$.

Semantic abstract and ambiguity of text description often makes the generator difficult capture accurate semantics of the given sentence. Such inaccurate semantics are not conducive to learning distribution, and then lead to incorrect spatial structures and semantics in synthesized images. To this end, we propose a Semantic Disentangling Module (SDM) (Fig. 2) to help generators suppress irrelevant spatial information and highlight relevant spatial information for generating high-quality images.

We build our SDM based on the widely adopted Cascaded Attentional Generative Model (CAGM) introduced by AttnGAN [50], because the Word-level Attention Mechanism (WAM) in CAGM can effectively enhance semantic details of the generated images.

Initially, the SDM was designed to directly extract the key and non-key information from text features. Because text and image belong to the heterogeneous domain, it is difficult to achieve reasonable semantics matching in the feature space [54]. Compared with the text features, the word-level context feature from WAM contain more structural and semantic information, and semantically better match the image features. Besides, the semantics of word-level context features and image features, are from the input word features $W$ and sentences features $s$. Therefore, SDM is designed to distill key information and filter out non-key information, on the word-level context features and image features of the CAGM.

In this section, we firstly revisit the WAM  [50] to acquire the word-level context features and image features. Secondly, we introduce the Spatial Self-Attention Mechanism (SSAM) to represent the key information and the non-key information. Finally, we introduce the Semantic Disentangling Loss (SDL) to drive the SDM conduct the semantic disentangling.

The aforementioned SDM can help the generator distill key semantic information for image generation. But diverse visual information, messy background, and other non-key visual information in images usually make the image distribution more complicated. On the discriminator’s side: such complicated image distributions make the distinction of real and synthesized images harder; the discriminator may fail to effectively identify the synthesized image.

We know that the data normalization mechanism [16, 2, 45, 49] can reduce the noise and internal covariate shift of data, and further improve manifold learning efficiency in the deep learning community. In the discriminant stage of GAN, image sampling is firstly carried out from real image distribution and synthesized image distribution. Secondly, the discriminator determines whether the current input image is sampled from the real image distribution or synthesized image distribution, i.e. True or False. Due to the diversity of images and the randomness of synthesized images, the distributions of synthesized images and real images are complicated. Such complex image distribution makes it difficult for the discriminator to distinguish whether the current input image is sampled from the real image distribution or generated image distribution. And it is difficult for the generator to align the generated distribution with the real image distribution. Therefore, it is necessary to reduce the complexity of the distribution. Normalization is an effective strategy for denoising and reducing complexity. So, we expect to introduce the normalization of latent distribution.

As a generative model, Variational Auto-Encoder (VAE) [24] can effectively denoise the latent distribution and reduce complexity of the distribution. In VAE, it assumes the latent embedding vector of an image follow a Gaussian distribution $N(\tilde{\mu},\tilde{\sigma})$, and then normalizes the $N(\tilde{\mu},\tilde{\sigma})$ to a standard normal distribution $N(0,1)$. Based on the advantage of image reconstruction in VAE, the normalized embedding vector can preserve key semantic visual information. Thus, we build a VAE module in DNM to normalize the image latent distributions to help the discriminator better distinguish between the “True” image and the “Fake” image.

The structure of the $i$-th DNM is shown in Fig. 4. DNM contains two sub-modules: the discriminator $D_{i}$ and the VAE module $A_{i}$. To simplify the notation, we omit the subscript $i$. $x$ can be a generated image $\hat{I}$ or a real image $I^{*}$. The discriminator is composed of an encoder $E^{D}(\cdot)$ and an logical classifier $\psi(\cdot)$. $E^{D}(\cdot)$ encodes image $x$ into an embedding vector $v$. The embedding $v$ combined with text embedding $s$ is fed to the logical classifier $\psi(\cdot)$, which identifies if $x$ is a real or generated image. As mentioned above, diverse visual information, messy background, and other non-key visual information in images make the distribution of embedding vectors $v$ complicated, and make the identification of $x$ harder. Thus, we adopt a VAE module to normalize and denoise the latent distribution of embedding vectors $v$. In addition to reducing the complexity of the image latent distribution, using a VAE can also push the encoded image feature vector $v$ to record important image semantics (through reconstruction).

Our VAE module $A$ adopts a standard design architecture of Variational Auto-Encoder (VAE) [24]. As shown in Fig. 4, $A$ has a variational encoder (which consists of an encoder $E^{D}(\cdot)$ and a variational sampling module $\varphi(\cdot)$), and a decoder $D^{E}(\cdot)$.

The information flow of the VAE module $A$ is as follows.

(1) Given an image $x$, $x$ is first fed to the encoder $E^{D}(\cdot)$, and $E^{D}(\cdot)$ outputs the image latent embedding $v$.

(2) $\varphi(\cdot)$ infers the mean and variance of $v$ and builds a Gaussian distribution $N(\tilde{\mu}(\varphi(v)),\tilde{\sigma}(\varphi(v)))$. $N(\tilde{\mu}(\varphi(v)),\tilde{\sigma}(\varphi(v)))$ is further normalized to a Normal distribution by the $KL(N(\tilde{\mu}(\varphi(v)),\tilde{\sigma}(\varphi(v))))||N(0,1))$. To make this procedure differentiable, the re-sampling trick [24] is adopted to get $z^{*}=z\cdot\tilde{\sigma}(\varphi(v)))+\tilde{\mu}(\varphi(v))$, $z\sim N(0,1)$.

(3) $z^{*}$ and the text embedding $s$ are concatenated, and then fed to the decoder $D^{E}(\cdot)$ to reconstruct image $x^{*}$. Note that here the decoder takes both $s$ and $z^{*}$ for reconstruction, because the image generation here is conditioned on the text description.

Combining the above modules, at the $i$-th stage of the DR-GAN, the Generative loss $\mathcal{L}_{G_{i}}$ and Discriminative loss $\mathcal{L}_{D_{i}}$ are defined as

where the unconditional loss is trained to generate high-quality images towards the real image distribution to fool the discriminator, and the conditional loss is trained to generate images to better match text descriptions. The $D_{i}(\cdot)=\hat{\psi}(E^{D}_{i}(\cdot))$ is the unconditional discriminator, $D_{i}(\cdot,\cdot)=\psi(E^{D}_{i}(\cdot),s)$ is the conditional discriminator. The $\hat{\psi}(\cdot)$ and $\psi(\cdot)$ are the unconditional logical classifier and the conditional logical classifier respectively.

The discriminator $D_{i}$ is trained to classify the input image into the “Fake” or “True” class by minimizing the cross-entropy loss

where $I^{*}_{i}$ is from the realistic image distribution $P_{data}$ at the $i$-th scale, and $\hat{I_{i}}$ is from distribution $P_{G_{i}}$ of the generative images at the same scale.

To generate realistic images, the final objective functions in the generation training stage ($\mathcal{L}_{G}$) and discrimination training stage ($\mathcal{L}_{D}$) are respectively defined as

The loss function $\mathcal{L}_{DAMSM}$ [50] is designed to measure the matching degree between images and text descriptions. The DAMSM loss makes generated images better conditioned on text descriptions. The DR-GAN has three-stage generators ($m=3$) like the most recent GAN-based T2I methods [31, 3, 42, 36, 25, 50].

Datasets. We conduct experiments on two widely-used datasets, CUB-Bird [47] and MS-COCO [27] datasets. The CUB-Bird [47] dataset contains $11,788$ bird images, and each bird image has $10$ sentences to describe the fine-grained visual details. The MS-COCO dataset [27] contains $80k$ training images and $40k$ test images, and each image has $5$ sentences to describe the visual information of the scene. We pre-process and split the images on the two datasets by following the same setting in [37], [55].

Evaluation. We from three aspects to compare our DR-GAN with other GAN-based T2I approaches: Image Diversity, Distribution Consistency, and Semantic Consistency. Each model generated $30,000$ images conditioning on the text descriptions from the unseen test set for evaluation. The $\uparrow$ means that the higher the value, the better the performance of the model, and vice versa.

(I) Image Diversity. Following almost all T2I approaches, we adopt the fine-tuned Inception models [55] to calculate the Inception Score (IS $\uparrow$), which measures images diversity.

(II) Distribution Consistency. We use the Fréchet Inception Distance (FID $\downarrow$) [13] and Mode Score (MS $\uparrow$) [44] to evaluate the distribution consistency between generated images and real images. The image features in FID and MS are extracted by a pre-trained Inception-V3 network [38].

(III) Semantic Consistency. We use the R-precision $\uparrow$ and Human Perceptual score (H.P. score $\uparrow$) to evaluate the semantic consistency between the text description and the synthesized image.

R-precision. Following  [50], we also use R-precision to evaluate semantic consistency. Given a pre-trained image-to-text retrieval model, we use generated images to query their corresponding text descriptions. First, given generated image $\hat{x}$ conditioned on sentence $s$ and $99$ random sampled sentences $\{s_{i}^{{}^{\prime}}:1\leq i\leq 99\}$, we rank these $100$ sentences by the pre-trained image-to-text retrieval model. If the ground truth sentence $s$ is ranked highest, we count this as a successful retrieval. For all the images in the test dataset, we perform this retrieval task once and finally count the percentage of success retrievals as the R-precision score. Higher R-precision means greater semantic consistency.

Human Perceptual score (H.P. score). To get H.P. score, we randomly select $2000$ text descriptions on CUB-Bird test set and $2000$ text descriptions on MS-COCO test set. Given the same text description, $30$ volunteers (not including any author) are asked to rank the images generated by different methods. The average ratio ranked as the best by human users is calculated to evaluate the compared methods.

Image Diversity. We use IS to evaluate the image diversity. As shown in Table I, DR-GAN achieves the second-highest IS score on the CUB-Bird test set, and achieves the highest IS score on the MS-COCO test set. The IS score ($5.23$) of RiFeGAN [21] is higher than that of ours ($4.90$) on the CUB-Bird dataset; but RiFeGAN [21] uses $10$ sentences to train the generator on the CUB-Bird dataset, while our DR-GAN only uses $1$ sentence following the standard T2I problem formulation. On the larger-scale and challenging MS-COCO dataset, RiFeGAN [21] uses $5$ sentences to train the model, but obtains a significantly lower IS score $31.70$ than that of our DR-GAN ($34.59$). When the given database only contains images with a single sentence (which is common in practical tasks such as Story Visualization [53], Text-to-Video [52], and other text-guided image generation [51]), RiFeGAN [21] can not be used. In contrast, methods such as our DR-GAN, AttnGAN [50], and DMGAN [31] that only need one sentence per image can be used.

Distribution Consistency. We use FID and MS to evaluate the distribution consistency between the generated image distribution and the real image distribution. As shown in Table I, compared with these GAN-based T2I methods, our DR-GAN achieves the competitive performance on the CUB-Bird and MS-COCO test sets over the FID and the MS. The FID of Obj-GAN [25] is lower than that of our DR-GAN. This is because Obj-GAN [25] and its similar methods [3, 43, 15, 20, 14] require additional information, including the interesting object’s bounding boxes and shapes, for training synthesizing. This additional information can help the generator better capture the object’s layout. This additional information, although available in the MS-COCO dataset, is often unavailable for other datasets such as the CUB-Bird dataset. In general, producing more descriptions or objects’ bounding boxes and shapes on a new database to train the generator is expensive. This limits its scalability and usability in more general text-guided image generation. Thus, our DR-GAN still achieves the competitive performance in the distribution consistency evaluation.

Semantic Consistency. We use the R-precision and the Human Perceptual score (H.P.) to evaluate the semantic consistency. As shown in Table I, compared with AttnGAN and DMGAN [31], our DR-GAN also achieves the best performance on the semantic consistency evaluation on these two datasets.

Generalization. To evaluate the generalizability of our proposed SDM and DNM, we integrate the SDM and DNM modules into several well-known/SOTA GAN-based T2I models including StackGANv2 [11], AttnGAN^- [50], and DMGAN [31]. Here, AttnGAN^- denotes the AttnGAN [50] with its Word-level Attention Mechanism (WAM) removed. As shown in Table II, SDM and DNM can help all these GAN-Based T2I models achieve better performance on all three measures. It indicates that SDM and DNM can be used as general mechanisms in T2I generation task.

Visualization. We qualitatively evaluate our DR-GAN and some GAN-based T2I methods by image visualization. As shown in Fig. 5, compared with AttnGAN [50] and DMGAN [31], our DG-GAN can synthesize higher-quality images on CUB-Bird and MS-COCO test sets.

On the CUB-Bird test set: (i) compared with our DR-GAN, the birds synthesized by AttnGAN and DMGAN contain some strange or incomplete details and structures; Because SDM can better distill key information for image generation, our DR-GAN can perform better on the structure generation; (ii) Due to the introduction of DNM, DR-GAN can better capture the real images distribution. Therefore, compared with AttnGAN and DMGAN, the bird images generated by DR-GAN are more realistic and contain more full semantic details. In particular, the birds wings in Fig.-b-c-d-k have very detailed textures and full colors. On the MS-COCO test set: the semantics of text description are rather sparse; it is very difficult for the generator to capture sufficient semantics to generate high-quality images; therefore, most current methods cannot synthesize vivid images; compared with AttnGAN and DMGAN, our proposed DR-GAN is more reasonable in object layout and richer in semantic details.

Besides, in Fig. 6, we have shown the synthesized images of StackGAN v2 [11], AttnGAN^- [50], and DMGAN [31], and synthesized images of StackGAN v2+SDM+DNM (StackGAN v2^∗), AttnGAN^-+SDM+DNM (AttnGAN^-∗), and DMGAN+SDM+DNM (DMGAN^∗). As shown in Fig. 6, our SDM and DNM can further help these methods (StackGAN v2 [11], AttnGAN^- [50], and DMGAN [31]) improve the structure and enrich the semantic details of synthesized birds, so as to make the generated image more realistic.

Model Cost. As shown in Tabel III, we compare our proposed DR-GAN with other SOTA T2I methods under four model cost measures including Training Time, Training Epoch, Model Size, and Testing Time. Using the MS-COCO dataset as an example, compared with AttnGAN (Baseline) and DM-GAN, our proposed DR-GAN is between AttnGAN and DM-GAN in terms of model cost while achieving the highest performance.

In this subsection, we mainly show and analyze the sensitivity of the hyper-parameters $\lambda_{1}$, $\lambda_{2}$, $\lambda_{3}$, $\lambda_{4}$, $\lambda_{5}$, $\alpha$. In Table I to Table VII (excluding Table III), the six parameters are assigned to values when the DR-GAN performance is around the median value in Fig. 11. So, $\lambda_{1}=10^{-3}$, $\lambda_{2}=10^{-1}$, $\lambda_{3}=10^{-5}$, $\lambda_{4}=1.0$, $\lambda_{5}=1.0$, and $\alpha=5.0$. The FID is an evaluation index to evaluate the quality of synthesized images on the CUB-Bird dataset. The FID results of DR-GAN under different values of the hyper-parameters $\lambda_{1}$, $\lambda_{2}$, $\lambda_{3}$, $\lambda_{4}$, $\lambda_{5}$, $\alpha$ are shown in Fig. 11.

The hyper-parameters $\lambda_{1}$, $\lambda_{2}$: In the Semantic Disentangling Loss Eq.  7, the loss Eq. 5 and the loss Eq. 6 are proposed to drive the SDM better distill the key information from image feature $H$ and word-level context feature $Q$ for image generation. The $\lambda_{1}$ and $\lambda_{2}$ are important balance parameters in the Semantic Disentangling Loss Eq.  7.

As shown in Fig. 11: (i) The Semantic Disentangling Loss has a great influence on the overall performance of DR-GAN. When $\lambda_{1}=0$ or $\lambda_{2}=0$, the FID score of DR-GAN increases, which means that the quality of the generated image deteriorates. (ii) As the values of $\lambda_{1}$ or $\lambda_{2}$ increase, the FID score also increases. The increase of weight means that model training pays more attention to the regression of distribution statistics. The statistics reflect the overall information of the distribution. The constraint of statistics is designed to assist the generator to better approximate the real image distribution. The learning of accurate distribution requires GAN itself to approximate the real distribution based on implicit sampling. Therefore, the weight of SDL should be selected appropriately. Based on the results shown in Fig. 11, the value range of $\lambda_{1}$ is $\{0.001,0.01,0.1\}$, and the value range of $\lambda_{2}$ is $\{0.001,0.1\}$.

The hyper-parameter $\lambda_{3}$: In the Semantic Disentangling Loss Eq.  7, the parameter $\lambda_{3}$ is designed to adjust the weight of the reconstruction loss $||RIRM(I^{*}_{i})-I^{*}_{i}||_{1}$. The reconstruction loss $||RIRM(I^{*}_{i})-I^{*}_{i}||_{1}$ can provide the real image feature $H^{*}$ for other terms in Semantic Disentangling Loss. When $\lambda_{3}=0$, the reconstruction mechanism (i.e. RIRM) doesn’t work. So, the performance of DR-GAN also drops significantly. With the removal of the loss, it is difficult for the SDM to match the valid mean and variance of real image features. Since this, real image features would be mixed with more semantic information irrelevant to image generation, or some important image semantic information would be suppressed. The bad real image features mislead the generator to learn the bad image distribution and make the quality of the generated image decline. Besides, We found that performance decreased as the value of $\lambda_{3}$ increased. Because the decoder in RIRM and the generation modules $G^{o}$ are the Siamese Networks. The purpose of this design is that the real image features $H^{*}$ and the generated image features $H$ can be mapped to the same semantic space. The increase of weight will make the generation modules $G^{o}$ pay more attention to the reconstruction of real images by real image features, and weaken the generation of generated images. That is, the Siamese Networks are prone to the imbalance between two feature sources in the training process. In all, based on the results shown in Fig. 11, the value range of $\lambda_{3}$ is about $[0.00001,1]$.

The hyper-parameter $\lambda_{4}$, $\lambda_{5}$: The hyper-parameter $\lambda_{4}$ balance the weight of the Eq. 9 in the generation stage loss of DR-GAN; The hyper-parameter $\lambda_{5}$ balance the weight of the Eq. 8 in the generation stage loss of DR-GAN; The loss Eq. 9 and loss form the Distribution Adversarial Loss. When $\lambda_{4}=0$ or $\lambda_{5}=0$, the FID score increases, and the generated image quality decreases. This means that the two-loss terms of Distribution Adversarial Loss can effectively improve the distribution quality of the generated image. When the value of the $\lambda_{4}$ or $\lambda_{5}$ increases, the quality of the image distribution tends to decline. When the weight is too large, the image latent distribution will be over-normalized. At this point, the discriminant model becomes very powerful. Just like GAN’s theory [8], a strong discriminator is not conducive to generator optimization. So, based on the results shown in Fig. 11, the value range of $\lambda_{4}$ and $\lambda_{5}$ is $[0.1,2]$.

The hyper-parameter $\alpha$: In the training stage of DR-GAN, we also utilize the DAMSM loss [50] to make generated images better conditioned on text descriptions. In Fig. 11, we show the performance of DR-GAN based on a different value of the hyper-parameter $\alpha$. When $\alpha$, the FID score increases, and the generated image quality decreases. This means that the constraints of semantic matching help to improve the quality of image generation. Besides, when the value of $\alpha$ changes constantly, the performance of DR-GAN also changes moderately, and the overall performance is relatively good. Based on the results shown in Fig. 11, the value range of $\alpha$ is about $[1,20]$.

The experiments showed the effectiveness of our DR-GAN in T2I generation. However, there are a few failure cases. Some failure cases from the CUB-Bird data are shown in the top row of Fig. 12. Distribution normalization in SDM could sometimes lead to missing spatial local structures/parts such as heads, feet, and necks. Some failure cases from the MS-COCO dataset are shown in the second row. Like in most other existing T2I models[50, 25, 36, 3, 39, 9, 31], it is difficult for the text encoder to parse reasonable location information of objects in a scene if it is not specifically provided in sentences. Then, the generator in DR-GAN tends to randomly place some objects that are sometimes unreasonable.

We will explore the objects’ location relationship from the text knowledge graph in our future work. We will also explore the application of SDM and DNM on other/broader GAN-based image generation tasks, such as Image-to-Image Translation [22, 35] and Virtual Try-On [10, 1].