CLIP2GAN: Towards Bridging Text with the Latent Space of GANs

Due to the great potential of GAN in generating realistic and high-resolution images, it is widely used in image generation and applications [28, 29, 53, 46]. In particular, high-quality face generation has been an attractive problem in image generation. PGGAN [21] first proposes the idea of resolution progressive generation to generate high-definition face images, which first discovers large-scale structures and then focuses on fine details. StyleGAN [24, 25, 23] introduces a novel style-based generator architecture that generates face images with high fidelity and high resolution. It controls the visual features represented in each layer individually, which can be coarse features influenced by style (e.g., pose, face shape, identity features, etc.) or detailed features influenced by noise (e.g., pupil, hair, wrinkles, etc.). Unlike subsequent studies [22, 3, 4] that mostly introduces different mechanisms or structures on StyleGAN, we use pre-trained StyleGAN to achieve high-quality face generation at a low cost and high efficiency.

With the remarkable progress in both computer vision and natural language processing, researchers turn their attentions to joint vision-language (VL) models for many task-specific VL problems, including image captioning [49, 20, 45], visual question and answer (VQA) [5, 51], image text matching [20, 19], etc., which are tailored for specific problems and each model only solves one task. After the introduction of the transformer [44], BERT [12] has achieved unprecedented success in various language tasks. A recent development, CLIP [36], pre-trained using over 400 million image-text pairs based on contrastive learning, learns a multi-modal co-embedding space and estimates the semantic similarity between texts and images. The robustness of learned joint representation enables CLIP to offer high performance and excellent generalization on various tasks.

Text-guided image generation is an interesting topic in image generation, where GAN-based models show better sample quality. StackGAN [55, 56] stacks several generators and discriminators to improve the resolution of generated images in multiple stages. AttnGAN [50] introduces a cross-modal attention mechanism to explore fine-grained text and image representations. XMC-GAN [54] utilizes contrastive learning for image generation. TediGAN [48] trains an encoder to map the text into the latent space of StyleGAN [24]. DF-GAN [42] proposes a one-stage backbone for the direct synthesis of high-resolution images. Compared to most previous work, we have achieved better performance without using text training.

Similar to text-guided generation, manipulating a given image using text produces results containing the desired properties. The difference is that the edited result should change the parts related to the text and retain the rest of it. For instance, Dong et al. [14] propose an encoder-decoder structure for text-guided manipulation, and Li et al. [31] generate high-quality images through a multi-stage network. Unlike most text-guided image manipulation based on a multi-stage framework, we propose a unified framework that allows text-guided image generation and image manipulation without requiring multi-stage processing.

Fig. 2 gives an overview of our framework, which consists of a pre-trained vision-language model (CLIP), a mapping network, and a pre-trained generation model (StyleGAN). Unlike previous work that uses a large number of image-text pairs for model training, our approach achieves text-free training by establishing a mapping relationship between the CLIP multi-modal embedding space and the StyleGAN latent space.

On the one hand, to achieve text-guided image generation without text training, we generate pseudo-text features by leveraging the image-text feature alignment of a pre-trained model. We require a universal multi-modal embedding space where the paired text and image features can be well aligned. The recent vision-language model CLIP achieves this by pre-training a large number of image-text pairs through Contrastive Learning, which is exactly what we need. On the other hand, given that StyleGAN has excellent latent space, we can perform a series of manipulations on the generated images by changing its latent code. We take advantage of the pre-trained StyleGAN2 as the model for image generation. With the help of StyleGAN’s latent space, we can get high-quality text-guided image generation and image editing of images.

To generate images from text, we build a bridge between CLIP and StyleGAN through a mapping network. With this mapping network, it is possible to obtain feature representations of text or images in the latent space of StyleGAN and thus generate images using StyleGAN. The source image $x$ is taken as the input of CLIP and the image encoder of CLIP is used to obtain the image features $f_{img}$, i.e. pseudo-text features $\tilde{f}_{text}$, in the multi-modal embedding space of CLIP. It is formulated as follows,

$$\tilde{f}_{text}=f_{img}=C_{img}(x),\vspace{-0.1cm}$$

(1)

where $C_{img}(\cdot)$ denotes the image encoder of the CLIP model [36]. The image features $f_{img}$ are mapped to latent codes $z$ of StyleGAN in $w+$ space by the mapping network as the input of the pre-trained StyleGAN, and the image $x^{\prime}$ is generated by StyleGAN. $x^{\prime}$ is expressed as follows,

$$x^{\prime}=G(M(C_{img}(x))),\vspace{-0.1cm}$$

(2)

where $M(\cdot)$ denotes the mapping network and $G(\cdot)$ denotes the pre-trained StyleGAN model. By learning the consistency of the source image $x$ and the generated image $x^{\prime}$, our generative model is implemented.

The mapping network is capable of mapping the multi-modal embedding space of CLIP into the $w+$ latent space of StyleGAN, which makes it possible to invert CLIP features back into the source images using StyleGAN. Our proposed mapping network is a 12-layer fully connected layer with each layer post-connected to the activation layer Leaky ReLU [33]. The mapping network converts a 512-dimensional text feature $f_{text}$ or image feature $f_{img}$ from the multi-modal embedding space of CLIP into the $w+$ space of StyleGAN for obtaining an 18$\times$512-dimensional latent code $\mathbf{z}$, which is formulated as follows,

$$\mathbf{f}^{i+1}=g(\mathbf{w}_{i}\cdot\mathbf{f}^{i}+\mathbf{b}_{i}),\ \mathbf{f}^{0}=f_{text}\ or\ f_{img},\vspace{-0.1cm}$$

(3)

where $\mathbf{f}^{i+1},\mathbf{f}^{i}\in\mathbb{R}^{H\times W\times C}$ are the input and output features, respectively. $g(\cdot)$ denotes the Leaky ReLU activation.

However, we discover that when using only a simple mapping network, CLIP suffers from missing details in both the inversion of the CLIP image features and generated images. Although it can ensure that the major features are presented, the image details, especially the hair and skin textures, backgrounds, etc., are lost to varying degrees, which is attributed to the fact that CLIP only extracts the major 512-dimensional features of the image and ignores the others. To tackle this issue, we introduce a discriminator to determine the truthfulness of the obtained images. It is trained adversarially with the mapping network to complement the image details and improve the generation quality without affecting the feature representations.

Furthermore, when constraints are applied only between the source images and the reconstructed generated images, it is observed that the generated image examples lack diversity. This is because the loss between pixels mainly focuses on the reconstructed identical images rather than the diverse images, although diversity contributes to the performance. To this end, we apply diversity loss on the latent space to generate diverse images as shown in Fig. 2. Inspired by the mode seeking regularization [34], the diversity loss is achieved by maximizing the ratio of the distance between the output images to the distance between the corresponding latent codes, and it encourages the generation of distinctive results when different noise vectors are brought in. Inputting an arbitrary specific text description, we are able to generate multiple images that match the text features but are unique with an additional Gaussian noise of $\mu=1,\sigma^{2}=0.36$.

Considering the image-text alignment property of CLIP’s multi-modal embedding space, we input the source image and learn the mapping of CLIP image features, i.e., pseudo-text features, to the latent space by image reconstruction. That means training using image data, as shown in Fig. 2. Meanwhile, during testing, the target text description is input and text-guided image generation is performed utilizing CLIP text features, which is shown in Fig. 3. Through the above process, a solution for text-guided image generation without text training is achieved.

We utilize several objective functions, i.e., reconstruction loss, perceptual loss, adversarial loss, identity loss, and diversity loss to optimize our framework.

With the pre-trained CLIP2GAN model, we further apply the network to text-guided image editing applications. Given a source image $x$, we are interested in editing certain regions of it by manipulating its latent codes $\mathbf{z}=\{\mathbf{z_{i}}\}_{i=1}^{i=18}$ to $\mathbf{z^{\prime}}=\{\mathbf{z^{\prime}_{i}}\}_{i=1}^{i=18}$ and getting a target image $x^{\prime}$ that meets the editing requirements, which is expressed as follows,

$$\mathbf{z^{\prime}_{i}}=\mathbf{z_{i}}+\beta\mathbf{n_{i}},\vspace{-0.1cm}$$

(11)

$$x^{\prime}=G(\mathbf{z^{\prime}}),\vspace{-0.1cm}$$

(12)

where $\mathbf{n}=\{\mathbf{n_{i}}\}_{i=1}^{i=18}$ corresponds to the normal direction of a particular semantics of the latent space, and $\beta$ denotes the degree of editing semantics. That is, if the latent code moves in a certain direction, the semantics contained in the output image should vary accordingly. This requires our framework to locate the semantic direction $\mathbf{n}$ of the text and the latent code $\mathbf{z}$ of the image.

As shown in Fig. 4, the text, $t$ i.e., simple descriptions on age, gender, hair, expression, etc., is fed into the CLIP [36] text encoder to get CLIP text features. Then the vector $\mathbf{n}$ in the StyleGAN latent space is derived by the pre-trained mapping network, which is considered as the normal direction of a particular semantics due to the mapping network bridging the CLIP feature space and the StyleGAN latent space. Meanwhile, by putting the source image $x$ through the CLIP image encoder and the mapping network, the representation $\mathbf{z}$ of $x$ in the StyleGAN latent space is obtained. They are formulated as follows,

$$\mathbf{n}=M[C_{text}(t)],\vspace{-0.1cm}$$

(13)

$$\mathbf{z}=M[C_{img}(x)],\vspace{-0.1cm}$$

(14)

where $C_{text}(\cdot)$ and $C_{img}(\cdot)$ denote the CLIP text and image encoder, respectively.

Finally, the semantic direction $\mathbf{n}$ and the latent feature $\mathbf{z}$ are weighted together and the weight of $\mathbf{n}$, i.e., $\beta$, ranges from 0 to 1, leading to the modified feature $\mathbf{z^{\prime}}$ in the latent space of StyleGAN, and thus StyleGAN generates the corresponding images. This allows the text description to control the degree of semantic modification of the image while not changing the identity features of the image itself. Our approach achieves high-quality text-guided image editing by simple arithmetic operations without optimization and additional network structures.

In addition, accuracy and realism are evaluated through a user study. For image generation, the accuracy is evaluated by the similarity between the text and the generated image. For image manipulation, accuracy is assessed by whether the visual properties of the modified image are aligned with the given description and whether contents unrelated to the text are preserved. Realism is required to be judged as to which is more realistic and consistent with reality. We tested accuracy and realism by collecting surveys on a random sample of 10 images from 20 people.

We compare our method with several state-of-the-art methods of text-guided image generation, i.e., AttnGAN [50], ControlGAN [30], DF-GAN [42], DM-GAN [59], TediGAN [48], and StyleCLIP [35]. We evaluate FID and LPIPS on a large number of samples generated from randomly selected text descriptions. Accuracy and realism are assessed by user studies. We generate 10 images using different methods and 20 users were asked to judge which one is the most realistic and aligned. The quantitative result is shown in Tab. 1 and Tab. 2, where the other methods are trained and tested on the MM-CelebA-HQ dataset, while our method is trained on CelebA-HQ and tested with the help of the MM-CelebA-HQ text descriptions. From these tables, it is observed that our method achieves new state-of-the-art performance.

Furthermore, we make a qualitative evaluation with several competitive methods, i.e., AttnGAN, ControlGAN, DF-GAN, DM-GAN, TediGAN, and StyleCLIP. From Fig. 5, it is observed that our method has higher image quality and more realistic image results compared with previous methods. The images we generate correspond very closely to the text description as well as showing fine-grained details. Benefiting from the CLIP model [36], we have an effective text-driven capability to generate face images with more text features in the category without being limited to text descriptions in the dataset.

When some specific features in the text are changed, our model ensures that the image is modified only in the corresponding features, while other features, including identity features, are guaranteed to remain invariant. This shows that our method is able to decouple different features with excellent robustness. The text description and visual results are presented in Fig. 8.

The other advantage is that our model can inherently generate diverse results given an arbitrary specific text description. With our approach, the generated images are guaranteed to be consistent with the given text features while other irrelevant features are varied to get multiple unique results. We present the text-guided image generation results in Fig. 6, which demonstrates that our method can generate diverse results with high quality.

There are several ablation experiments performed to demonstrate the effectiveness of the framework. To evaluate the effectiveness and necessity of the network design, we modified the mapping network with a different number of network layers and the pre-trained models of CLIP. From Tab. 3, it is observed that the framework performs worse in several metrics if the number of network layers is less. Whereas, with a higher number of layers, it is less significant for performance improvement, while greatly increasing the complexity of the network. In addition, the feature space of CLIP has semantic meanings between images and text, thus using a stronger joint space (ViT/B-16) can improve the generated results.

We also investigate the impact of each component in the objective function. Ensuring that reconstruction loss, perceptual loss, and identity loss are preserved, we ablate by excluding adversarial loss and diversity loss one by one. The results are shown in Tab. 4, and the performance decreases after each removal. This is because removing the adversarial loss loses image realism while removing the diversity loss leads to pattern convergence.

To evaluate the performance of image editing, we use the FID metrics and conduct a user study. As shown in Tab. 5, the results show that our method has better FID, accuracy, and realism than other methods [31, 48, 35] on both CelebA-HQ dataset and arbitrary images that are not within the CelebA-HQ dataset. As well as generating high-quality text-guided images, we can achieve image editing for a given text description while ensuring that irrelevant content remains unchanged.

Fig. 7 shows the visual results of our text-guided image editing. With simple word descriptions, we can apply different manipulations to the image without changing other irrelevant features. For the various text features used for modification, the extent to which the corresponding features of the image are changed is controlled by our framework, i.e., the value of the weight $\beta$, as illustrated in Fig. 9. Since CLIP is trained on a massive dataset of image-text alignment, which possesses a huge text latent space. Therefore, using the pre-trained CLIP, our method can achieve text editing for numerous different features without being limited to some specific text descriptions.