DeltaEdit: Exploring Text-free Training for Text-Driven Image Manipulation

Learning generic vision-language representations is of great importance in the multi-modal field. Following the success of BERT [2], many large scale pre-trained vision-language models [13, 18, 26, 29, 39] are proposed. A recent development, CLIP [21], is extremely powerful for joint vision-language learning. It is trained on 400 million (image, text) pairs and learns a joint multi-modal semantic space for both images and texts. Benefit from the excellent image/text representation ability, it has been widely used in various directions, such as domain adaptation [5], image segmentation [40, 22], and image generation [4, 42, 35, 16, 14, 27].

The goal of text-guided image generation [23, 1, 28, 37, 38, 34] is to generate realistic images according to given text descriptions. As a pioneering work, Reed et al. [23] embed the text features as the conditional input for a GAN-based one-stage architecture. After that, StackGAN [37] and StackGAN++ [38] introduce multi-stage frameworks to further improve the generation quality. Recently, some works [25, 27, 33, 41] based on StyleGAN [7, 8, 6] have been proposed to improve the quality of generated images.

Compared with generating, text-guided image manipulation [3, 19, 17, 12, 33, 35, 9] aims to manipulate the input images by using texts describing desired visual attributes (e.g., gender, age). For example, ManiGAN [12] proposes affine combination module (ACM) and detail correction module (DCM) to generate new attributes matching the given text. Recently, some works [25, 33] have adopted StyleGAN as their backbone to perform image manipulation tasks. For TediGAN [33], it aligns two modalities in the latent space of pre-trained StyleGAN by the proposed visual-linguistic similarity module. More recently, StyleCLIP [20] combines the generation power of StyleGAN and the image-text representation ability of CLIP [21] to discover manipulation direction. They outline three approaches, namely Latent Optimization, Latent Mapper and Global Directions, which are denoted as StyleCLIP-OP, StyleCLIP-LM, and StyleCLIP-GD in this paper. The StyleCLIP-OP and StyleCLIP-LM are per-prompt training methods, which require optimizing the latent code or training a separate model for each text prompt. The third approach, StyleCLIP-GD, is a per-prompt fine-tuning method. It first finds global directions of semantic change in StyleGAN’s style space $\mathcal{S}$ by pre-defined relevance matrix of each channel in $\mathcal{S}$ to the image-space change. During inference, it needs to manually tune hyper-parameters to discover the fine-grained directions for each specific text prompt. In addition, HairCLIP [31] focuses more on the hair manipulation with the help of CLIP. FFCLIP [43] collects 44 text prompts for face images to learn the facial attributes manipulation.

As illustrated in Figure 3, in the training phase, the source embeddings, $i_{1}$ and $s_{1}$, are taken as the input. The condition is from a target CLIP image embedding $i_{2}$ instead of a target CLIP text embedding. Then, the source embeddings and the image condition are sent to a latent mapper network to predict the manipulation direction $\Delta s^{\prime}$:

$$\Delta s^{\prime}=LatentMapper(s_{1},i_{1},i_{2})\text{.}$$

(1)

During inference, we can predict the editing direction $\Delta s^{\prime}$ as:

$$\Delta s^{\prime}=LatentMapper(s_{1},i_{1},t)\text{,}$$

(2)

where $t$ is a CLIP text embedding constructed from a target text prompt.

However, as shown in Figure 4, given different source images, the naïve method maps them all to average faces corresponding to the text-related attributes, leading to inaccuate manipulation. We doubt that this phenomenon occurs because there is still a modality gap of the CLIP visual-linguistic modalities, even though the cosine similarity between paired image-text features are maximized to make the image-text space aligned in CLIP. To verify it, we perform t-SNE visualization on the MM-CelebA-HQ dataset [33] in Figure 2, which shows that the CLIP image space and text space are actually not that close to each other. A recent work [15] has also demonstrated that the joint multi-modal space in CLIP is not well-aligned caused by model initialization and contrastive representation learning.

After further practice, it is encouraging that we have identified a better semantically aligned multi-modal feature space, namely the CLIP delta image-text space, which is extremely close to each other, making it possible to approximate the text conditions with image embeddings in the proposed delta space. Additionally, there is an advantage of leveraging the text embeddings change as conditions, such as “face” as the source text and “face with yellow hair” as the target text. That is, the importance of words that indicate the desired attributes (like “yellow hair”) can be enhanced, which alleviates the problem that CLIP is not sensitive to fine-grained or complex words [21, 24].

In our DeltaEdit framework, we extract delta image-text features, $\Delta i=i_{2}-i_{1}$ and $\Delta t=t_{2}-t_{1}$, as the condition vectors in the training phase and inference phase, respectively. As shown in Figure 5, during training, we take the extracted codes to get CLIP image space direction $\Delta i=i_{2}-i_{1}$ and StyleGAN $\mathcal{S}$ space direction $\Delta s=s_{2}-s_{1}$. Then, we propose a latent mapper network in delta space, called Delta Mapper, to predict the editing direction as:

$$\Delta s^{\prime}=DeltaMapper(s_{1},i_{1},\Delta i)\text{,}$$

(3)

where $s_{1}$ and $i_{1}$ are used as the input of Delta Mapper to provide specialized information for the source image.

Delta Mapper. The architecture of the Delta Mapper is illustrated in Figure 5 (b). Since StyleGAN has the property that different layers correspond to different semantic levels [7, 33], it is common to divide these layers into different levels and implement coarse-to-fine manipulation within each level. Following [20, 31], we adopt three levels of sub-modules (coarse, medium, and fine) for each designed module. Each sub-module contains several fully-connected layers. In this way, with different levels of the source latent code $s_{1}$, we propose Style Module to obtain coarse-to-fine intermediate features $(e^{s}_{c},e^{s}_{m},e^{s}_{f})$, where subscripts stand for coarse-to-fine level and superscripts stand for the Style Module. Then, we concatenate the $\Delta i$ and $i_{1}$ as input, and propose Condition Module to learn coarse, medium and fine embeddings $(e^{i}_{c},e^{i}_{m},e^{i}_{f})$ separately, which have same dimensions as $(e^{s}_{c},e^{s}_{m},e^{s}_{f})$. In the final step, we fuse generated coarse-to-fine features using proposed Fusion Module with three sub-modules ($M^{f}_{c}(\cdot,\cdot),M^{f}_{m}(\cdot,\cdot),M^{f}_{f}(\cdot,\cdot)$) and predict editing direction as:

$\displaystyle\Delta s^{\prime}=$

$\displaystyle(M^{f}_{c}(e^{i}_{c},e^{s}_{c}),M^{f}_{m}(e^{i}_{m},e^{s}_{m}),M^{f}_{f}(e^{i}_{f},e^{s}_{f})).$

(4)

To train the proposed Delta Mapper, our full objective function contains two losses, which can be denoted as:

$\displaystyle\mathcal{L}\!=\!\mathcal{L}_{rec}\!+\!\mathcal{L}_{sim}\!=\!\|\Delta s^{\prime}\!-\!\Delta s\|_{2}+\!1\!-cos(\Delta s^{\prime},\Delta s),$

(5)

where L-2 distance reconstruction loss is utilized to add supervision for learning the editing direction $\Delta s^{\prime}$ and cosine similarity loss is introduced to explicitly encourage the network to minimize the cosine distance between the predicted embedding direction $\Delta s^{\prime}$ and $\Delta s$ in the $\mathcal{S}$ space.

Text-driven Image Manipulation. During inference, based on the well-aligned multi-modal delta feature space, we can achieve text-driven image manipulation with the trained Delta Mapper. As shown in Figure 5 (c), given the source image $I$, we first extract its CLIP image embedding $i$ and StyleGAN $\mathcal{S}$ space embedding $s$. Then, we construct $\Delta t$ with source and target text prompt, and we can predict the editing direction $\Delta s^{\prime}$ as:

$$\Delta s^{\prime}=DeltaMapper(s,i,\Delta t),$$

(6)

where the editing direction is subsequently used to generate edited latent embedding $s^{\prime}=s+\Delta s^{\prime}$. In the final step, we can generate the synthesized image $I^{\prime}$ with StyleGAN based on $s^{\prime}$.

Text Prompts. In inference phase pipeline, the crucial issue is how to construct $\Delta t=t_{2}-t_{1}$. To establish the correspondence with the training phase, the source text and target text should be specific text descriptions of two different images. However, the DeltaEdit network will generate images towards the direction of the target text and the opposite direction of the source text in this manner. Therefore, to remove the reverse effect of the source text prompt containing attributes, we naturally place all the user-described attributes in the target text prompts. For human portrait images, if the user intends to add “smile” and “eyeglasses” to a face, then our DeltaEdit uses “face with smile and eyeglasses” as the target text and “face” as the source text. This text prompt setting is also applicable to other manipulation domains. In experiments, we further verify that it enables accurate image manipulation.

Disentanglement. To improve the disentanglement, we further optimize the obtained editing direction $\Delta s^{\prime}$ using the pre-computed relevance matrix $R_{s}$, which records how CLIP image embedding changes when modifying each dimension in $\mathcal{S}$ space  [20]. We can set some channels of $\Delta s^{\prime}$ as zero if the channels have a low correlation to the target text according to $R_{s}$.

To verify the effectiveness and generalization of the proposed method, we conduct extensive experiments on a diverse set of challenging domains. For the facial domain, we randomly choose 58,000 images from FFHQ [7] dataset and sample 200,000 fake images from the $z$ space in StyleGAN for training; we use the remained 12,000 FFHQ [7] images for evaluation. We additionally provide results on the LSUN [36] Cat, Church and Horse datasets. Note that all real images are inverted by e4e encoder [30], and all generated images are obtained using pre-trained StyleGAN2 [8] generators. The proposed method is trained on 1 NVIDIA Tesla P40 GPU. During training, we set batch size as 64 and adopt ADAM [10] optimizer with $\beta_{1}=0.9$, $\beta_{2}=0.999$ and a constant learning rate of 0.5.

We perform multiple qualitative evaluations with different text descriptions on several challenging datasets. More results are shown in Supp.

Various Text-driven Manipulation Results. To evaluate the generalization ability of our method that can be driven by texts with different semantic meanings, we show facial manipulation results of 21 attributes in Figure 6, which are generated from one trained model. The results show that only the target attributes are manipulated, while other irrelevant attributes are well preserved. Meanwhile, the results are well adapted to the individual with diverse details, rather than overfitting to the same color or shape, which can be seen in “red hair” and “ bowlcut hairstyle” obviously. In addition, the manipulated results on images of cats, churches and horses are shown in Figure 17 and Figure 8, where StyleGAN2 pretrained on LSUN cat, church and horse datasets [36] are used. It is worth mentioning that all text prompts have never been seen during training, which further indicates the effectiveness of the proposed method.

Comparisons. We compare our method with state-of-the-art text-guided image manipulation methods, TediGAN [33], StyleCLIP-LM [20], and StyleCLI-GD [20]. As shown in Figure 9, the results produced by TediGAN are highly entangled and almost fail in editing attributes, such as “long hair” and “eyeglasses”. The results of StyleCLIP-LM are unstable and also entangled. Moreover, the results of StyleCLIP-GD have better disentanglement than StyleCLIP-LM as it is well-tunned for each case. However, the disadvantage of StyleCLIP-GD is the complex hyper-parameters tunning and high inference time reported in Table 3. In comparison, our method yields the most impressive and disentangled results almost in each case.

To verify the inference flexibility of our method, we further compare our method with the strong baseline, i.e., StyleCLIP-GD, which achieves manipulation by manual tuning two hyper-parameters, disentanglement threshold $\beta$ and manipulation strength $\alpha$. Since we also introduce disentanglement threshold $\beta$ to control some channels of the obtained $\Delta s^{\prime}$ as zero. For a fair comparison, we empirically set $\beta$ at 0.03 and compare with StyleCLIP-GD under different manipulation strength $\alpha$ in Figure 10. The results demonstrate that StyleGAN-GD cannot be generalized to different image attributes and produce feasible editing results under the same hyper-parameters. For example, the result conditioned on “big eyes” is the most accurate at $\alpha=1$ while the result conditioned on “black clothes” is the most correct when $\alpha=4$. In the contrast, our method achieves ideal manipulation results of various text prompts without manually tuning the manipulation strength. Moreover, under the same disentanglement threshold $\beta$, StyleCLIP-GD results tend to entangle with some irrelevant attributes. For example, for text prompt “blue suit”, the generated results are entangled with “blue eyes”, “mouth slightly open”, “wearing earrings” and etc. On the contrary, our method can achieve more disentangled results without complex hyper-parameters tuning.

In Table 1, we present objective measurements of FID, PSNR, and IDS (identity similarity before and after manipulation by Arcface) for comparison. The results are the average on ten editing texts. Compared with the state-of-the-art approaches, we achieve the best performance on all metrics. We also compare with the naïve solution to text-free training. For fairness, it is also trained in the $\mathcal{S}$ space and supervised by same loss functions in DeltaEdit. Compared with the naïve solution, the final solution (Ours) can largely improve the inference performance on all metrics, benefiting from the well-aligned CLIP delta image-text space. Finally, we conduct human subjective evaluations upon the manipulation accuracy (Acc) and visual realism (Real). We compare DeltaEdit with the strong baseline, StyleCLIP-GD, under different hyper-parameter $\alpha$ settings. In total 20 evaluation rounds are performed and 40 participants are invited. At each round, we present results of randomly sampled editing text to each participant. Participants were asked to choose the best text-driven image manipulation output considering Acc and to select the output images (not limited to 1 image) which are visually realistic (Real). The results are listed in Table 2, showing the superiority of our model.

To validate the efficiency and flexibility of the proposed method, we compare the computation time with TediGAN and three StyleCLIP approaches in Table 3. Specifically, TediGAN first requires 12+ hours to encode images and texts into a common space and then trains the encoding module. StyleCLIP-OP manipulates images with an optimization process, which requires several tens of seconds. StyleCLIP-LM is fast during inference, but needs 10-12 hours to train a mapper network for per text prompt. StyleCLIP-GD requires about 4 hours to pre-compute the global edit directions. However, since the pre-computed directions are rough and cannot be directly applied to control the manipulation, it requires additional manual tuning for different text prompts, which typically takes 8-9 seconds for each case. On the contrary, we avoid the high labor time during training and inference. Once trained, our Delta Mapper is universal and can directly manipulate images with new text prompt efficiently. Meanwhile, our method only needs 2-3 hours for training, since the training is directly conducted on the latent space without generating images in each iteration.

In Figure 11, we explore how different text prompt settings can affect the editing results. During inference, the editing direction $\Delta s^{\prime}$ is driven by the CLIP text space direction $\Delta t$ between source text and target text. (a) We first construct three different (source text, target text) pairs and find that regardless of the content of them, the editing results are affected by the difference between them. For example, although both the source and target texts contain the attribute of “bangs”, the editing result is only influenced by the difference, namely “black hair”. (b) Moreover, to align with the training phase, we construct (source text, target text) pairs are each a specific text description for an image, such as “face is sad” as the source text and “face with bangs” as the target text. We find that the generated face to become more “happy” (the opposite direction of the source text) and with “bangs” (the direction of the target text), which is equivalent to directly putting the attributes to be edited all into the target text. (c) In addition, we construct target texts with multiple combinations of facial attributes, including hairstyles, smile and eyeglasses, and our method can yield desired results driven by text prompts containing multiple semantics. Note that our method can directly perform multi-attributes manipulation without additional training processes, since the different manipulation direction has been learned well by training on large-scale data. More exploration experiments about text prompts are seen in Supp.

Choice of the manipulation space. To find an appropriate space for the proposed method, we conduct experiments by performing manipulations in $\mathcal{W+}$ space and $\mathcal{S}$ space, respectively. The quantitative and qualitative results in Table 1 and Figure 12 shows that $\mathcal{S}$ results can achieve better visual quality and identity preservation than $\mathcal{W+}$. Thus, we implement our DeltaEdit in $\mathcal{S}$ space, and introduce $R_{s}$ to further improve the disentanglement performance.

The effectiveness of relevance matrix $R_{s}$. For further improving the disentanglement, we introduce relevance matrix $R_{s}$ to limit some irrelevant channels from changing. The fourth row in Figure 12 shows that, with $R_{s}$, our method can successfully edit the desired attributes while preserving the text-irrelevant content unchanged. For example, for ”bangs”, the generated results have bangs while unchanging the background or facial pose.