SynCDR : Training Cross Domain Retrieval Models with Synthetic Data

In the main paper under limitations, we mentioned that our synthetic data generation methods use simply a word or a phrase to represent the target domain (Tab. 12 lists these for different methods). This may not be possible in scenarios where this description is unavailable, e.g. in data collected from specific sensors. We hypothesize that in such a case, Textual Inversion [9] can be used. It allows us to encode a concept exemplified via a set of input images in a new token by optimizing its embeddings such that a pre-trained and frozen Stable Diffusion model generates the input images conditioned on this new token. Note that this description obfuscates some details to provide an intuitive understanding. For a full and precise description of textual inversion, we refer readers to Sec 3 of [9].

While in this paper we did not experiment with datasets where domains cannot be verbally described, we evaluated textually inverting domains on the DomainNet dataset. For textual inversion, we used a pre-trained Stable-Diffusion v1.4 model (which is the same version as used by ELITE), and encoded each domain (clipart, painting or sketch) using its unlabeled training images by using 5000 gradient steps, with a learning rate of $5\times 10^{-4}$ and a batch size of 4. We used code available in [27]. For generating a synthetic image in domain A, we use the prompt “An image of S in the style of $<d_{A}>$”, where S is the token corresponding to the input image object encoded by ELITE and $<d_{A}>$ corresponds to the textually inverted token encoding the properties of domain A.

From the results in Tab. 7, we see that textually inverting a domain’s properties leads to similar or poorer results overall. It is poorer in the case when sketch is involved as a domain. On digging deeper, we find that when generating paintings or cliparts from sketches, using textual inversion is poorer at preserving the category of the input sketch example. For instance, NCM accuracy (as defined in Sec. 4.3 of the main paper) of synthetic paintings generated with textual inversion is 0.54 compared to 0.61 without (see some examples of such cases in Fig. 3). This suggests the generation process somehow get skewed more to pay attention on only the style in such cases. We leave further investigation of this method to future work.

As mentioned in Sec. 4.1 (main paper) and as shown in Tab. 8, image embeddings from large CLIP transformer models are robust across domains, but are expensive to obtain on smaller memory devices at inference time. In the top section of Tab. 8, we report the performances of two of these CLIP pre-trained models. The CLIP RN-50 is a larger version of the ResNet-50 model with $\sim$40M parameters compared to the latter’s $\sim$25M. The CLIP ViT-L is a large transformer model, which provides highly robust image embeddings at test time, but is expensive to store and run with $\sim$305M parameters.

We additionally note that ELITE’s image encoder, which generates the token embedding encoding an input image concept, uses the CLIP ViT-L model. ELITE can hence be interpreted as a way of distilling the information in the CLIP ViT-L via synthetic training data. An alternative approach would be to distill the CLIP ViT-L model based on its feature similarities between pairs of images in the training data. This approach is described next.

CLIP-distill (Tab. 8) distills a CLIP ViT-L into a ResNet-50. Its optimization criterion is similar to Sun et al. [41], but uses only unlabeled images available to us as training data. More precisely, during training we are given a teacher (CLIP ViT-L) feature extractor $g:\mathcal{X}\rightarrow\mathbb{S}^{d}$ (recall $\mathbb{S}^{d}$ is the $d$-dimensional unit hypersphere) and a student model $f:\mathcal{X}\rightarrow\mathbb{S}^{d}$. Additionally for $m$-sized minibatches of images $X_{A}$ from the domain A and $X_{B}$ from domain B, let us define $p_{f}(\boldsymbol{x},X)=\text{softmax}(f(\boldsymbol{x})^{\top}f(X))$ (where $\boldsymbol{x}\in\mathcal{X}$ and we overloaded notation for the batch of $m$ images $X$ so that $f(X)=[f(\boldsymbol{x});\boldsymbol{x}\in X]\in\mathbb{R}^{d\times m}$ )

where $D_{KL}$ is the KL divergence between two distributions. In essence, we attempt to distill the similarity predictions made by the teacher CLIP model $g$ for training images across domains (terms 1 and 3 of Sec. 0.A.2), and those within the same domain (terms 2 and 4 of Sec. 0.A.2).

From the results, we see that distilling the large CLIP model helps with performance compared to ID or CDS, but SynCDR (i.e. training with synthetic data) is a much more effective approach in terms of performance.

In Tabs. 9, LABEL:, 10, LABEL: and 11, we report Precs @1, 5 and 15 retrieved images as done in [22]. We observe largely similar trends using Prec@5 and Prec@15 as those using Prec@1, and results are reported for completeness.

Fig. 4 shows t-SNE visualization of the output features from the network before and after SynCDR training. The two domains in the plot are Clipart and Painting from the DomainNet dataset and we used 1000 points from the test set of each domain picked at random for the visualization. We see that features get better clustered class-wise and are more aligned across domains after SynCDR training.

CUT [25] trains a generator that can translate an input image to visually resemble a different domain. It can be trained using images from the two visual domains without the need for paired illustrations, making it viable for our use-case. CUT relies on optimizing a combination of a GAN loss and patchwise contrastive losses where the latter reuses features from the encoder of the image generator (which is a composition of a convolutional encoder and decoder) and uses negative patches from within the input image, resulting in a simple efficient approach without many components.

For our experiments, we used two CUT generators from code provided by Park et al. [25] trained for translation in either direction on the unlabeled training images in each of the two domains. This is the relatively slower higher quality version of CUT compared to FastCUT, a faster variant that Park et al. [25] also provide. For training each generator, we used a batch size of 4, and trained for 400 epochs (where each epoch was clipped to a maximum of 250 steps). While after training, image generation is fast (90 ms compared to few seconds for diffusion sampling methods—as described next), the training itself is relatively slow, taking approximately 20 hrs on an NVIDIA Tesla V100 gpu.

SDEdit [24] was a method first proposed for editing images before the advent of open large scale pre-trained text-to-image diffusion models. With a pre-trained denoising diffusion model the approach is straightforward, following the diffusion noising process on an input image, and then subsequently denoising it. What generates the edit is following the noising process partially based on an edit strength $\in[0,1]$, such that not all information in the input image is lost. At the extremes, the value 0 for this parameter leads to no edit, and the value 1 leads to losing all input image information.

With a text-guided generative model like Stable Diffusion [32], this approach readily facilitates text-guided image editing using prompts. For our experiments, we use the implementation of StableDiffusionImg2ImgPipeline in [27] with the pre-trained model runwayml/stable-diffusion-v1-5. We used 50 diffusion steps, a guidance scale of 10 (these values being somewhat standard for Stable Diffusion image generation) and chose the final edit strength as 0.3, after computing validation accuracy for SynCDR using data for 3 different values : 0.3, 0.5 and 0.7. The prompts we used for translating to each domain are mentioned in Tab. 12. Since edit strength determines number of diffusion steps, the speed of the process depends on the edit strength parameter. For 0.3, we generated synthetic data at $\sim$2.8 seconds per image using an Nvidia Tesla V100 GPU, resulting in a total generation time of 6.7 hrs for our largest training dataset (painting domain of domainnet with $\sim$8600 images).

In Fig. 5, we show examples with different edit strength values. We notice in that example, that a higher strength can make a generated image look more like a painting, but at the cost of typically not retaining the input image category.

InstructPix2Pix [2] is a denoising diffusion model that can be conditioned on both an image and a text-prompt trained to carry out an edit to the input image, specified by the prompt. To control the amount of edit, we can use a parameter $image\_guidance\_scale\geq 1$, larger values of which lead to the output image looking more like the input image.

For our experiments, we used StableDiffusionInstructPix2PixPipeline from [27], with the pre-trained model provided by [2] in timbrooks/instruct-pix2pix. We used 50 diffusion steps and a guidance scale of 10, and for setting the $image\_guidance\_scale$ we used validation accuracy across 4 different values of generation : [1, 1.2, 1.4, 1.6], finally choosing 1.2. The text prompts used to provide the edit instructions for each domain are in Tab. 12. Generation speed for this method is relatively slower than Img2Img at around 9 seconds per image resulting in a total generation time of 21 hrs on an NVIDIA Tesla V100 GPU for our largest training dataset (painting domain of domainnet with $\sim$8600 images).

Fig. 6 shows examples of InstructPix2Pix edits made with different values of $image\_guidance\_scale$ to convert a realistic photo of a bird to a painting. We notice that InstructPix2Pix does a decent job of editing across the different scales, however on occasion a smaller guidance can lead to large edits which may not preserve the bird category (for e.g. $image\_guidance\_scale=1$ in the second row).

ELITE [47] is a method for fast personalized image generation from Stable Diffusion. Its goal is to learn an object from a single image and generate it in different contexts and styles based on a text prompt. This, ELITE does by encoding an input image into a token embedding that encodes the object into the dictionary of Stable Diffusion’s text encoder. Here we note that both ELITE and Textual Inversion [9] are personalized generation methods which can learn concepts from one or a few input images in the form of a new token embedding. While textual inversion does this via optimization for each new object/concept, which is slower, ELITE can do the same by using a trained encoder module which generates a new concept token embedding in one forward pass.

The full ELITE method uses both a global and a local image encoding module, where the latter is useful for better preservation of details from the input object (interested readers may see Figure 6 of [47]). In our experiments, we utilize only the global module since the latter additionally requires a segmentation mask around the primary object in the input image. For generation, we used the implementation of [47], which uses an image encoder based on a CLIP pre-trained ViT-L model from [30], and Stable Diffusion v1.4 for generation. The different prompts we used for generation are listed in Tab. 12. In these prompts, “S” corresponds to the special token that is output by ELITE’s image encoder representing the input object. Generation speed was about the same as InstructPix2Pix at around 9s per image on an NVIDIA Tesla V100 gpu resulting in 21 hrs total generation time for domainnet paintings domain (with $\sim$8600 images).