DiffFashion: Reference-based Fashion Design with Structure-aware Transfer by Diffusion Models

Fashion design models aim to design new clothing from a given clothing collection. Sbai et al. [3] use GAN to learn the encoding of clothes, and then use the latent vector to perform the stylistic transformation. Cui et al. [8] use the sketch image of the clothes to control the generated structure. Good results have been achieved in terms of structural control. Yan et al. [2] use a patch-based structure to implement texture transfer on generated objects. However, they cannot use other images as texture references and their tasks is limited to generating new samples from existing clothes collections. As a result, due to the unreliable training problem of GAN, more advanced methods are needed to achieve improved realism in generated effects.

The image-to-image translation aims to learn the mapping between the source and the target domains, often using a GAN network. Paired data methods like [19, 20] use the target image corresponding to each input for the condition in the discriminator. Unpaired data methods like [21, 22, 23] decouple the common content space and the specific style space in an unsupervised way. But both these methods require amounts of data from both domains. Besides, the encoding structure of GANs makes it difficult to decouple appearance and structural information. When the gap between the two domains is too large, the result may not be transformed [23, 24, 25] or have lost information from the original domain [26].

Recently, denoising diffusion probabilistic models (DDPMs) have emerged as a promising alternative to GANs in image-to-image translation tasks. Palette [18] firstly applies the diffusion model in image translation and achieves good results in colorization, inpainting, and other tasks. However, this approach requires the target image as a condition for diffusion, making it infeasible for unsupervised tasks. For appearance transfer, DiffuseIT [15] uses the same DINO-ViT guidance as [16], which greatly improves the realism of the transformation. However, it still cannot solve the problem of lacking matching objects in the clothing design task.

Neural style transfer (NST) has shown great success in transferring artistic styles. There are mainly two types of approaches to modeling the style or visual texture in NST. One is based on statistical methods [27, 28], in which the style is characterized as a set of spatial summary statistics. The other is based on non-parametric methods, such as using Markov Random Field [29, 30], in which they swap the content neural patches with the most similar ones to transfer the style. After texture modeling, a pre-trained convolutional neural network (CNN) network is used to complete the style transfer. Although NST-based methods work well for global artistic style transfer, their content/style decoupling process is not suitable for fashion design. In addition, NST-based methods assume the transfer is between similar objects or domains. Tumanyan et al. [16] propose a new NST loss from DINO-ViT, which succeeds in transferring appearance between two semantically related objects, such as “cat and dog” or “orange and ball”. However, in our task, there are no specific related objects between the clothing image and the appearance image.

Given a clothing image $x^{S}_{0}$ and a reference appearance image $x^{A}_{0}$, our proposed DiffFashion aims to design a new clothing fashion that preserves the structure in $x^{S}_{0}$ and transfers the appearance from $x^{A}_{0}$ while keeping it natural, as shown in Fig. 2. We list two main challenges in this task. First, there are no given reference images for the output result since there is no standard answer for fashion design. Without the supervision of the ground truth, it is difficult to train the model. Second, preserving the structure information from the given input clothing image while transferring the appearance is also being under-explored. To address those two challenges, we present the DiffFashon, which is a novel structure-aware transfer model with the diffusion model. We use the diffusion model [32] pre-trained on Imagenet [33] for all the denoising processes in DiffFashion. First, we decouple the foreground clothing with a generated semantic mask by conditioned labels, as shown in Fig. 2 (a). Then, we encode the appearance image $x^{A}_{0}$ with DDPM, and denoise it with mask guidance to preserve the structure information, as shown in Fig. 2 (b). Moreover, we use the DINO-ViT [17] for both appearance and structure guidance during the denoising process, as shown in Fig. 2 (c) and (d). The details are illustrated in the following sections.

To decouple the foreground clothing and background, we generate a semantic mask for the input clothing image $x^{S}_{0}$ with label conditions. The generated semantic mask is also used for preserving the structure information in later steps. Existing methods commonly use additional inputs to obtain the foreground region. However, this leads to increased annotation expenses. Inspired by [34], we propose a mask generation approach that can obtain the foreground clothing area without external information or segmentation models. Our approach leverages the label-conditional diffusion model to obtain the desired result.

In the denoising process of the label-conditional diffusion model, there can be different noise estimates for the same latent given negative label conditions like phone and bag. For these different noise estimates, the regions of the foreground object that are denoised tend to vary little in background regions but greatly in object regions. By taking the difference in the noise area, we can obtain the mask of the object to be edited, as shown in Fig. 2(a).

Instead of generating a mask with the latent of the forward process like [34], we observe that in the denoising process, $x^{S}_{t}$ has less perceptual appearance information than $x^{S}_{qt}$ (the image in the forward process with timestamp $t$). Therefore, we generate a mask from the image in the denoising process $x^{S}_{t}$ instead of the image $x^{S}_{qt}$ in the forward process. Although the structure of $x^{S}_{t}$ may have some slight variations, it still provides a better representation of the overall structure information of the foreground object.

Specifically, we input the clothing image $x^{S}_{0}$ into the diffusion model. After DDPM encoding in the forward process, we obtain the image latent $X^{S}_{T/2}$ in half of the reverse process. Denote the foreground label as $y_{p}$, representing the foreground clothing object. Then the noise map for the foreground clothing can be obtained by

where $\hat{x}^{S}_{T/2}$ is the estimated source image predicted from $x^{S}_{T/2}$ by Tweedie’s method [35], i.e.,

Denote non-foreground labels as $y_{n}$, representing negative objects. We use $N$ different non-foreground label conditions to get an averaged noise map, i.e.,

where $i\in\{1,...,N\}$. The difference between the two noise maps $M_{p}$ and $M_{n}$ can be obtained. Then we set a threshold for binarization, which returns an editable semantic mask $M$ for the foreground clothing region.

It is difficult to transfer the appearance of the original image to a new fashion clothing image when the gap between the two domains is too large [16]. Because such methods control the appearance by a single loss of guidance, the redundant appearance information of the structure clothing reference image cannot be completely eliminated. Besides, when using a natural non-clothing image for appearance reference, the generated texture maybe not be suitable for clothing. Because these models only transfer the style or appearance. The appearance cannot be converted to a suitable texture material like cotton for clothing. In DiffFashion, to address this problem, rather than transferring from the input clothing image $x^{S}_{0}$, we transfer from the reference appearance image $x_{0}^{A}$ to the output fashion clothing image with the guidance of the structural information of the input clothing image.

Inspired by [36], it has been shown that for the same DDPM encoding latent with different label conditions used for denoising, the resulting natural images have similar textures and semantic structures. We use the latent $x^{A}_{t}$ of the reference appearance image to transfer more appearance information to the output fashion. Besides, the texture of the appearance image can be transferred more realistic and suitable for clothing in the denoising process. Meanwhile, the semantic mask $M$ obtained from the previous step is used to preserve the structure of the clothing image. As shown in Fig. 2(b), the appearance image $x^{A}_{0}$ is first used to encode by the forward process of DDPM. Then the mask-guided denoising process is employed.

Specifically, at each step in the denoising process, we estimate the new prediction $x^{A}_{t}$ from the diffusion model as follows,

Then we combine the transferred foreground appearance $x^{A}_{t}$ and the clothing image of corresponding timestamp $x^{S}_{qt}$ with the generated mask $M$ as guidance, i.e.,

where $\omega_{mix}$ is the mix ratio of the appearance image and the clothing image. This change ensures that the appearance information in the mask is transferred, while other structural information keeps consistent with the clothing image.

As mentioned in [16, 15], the structure features and appearance features can be separated with DINO-ViT [17]. We use both appearance guidance and structure guidance in the denoising process to keep the output image realistic.

Following [16, 15], we employ the $[CLS]$ tokens in the last layer of ViT to guide the semantic appearance information as follows,

where $e_{[CLS]}^{L}$ is the last layer $[CLS]$ token, and $\lambda_{MSE}$ is the coefficient of global statistic loss between images. To better leverage the appearance between the object and the appearance image, we use the object semantic mask $M$ to remove the background pixel of $\hat{x}^{A}_{t}$ in Eq. LABEL:eq12, and only compute the appearance loss of the object within the mask.

In addition, we adopt a patch-wise method in the structural loss to better leverage the local features. We adopt the $i$-th key vector in the $l$-th attention layer of the ViT model, denoted as $k_{i}^{l}(x_{t})$, to guide the structural information of the $i$-th patch of the original clothing image as follows,

where $\mathrm{sim}(\cdot,\cdot)$ is the exponential value of normalized cosine similarity, i.e.,

and $\tau$ is the temperature parameter. By using the loss in Eq. (LABEL:eq11), we minimize the loss between keys at the same position of two images while maximizing the loss between keys of different positions. Then our total loss for guidance as follow:

where $\lambda_{struct},\lambda_{app}$ are the coefficient of structure loss and appearance loss.

To our best knowledge, there is no specific reference-based fashion design dataset currently. Thus, we collect a new dataset, namely OceanBag, with real handbag images and ocean animal images as reference appearances for generating new fashion designs. OceanBag has 6,000 photos of handbags in various scenes and 2,400 pictures of various marine lives in the real world, among various marine scenes such as fish swimming on the ocean floor. The 2,400 marine scene images contain more than 80 kinds of marine organisms, 50% of which are fish, as well as starfish, crabs, algae, and other sea creatures, as shown in Fig. 3. In our experiments, we screened 30 images for experiments based on diversity such as background complexity, species, and quantity of organisms.

We refer to images with solid backgrounds as simple backgrounds, while those with real scenes are referred to as complex backgrounds. The complex ratio in Table I shows the proportion of complex background images in the dataset. The complex background of the marine biological dataset is usually real ocean pictures such as the seabed and the deep sea. For the bag images in the dataset, the complex backgrounds often include scenes of mall containers or tables.

We conduct all experiments using a label-conditional diffusion model [32] pre-trained on the ImageNet dataset [33] with $256\times 256$ resolution. In all experiments, we use a diffusion step of $T$ = 60 and re-sampling repetitions of $N$ = 10. In a single RTX 3090 unit, it takes 20 seconds to generate each mask and 120 seconds to generate each image. For fairness of comparison, other parameters in the diffusion model are kept the same as [15].

In the mask generation part, we set the binarization threshold to -0.2. Due to the stochastic nature of the diffusion model, we generate masks using three different sets of labels, including “cellphone, forklift, pillow”, “waffle iron, washer, guinea pig” and “brambling, echidna, custard apple”. Then we choose the best one among them for guidance. To ensure a fair comparison, We run the baseline DiffuseIT [15] three times as ours.

In the guidance part, to mitigate the uncontrollable effect of the mask and avoid information loss when the structural gap between the two objects is too large, we use mask guidance in the first 50% steps of the denoising stage, and the mix ratio $\omega_{mix}$ is set to 0.98. In the ViT guidance part, we set the coefficient of appearance loss $\lambda_{app}$ to 0.1 and 1 for structure loss $\lambda_{struct}$. And we keep other parameters the same as DiffuseIT [15].

There is currently no existing automatic metric suitable for evaluating fashion design across two natural images. To keep the fashion image realistic, the migration degree of the appearance and the similarity of the structure sometimes are mutually contradictory when measured. To compare among different methods, we follow existing appearance transfer/fashion design works [16, 15, 37, 38, 39, 40], which rely on human perceptual evaluation to validate the results.

We perform both quantitative and qualitative evaluations on the OceanBag dataset. We compare our model with SplicingViT [16], DiffuseIT [15] , WCT2 [41] and STROTSS [42]. Fig. 4 shows qualitative results for all methods. In all examples, it can be seen that in terms of fashion design, our method has achieved better performances in terms of realism and structure, while completing appearance transfer. As for the DINO-ViT-based image-to-image translation methods, DiffuseIT successfully keeps the structure for most images, but it shows less appearance similarity. SplicingViT transfers the appearance well, but its results are far away from realistic fashion images. NST methods like STROTSS and WCT2 effectively retain the structure of the source image, but WCT2 outputs exhibit limited changes apart from color adjustments. Although STROTSS successfully transfers the appearance, its results often suffer from color bleeding artifacts and thus show less authenticity.

We also conduct a user study to evaluate the samples and obtain subjective evaluations from participants. Specifically, we ask 30 users to score all the output fashion images from all methods for each input pair. Detailed questions we have asked are as follows: 1) Is the picture realistic? 2) Is the image’s structure similar to the input image? 3) Is the output appearance similar to the input appearance image? The scores range from 0 to 100. The overall score is the average of the three scores. We show the averaged subjective evaluation results in Table II. Our model obtains the best score in the overall performance and appearance correlation, and the second place in structure similarity and realism. WCT2 shows the best in realism and structure similarity scores, but it shows the worst score in appearance correlation because the outputs are almost unchanged from the inputs except for the overall color. Both the qualitative and subjective evaluations show the effectiveness of our proposed method.

Following [16], we also adopt other pre-trained models to evaluate the result. We use the classifier pre-trained with the ImageNet dataset given by improved DDPM [32] and calculate the average classification loss. We also apply Mask-RCNN pre-trained on the COCO dataset to detect the mask of the object of each method. The results are shown in Table III. Our model achieves the lowest classification loss. At the same time, since Mask-RCNN is trained on out-of-distribution (OOD) data, the overall recall rate is quite low. Our model demonstrates the second-best performance after WCT2, but WCT2 only transforms the color for the whole image. Besides, we calculate the color difference histogram (CDH) [43] between the result and appearance image for each method. Our method achieves better appearance similarity than image translation methods. Although NST methods like STROTSS have a better CDH, they tend to transfer the whole image with simple color transformation, as shown in Fig. 4.

In order to verify the effectiveness of the method, we study the individual components of our technical designs through several ablation studies as illustrated from Fig. 5 to Fig. 9.