Self-Supervised Vision Transformer for Enhanced Virtual Clothes Try-On

Virtual try-on technology has increasingly become a focal point in research, driven by its potential to revolutionize the consumer shopping experience. The domain encompasses two primary methodologies: 2D and 3D approaches, as outlined in previous studies [13]. The 3D virtual try-on [7, 39, 16, 21], offering an immersive user experience, requires sophisticated 3D parametric human models and extensive 3D datasets, often entailing significant costs. Conversely, 2D virtual try-on methods are more accessible and prevalent in practical applications due to their lighter computational demands. Prior research in 2D virtual try-on [12, 23, 34], has concentrated on adapting clothing to fit human figures flexibly, yet it faces challenges in managing substantial geometric deformations. Furthermore, flow-based methods [11, 9, 13] have been explored to depict the appearance flow field between clothing and the human body, aiming to enhance the fit of garments. While these methods offer significant advancements, they frequently fall short in generating high-quality, high-resolution imagery, particularly when it comes to replicating intricate details of clothing.

Moreover, with increasing resolution, synthesis stages based on Generative Adversarial Networks (GANs) [17, 6, 20, 18] encounter difficulties in preserving the integrity and characteristics of clothing, resulting in reduced realism and an escalation of artifacts. The generative capabilities of GANs, particularly in conjunction with high-resolution clothing warping, are constrained, impacting the overall quality of the synthesized images. These challenges underscore the need for more sophisticated approaches capable of surpassing the limitations inherent in current virtual try-on methods. Employing diffusion models emerges as a promising solution, offering significant improvements in virtual try-on performance. Specifically, diffusion models excel in managing complex clothing attributes and achieving realistic image synthesis, addressing the critical shortcomings of existing technologies.

In recent years, Diffusion Models [15, 32] have been proposed to generate realistic images by reversing a gradual noising process. And latent diffusion models [29], trained on latent space representations, incorporate cross-attention layers within their network structure to handle generic conditions as input. This advancement has established diffusion models as formidable rivals to GANs in image synthesis tasks. Recent studies have also delved into using text information as a condition in the denoising process, enabling diffusion models to create images with text-relevant features [28, 33, 10, 19, 4].

LaDI-VTON [24] introduces a novel textual inversion component, mapping the visual attributes of in-shop garments to CLIP token embeddings to generate pseudo-word token embeddings, conditioning the generation process and preserving detailed textures. Furthermore, the development of spatial-level guidance techniques in diffusion models [22, 5], particularly through targeted interventions in the denoising phase, has significantly enhanced their applicability across various domains. Despite these advancements, effective implementation of virtual try-on using diffusion models remains a challenge. Text-to-image methods [28, 31] alone are insufficient for accurately depicting the varied appearances of clothing. Similarly, inpainting techniques [36] struggle with maintaining precise control over finer details.

In response to these limitations, our approach introduces a novel self-supervised component specifically designed to enhance detail accuracy and realism in virtual clothing try-ons, focusing on the extraction of critical keypoints.

In this research, our objective is to harness diffusion models within an inpainting framework for virtual try-on tasks, focusing on intricacies of clothing such as sleeves, collars, and textual patterns. Previous methodologies have explored various approaches for injecting explicit information, yet they often overlook these critical clothing details. To address this, we introduce a self-supervised learning-based detail enhancer, designed to aid our network in better recognizing and integrating these essential features. As depicted in Figure 1, our methodology involves processing a person image $I_{o}\in\mathbb{R}^{H\times W\times 3}$ and a clothing image $I_{c}\in\mathbb{R}^{H^{\prime}\times W^{\prime}\times 3}$ to generate a synthesized image $\hat{I}\in\mathbb{R}^{H\times W\times 3}$. This image aims to realistically blend the person’s attributes from $I_{o}$ with the clothing elements of $I_{c}$. For try-on scenarios, we utilize a binary mask $m\in{0,1}^{H\times W}$ to identify the areas needing inpainting, generally covering the upper body and arms of the person. In the resulting image $\hat{I}$, areas where $m=0$ should faithfully reflect $I_{p}$, and those with $m=1$ should seamlessly incorporate elements from $I_{c}$, ensuring a natural integration with the person’s figure.

The clothing image $I_{c}$ plays a pivotal role in guiding the model to generate try-on images. It is input into a Vision Transformer (ViT) [8] to produce conditional embeddings $c$ that direct the diffusion models. To improve the accuracy of detail replication, we fine-tune ViT as a condition encoder $\tau$ using our specialized clothing dataset. The fine-tuning process adopts distillation approaches, involving the collaboration of teacher and student networks. Throughout this phase, the teacher model processes the entire global crop $I_{c}$, whereas the student model processes both $I_{c}$ and its locally cropped segments. Such an arrangement allows the student model to effectively assimilate knowledge from the teacher, thereby promoting self-supervised learning within ViT. Post fine-tuning, the ViT teacher model, now adept at capturing details, is employed as a detail injector in our stable diffusion inpainting network. It inputs complete clothing images to ensure enhanced precision in the final output.

To facilitate improved interaction with ViT features, our model’s input incorporates a warped image. During this warping process, the clothing image $I_{c}$ is input into a warping network, designed to predict an appearance flow field that aligns the clothing accurately. The outcome is a warped clothing image, which, when combined with a clothes-agnostic person image and a mask, yields a masked coarse result $I_{lc}$. The warping process in our framework adopts an iterative refinement strategy, adept at capturing long-range correspondences. This approach is particularly effective in addressing significant misalignments in clothing images. For a more comprehensive understanding of the warping techniques employed, please refer to [9, 11].

Subsequently, we will explore the intricacies of the diffusion model, our method for self-supervised ViT fine-tuning, and the content correction loss mechanism that is specifically developed based on ViT insights.

Existing studies [36, 24] have demonstrated the considerable capabilities of diffusion models in try-on tasks, which is why our work also adopts a diffusion model, specifically leveraging the open-source StableDiffusion (SD) [17] framework. The SD Network excels in generating realistic images by leveraging the reverse diffusion process. Starting with the target image $I_{0}$, an initial forward diffusion process denoted as $q(\cdot)$ is applied. This process incrementally introduces noise following a Markov chain, ultimately transforming the image into a Gaussian distribution. To optimize computational efficiency, a latent diffusion model is employed. This involves transforming images from the image space to the latent space using a pre-trained encoder $E$, followed by image reconstruction with a pre-trained decoder $D$. The forward diffusion process on the latent variable $z_{0}=E(I_{0})$ at any given timestamp $t$ is defined as:

$$z_{t}=\sqrt{\alpha_{t}}z_{0}+\sqrt{1-\alpha_{t}}\epsilon,$$

(1)

where $\alpha:=\prod_{s=1}^{t}(1-\beta_{s})$ and $\epsilon\sim N(0,I)$. Here, $\beta$ denotes a predefined variance schedule spanning $T$ steps. The latent code $z_{lc}$ is derived by passing $I_{lc}$ through the encoder $E$, which is then concatenated with the downsampled mask $m$ to form the input ${z_{t},z_{lc},m}$. For the denoising phase, an enhanced Diffusion UNet is utilized to predict a refined version of the input. Within this framework, the condition $c$ extracted from $I_{c}$, using a condition encoder $\tau$ that $c=\tau(I_{c})$, is incorporated into the Diffusion UNet through a cross-attention mechanism. The objective function of the UNet is defined as:

$$\mathcal{L}_{\text{SD}}=||\epsilon-\epsilon_{\theta}(z_{t},z_{lc},m,c,t)||_{2}^{2}$$

(2)

The condition $c$ is critical in the SD framework for inpainting tasks, as it provides essential clothing information for the process. Therefore, the selection and training of the condition encoder $\tau$ are of utmost importance. The Paint-by-Example approach [36] suggests using the CLIP [27] image ViT encoder to extract this condition. However, our findings indicate that this is not the most effective strategy, given that the CLIP encoder is pre-trained on open-world images rather than clothing-specific images. Consequently, we propose fine-tuning the ViT encoder $\tau$ to yield a more accurate condition, tailored for clothing images.

Paint by Example utilizes CLIP to enhance detailed information, yet CLIP is inclined towards classification, leading to significant inter-class differences but minimal intra-class variation. This results in a lack of distinctiveness for items within the same category. Consequently, extracting effective information from clothing datasets becomes crucial for producing optimal conditions for SD to generate superior images. However, this is challenging due to the scarcity of annotations or other supervised methods beneficial for virtual try-on in clothing datasets. Inspired by [3], which utilizes self-supervised learning for visual representation, we implement a similar approach. The core of this training methodology lies in contrasting representations from different perspectives of the same data instance.

This concept is operationalized through a dual training setup comprising teacher and student networks. The teacher network $\tau$ produces pseudo-labels for the student network $\tau^{\prime}$ by analyzing representations from augmented views of the input data. Notably, $\tau$ is maintained as the moving average of $\tau^{\prime}$, ensuring more stable and consistent feature extraction. In parallel, the student network $\tau^{\prime}$ is trained to conform to these pseudo-labels, striving to minimize similarity across representations from various instances. This strategy facilitates the extraction of semantically rich features from clothing data, significantly enhancing the conditioning quality for SD. The loss function of the finetuning process can be expressed as follows:

$$\mathcal{L}_{\text{SS}}=\sum_{i=1}^{M}\sum_{\begin{subarray}{c}j=1\ j\neq i\end{subarray}}^{N+M}H(\tau(I_{c_{i}}),\tau^{\prime}(I_{c_{j}})),$$

(3)

where $M$ and $N$ represent the total number of global and local crops, respectively. The functions $\tau(\cdot)$ and $\tau^{\prime}(\cdot)$ denote the ViT feature extraction process conducted by the teacher and student networks, respectively. The term $H(a,b)=-a\log b$ signifies information entropy. $I_{c_{j}}$ refers to the augmented view of the input data instance $I_{c}$. The augmentation process begins with a random resized crop operation, applying a scale factor and bicubic interpolation. This is followed by flip and color jitter transformations. Additionally, the process incorporates Gaussian blur and normalization for enhanced diversity in the data representation. The scales for the local crops are specified in the range $[0.05,0.25]$ of the whole image, while those for the global crops fall within the range $[0.25,1]$.

The teacher network is utilized as the final condition encoder in our framework. In Figure 2, we present visualizations of the average head attention corresponding to the class token in ViT, both before and after self-supervised fine-tuning. In these visualizations, “SS-” represents the state without fine-tuning, while “SS+RF” indicates the application of self-supervised fine-tuning. A comparative analysis between “SS-” and “SS+RF” demonstrates that self-supervised fine-tuning on our dataset results in heightened attention to specific key areas of the clothing. Overall, this self-supervised approach empowers ViT to focus distinctively on each feature of the clothing. After undergoing self-supervised fine-tuning, ViT shows increased attention to particular local areas within the clothing images, reflecting an enhanced understanding of clothing features and details.

While we implement a self-supervised approach to ViT, we find that using a pre-trained ViT directly offers limited adaptability to our dataset and inadequate capability to capture clothing-specific features. During the initial stages of the fine-tuning process, local crops are randomly sampled. However, this approach often overlooks vital keypoints of the clothing, resulting in the omission of important details. Given our objective for ViT to accurately identify key elements of clothing, such as collars, sleeves, text, and patterns, we intentionally fine-tune ViT on our dataset to enhance its attentiveness to these critical aspects. To overcome the shortcomings of random sampling, we refine our methodology to selectively sample local crops that encompass these essential keypoints. This adjustment ensures that the network is furnished with the detailed information, enabling a more precise representation of clothing.

In the self-attention maps of the ViT, we observe that different heads focus on distinct segments of clothing, closely aligning with key areas. Figure 3 showcases this, where subfigures (b) and (c) accurately target text and sleeves on the clothing. Consequently, we compile all points with attention surpassing a specified threshold, referred to as high-attention points, from each head’s attention and merge them into a singular map, as depicted in subfigure (d). We then cluster these points based on their proximity, ensuring the number of clusters corresponds to the quantity of local crops. As revealed in Figure 3 (e), the centroids of these clusters effectively encompass vital garment areas like sleeves, collars, hems, and patterns. These identified centroid points are then utilized as the basis for generating new local crops. Following this, we apply a range of data augmentation techniques to derive the final local crops, which are integral to the fine-tuning of ViT on clothes dataset. After fine-tuning, the stabilized ViT model is incorporated into the training of SD inpainting network, enhancing its effectiveness.

Specifically, we identify 10 centroids in the clustering process as the keypoints for each clothing item. The local crops are then centered around these keypoints, maintaining a $[0.05,0.25]$ ratio to the entire image. In Figure 2, ’SS+SF’ denotes the application of our method in selecting local crops for self-supervised fine-tuning. Comparing “SS-” and “SS+RF”, it is apparent that our dataset’s self-supervised fine-tuning leads to increased attention on specific key areas. However, ’SS+SF’ demonstrates a more pronounced focus on crucial points, text, patterns, and other intricate details on the clothing. This observation confirms that our self-supervised approach enables ViT to more effectively extract and emphasize the unique characteristics of clothing compared to previous methods.

Our experimental evaluations primarily utilize the VITON-HD dataset [6], comprising 13,679 pairs of frontal-view women and top clothing images. Following precedent [6, 17], we partition the dataset into training and test sets, encompassing 11,647 and 2,032 pairs, respectively. Experiments are conducted at $512\times 384$. Furthermore, to validate the robustness of our method in more intricate scenarios, we extend our experiments to include the the DressCode dataset [25]. Detailed results of these extended evaluations will be provided in the supplementary material.

To evaluate the effectiveness of our method in different test scenarios, we employ a range of metrics. In the paired setting, where a clothing image is used to reconstruct a person’s image, we use two widely accepted metrics: Structural Similarity (SSIM) [35] and Learned Perceptual Image Patch Similarity (LPIPS) [38]. For the unpaired setting, involving the alteration of the person’s clothing, we measure the performance using Frechet Inception Distance FID [14] and Kernel Inception Distance KID [1]. To capture the complex aspects of human perception, we have incorporated a user study into our evaluation process. Specifically, we have created composite images using various methods for 200 randomly selected pairs from our test set, each at a resolution of $512\times 384$. These images were then evaluated by a panel of 30 human judges who were tasked with identifying the method that most effectively restored clothing details and produced the most lifelike results for each pair. The frequency of each method being chosen as the best performer in these two aspects is thoroughly documented, providing a detailed insight into our method’s performance.

The training of our network is systematically divided into two distinct stages. In the initial stage, we leverage local crops derived from attention maps to train the ViT model. We use 10 local crops and opt for the VIT-base model with a patch size of 16. The training commences with an initial learning rate of 2e-5, and a batch size is set at 32. This model undergoes training for more than 30 epochs on our dataset. In the second stage, we employ the pre-trained ViT model with its parameters fixed and focus on training the entire network. For this phase, the learning rate is adjusted to 1e-5, and the batch size is reduced to 8. This stage of training extends over 40 epochs, allowing for comprehensive learning and adaptation. Overall, the training process requires only 6 hours to finetune ViT and 24 hours to finetune SD, which is significantly faster compared to full-scale SD training. Besides, the testing is conducted on a single 3090 GPU. And we adopt PLMS Sampling for inference with 100 steps. It is crucial to highlight that our model does not incur any additional inference time to SD. The average generation time per image is 6.31 seconds, which is roughly equivalent to that of PbE with 6.28 seconds.

Our target baselines are CPVTON [34], PF-AFN [9], VITON-HD [6] and HR-VTON [17], and diffusion methods Paint-by-Example (PbE) [36] and LaDI-VTON [24]. The comparison results on VITON-HD dataset are shown in Table 1. The performance of traditional methods such as CP-VTON, VITON-HD and PF-AFN is relatively inferior, while the PbE method demonstrates acceptable results after finetuning on the dataset. Furthermore, the HR-VITON method, which performs warp operations on high-resolution images, exhibits favorable outcomes. Similarly, LaDI-VTON, which utilizes textural inversion to map images to image clip embeddings, also yields promising results. While our method achieves superior performance, attributed to the effective utilization of self-supervised learning techniques, which enables the network to focus on details and key points, thereby addressing the shortcomings of previous models.

We further compare our method against competitive approaches, namely PF-AFN [9], HR-VTON [17], and LaDI-VTON [24] on the DressCode dataset [25]. Table 2 shows that our method achieves the best results across different categories, including upper, lower, and dresses, in terms of similarity and realism metrics. These qualitative results demonstrate the effectiveness of our method on a diverse range of datasets, highlighting its strong transferability and ability to adapt to various clothing categories.

In Figure 4, we present qualitative comparisons of our proposed method with other state-of-the-art methods on VITON-HD dataset. The analysis reveals distinct differences in performance. CP-VTON, for instance, tends to produce results that appear more artificial and less authentic in terms of integration. VTON-HD, on the other hand, falls short in capturing textural details. Compared to these methods, PF-AFN and HR-VTON come closer to replicating the original images but still exhibit deficiencies in several detail aspects. Paint-by-Example (PbE) shows some ability to learn similar features but struggles with capturing variations between individuals. LaDI-VTON, while showing promise, faces challenges similar to PbE.

Our method stands out in handling numerous details more effectively. Particular attention has been given to key areas such as collars and hems, as evidenced in the first and third rows of the figure. Additionally, our approach more accurately reflects the overall texture and material of the original images, as seen in the second and fourth rows, and demonstrates a slight superiority in handling various intricate details, as shown in the fifth and sixth rows. These successes are largely due to our fine-tuning process, which has enhanced the network’s ability to concentrate on crucial details at different positions, thereby improving overall precision. More visual results are shown in figure 8.

In order to compare the visual results of our method on DressCode Dataset with the highest-scoring baseline LaDI-VTON [24] in terms of evaluation metrics, we present a visual comparison in Figure7. This comparison allows for a direct assessment of the performance of our method against the state-of-the-art baseline. The results obtained from our proposed method represent a significant breakthrough in the field of clothing generation. Specifically, we have demonstrated the ability to achieve an exceptional level of fidelity, encompassing various intricate texture patterns and other essential features, that faithfully replicates the target garments. Notably, our approach also exhibits a heightened realism in terms of the overall fit and texture of the generated clothing, which further accentuates the superiority of our method over existing approaches.

In our ablation studies, we assess the impact of each individual enhancement on the overall performance. These experiments utilize images of size $512\times 384$. The CLIP encoder has a patch size of 14 and an image embedding size of 1024, while ViT has a patch size of 16 and an image embedding of 768. To ensure fairness, we maintain consistency by mapping the embedding of ViT to 1024 dimensions using a linear layer. We compare the following configurations: (a) CLIP: Utilizing CLIP as the encoder for conditioning. (b) SS-Encoder: Employing ViT for conditioning through self-supervised learning, without any fine-tuning. (c) SS+RF Encoder: Conditioning with ViT post fine-tuning using random local crops. (d) SS+SF Encoder: Conditioning with ViT after fine-tuning with selected local crops based on self-attention maps. Lastly, (e) the Final Network: Our complete network, integrating warping to interact with the features.

As shown in Table 3, it is evident that ViT, when subjected to self-supervised training, demonstrates notably superior improvement in comparison to CLIP. Notably, the application of fine-tuning with random local crop on our dataset fails to yield substantial improvements. However, our proposed fine-tuning method encompasses pivotal considerations, facilitating a more comprehensive capture of intricate clothing features and consequently leading to further enhancements in the experimental outcomes. Lastly, our comprehensive network analysis reveals that the incorporation of warping for interactions with ViT features highlights an inadequate focus of UNet features on ViT. Subsequent to the introduction of warp features, UNet features exhibit a notable improvement in effectively attending to the corresponding ViT feature regions, thereby significantly elevating the overall quality of the results.

As shown in Figure 6, the results clearly indicate that ViT, when subjected to self-supervised training, exhibits a significantly superior improvement compared to CLIP. It is worth noting that our proposed fine-tuning method takes critical factors into account, allowing for a more comprehensive capture of intricate clothing features, consequently leading to further enhancements in the experimental outcomes. Moreover, our comprehensive network analysis reveals that the incorporation of warping for interactions with ViT features highlights an insufficient focus on UNet features on ViT. Following the introduction of warp features, UNet features demonstrate a noticeable improvement in effectively attending to the corresponding ViT feature regions, thereby significantly elevating the overall quality of the results.