Shape Controllable Virtual Try-on for Underwear Models

Virtual try-on systems have been an attractive topic even before the renaissance of deep learning.

3D virtual try-on systems based on human reconstruction in computer graphics have been widely researched, such as (Alldieck et al., 2019; Bhatnagar et al., 2019; Li et al., 2010; Loper et al., 2015; Umetani et al., 2011). While Physics-Based Simulation (PBS) can accurately drape a 3D garment on a 3D body (Gundogdu et al., 2019; Patel et al., 2020; Santesteban et al., 2019; Vidaurre et al., 2020), it remains too costly for real-time applications. 3D human estimation methods try to digitalize real-world 2D character photos to 3D models, which have made significant progress after the popularity of deep learning, such as (Guler and Kokkinos, 2019; Kanazawa et al., 2018; Kolotouros et al., 2019; Xu et al., 2019), while (Habermann et al., 2020; Jiang et al., 2020; Saito et al., 2019; Saito et al., 2020) also digitizing with garments, these methods still need to be optimized in texture authenticity.

Recently 2D image-based virtual try-on tasks have gained much attention from academia and industry. It is highly challenging because it requires to warp the clothing on the target person while preserving its patterns and characteristics.

Han et al. (Han et al., 2018) proposed a 2D image-based virtual try-on task that adopts model and clothing pairs to train a clothing deformation sub-network and then composites the deformed clothing and coarse model with a generation network. Based on (Han et al., 2018), Wang et al. (Wang et al., 2018b) refined the architecture with a geometric matching module to learn parameters for Thin-Plate Splines (TPS) deformation (Wood, 2003). Considering the complexity of pose and self-occlusion cases, Yu et al. (Yu et al., 2019) and Raffiee et al. (Raffiee and Sollami, 2020) took parsing information into account and used a generator to predict the target model’s parsing result, which could generate better results in self-occlusion cases. What’s more, Wu et al. (Wu et al., 2019) adopted DensePose (Alp Güler et al., 2018) to get more precise information of the model. Minar et al. (Minar et al., 2020a) solved the problem by first mapping 2D clothing texture to 3D character model and deforming the clothing in 3D space, and then back to 2D space. Chaudhuri et al. (Chaudhuri et al., 2021) proposed a method to generate diverse high fidelity texture maps for 3D human meshes in a semi-supervised setup. On the other hand, some researchers focus on the improvement of the deformation module. In (Issenhuth et al., 2019, 2020; Jandial et al., 2020; Yang et al., 2020), the spatial transformer networks (Jaderberg et al., 2015) is adopted to get more accurate deformation result. Roy et al. (Roy et al., 2020) adopted human and fashion landmarks to help optimize thin-plate splines deformation model. Han et al. (Han et al., 2019) and Ren et al. (Ren et al., 2020) adopted optical flow to get a more natural texture deformation effect. Li et al. (Li et al., 2020) tried to get better results by using multiple specialized warps. Ge et al. (Ge et al., 2021) proposed a knowledge distillation method to produce images without human parsing.

Based on the generated high-resolution images of StyleGAN (Karras et al., 2019), Yildirim et al. (Yildirim et al., 2019) trained a conditional StyleGAN for generating virtual try-on images with clothing image as conditional input on around 380K entries. Cheng et al. (Cheng et al., 2020) embedded human geometries into the latent space as independent codes and achieved flexible and continuous control of geometries via mixing and interpolation operations in explicit style representations. However, acceptable visual results of StyleGAN usually requires hundreds of thousands of samples, which is very difficult to obtain.

Graph Convolutional Network (GCN) (Kipf and Welling, 2016) extends convolution operation from euclidean data (such as images) to a more extensive topological structure and is widely used in natural language processing, recommendation system, computer vision, and computer graphics domains. Wang et al. (Wang et al., 2018a) represented 3D meshes in a graph-based CNN and produced correct geometry by progressively deforming an ellipsoid. Based on (Wang et al., 2018a), Wen et al. (Wen et al., 2019) used GCN to solve the problem of shape generation in 3D mesh representation from a few color images with known camera poses. For the virtual try-on task, Vidaurre (Vidaurre et al., 2020) used GCN for parametric predefined 2D panels with arbitrary mesh topology. Kolotouros et al. (Kolotouros et al., 2019) adopted a Graph-CNN to estimate 3D human pose and shape from a single image. Zhu et al. (Zhu et al., 2020) used GCN to predict 3D clothing’s feature line and use it to warp clothing.

Graph Attention Networks (GATs) (Veličković et al., 2017) enable specifying different weights to different nodes in a neighborhood, leveraging masked self-attentional layers to address the shortcomings of prior methods based on graph convolutions. In this work, we consider using GAT to get more precise warping results.

The GAT based clothing deformation contains three parts: the shape information extraction module, the control points guided deformation module and the differentiable rendering module.

The shape information extraction module is devised for extracting features from the clothing image and the underwear model image. The extracted feature maps will be concatenated as shape vectors and sent to the deformation module as part of the guiding information. For the control points guided deformation module, the clothing firstly meshes with the Delaunay Triangulation Algorithm(Lee and Schachter, 1980), then the meshed coordinate points are set as vertices $V$ of a graph $G=(V,E)$, where $E$ denotes the edge set (See supplementary materials for more details). For each vertex in $V$, the vertex’s information is expressed as a vertex vector. Next, the shape control points and the shape vectors are integrated into the corresponding vertex vector as the deformation’s guiding information. GAT predicts coordinate offsets of all mesh vertices. The warped clothing image is obtained by using the differentiable rendering module. The whole deformation process is trained in an iterative manner to obtain higher accuracy. The following subsections give detailed descriptions.

In Section 3.1, large number of pseudo-labeled pairs $\{I_{m},I_{c},I_{g^{\prime}}\}$ are adopted to train SC-VTON, the pseudo-labeled pairs are generated with an ARAP tool, which limits the performance on real data.

To solve the above drawbacks, real pairs $\{I_{c},I_{g}\}$ of the clothing image $I_{c}$ and the model $I_{g}$ in the same clothing are adopted to optimize SC-VTON. We devise two sub-networks: a splitting network and a synthesis network, and add them into SC-VTON, which can generate self-loop supervision information for promoting robustness and performance of SC-VTON. The splitting network is devised for predicting underwear model $I^{\prime}_{m}$ and control points from the clothed model image $I_{g}$. The synthesis network is devised for synthesizing the clothed model image $I^{\prime}_{g}$ from the generated underwear model image $I^{\prime}_{m}$ and the warped clothing image $\tilde{I_{c}}$ as input. The difference between real clothed model image $I_{g}$ and synthesis clothed image $I^{\prime}_{g}$ will generate the supervision information for training the whole framework.

Underwear Model Virtual Try-On (UMV) Dataset. The UVM dataset consists of $143,428$ pseudo labeled pairs in total. To get the pseudo labeled pairs, we photographed $2648$ underwear models with different poses and collected tens of thousands of clothing images from the e-commerce platform. Key points and parsing labels for both models and clothing are defined. We manually label the model images to get accurate results. For clothing images, we use pre-trained models (Li et al., 2019; Sun et al., 2019) to predict key points and parsing results, and remap to our own definitions. An ARAP based annotation tool is used with manually fine-tune to get the pseudo label $I_{g^{\prime}}$. supplementary materials provide more detailed descriptions of the UMV dataset.

Multi-Pose Virtual Try-On (MPV) Dataset. MPV dataset (Dong et al., 2019) consists of $37,723$/$14,360$ person/clothing images, the dataset used for experiments is one part of the MPV dataset, consisting of 14,754 pairs of top clothing images and positive perspective images of female models. The resolution of images in the dataset is $256\times 192$.

Network Architecture. The GAT module contains two graph attention blocks, each with three graph attention layers followed by a ReLU activation layer. The output dimension of each layer is $256$, and the final output dimension is $2$. The feature maps of the former three layers with $64$, $128$, $256$ channels of VGG network are used in the shape information extraction module. For each vertex, Perceptual Feature Pooling extract the feature from four near pixels using bilinear interpolation according to its coordinate position and get a $1\times 448$ vector. Two $1\times 448$ vectors extracted from the model and the clothing are concatenated into a $1\times 896$ vector, and then reduce the dimension by a fully connection layer to get the final $1\times 256$ vector.

The Splitting Network is an U-Net (Ronneberger et al., 2015) like encoder-decoder network. Three contracting blocks are used in Encoder, each with two convolution layers, the output stride is $8$. The decoder uses three expansive blocks, each with two convolution layers and a transposed convolution to recover feature map size and concatenate with the corresponding contracting block’s output. The Synthesis Network’s has the similar network architecture as the Splitting Network. More details about network architecture are given in the supplementary materials.

Parameters Setting. As for training parameters, we set batch size $4$, the epoch is $30$, and $2$ iterations are used. ADAM optimizer is used with $\alpha=0.5$ and $\beta=0.999$, the learning rate is $10^{-3}$. We set $\lambda_{1}=1$, $\lambda_{2}=10$, $\alpha_{1}=0.1$, $\alpha_{2}=1$, $\alpha_{3}=1$, $\beta=0.1$, $\gamma=0.1$, $\eta=1$. Hyperparameters are set in terms of the importance of each subterm, and the balance of magnitudes between each loss.

To verify the effectiveness of shape control, we set different control points as guidance for the same clothing and underwear model. Fig. 5 shows the results of various models and clothing, where we can see that the tightness of sweater and pants, the length of skirt and sleeves are successfully controlled with different control points as inputs. What’s more, for the pants in the middle column of Fig. 5, the tightness and length are controlled simultaneously.

Fig. 6 illustrates our results compared with other methods on test samples selected from the MPV dataset. CP-VTON (Wang et al., 2018b), CP-VTON+ (Minar et al., 2020b) and ACGPN (Yang et al., 2020) are adopted for comparison. The textures of clothing range from simple to complex. Compared with CP-VTON and CP-VTON+, the texture of our approach is much clearer, please see the results of the last two rows in Fig. 6. Meanwhile, compared with ACGPN, our approach can better maintain the clothing’s silhouette. For example, for the two cases in the last row in Fig. 6, ACGPN has turned short sleeves into sleeveless, which is an obvious mistake. Overall, our method is better at keeping the clothing’s texture and can better maintain the clothing’s silhouette.

We conduct a user study on the Qince Platform (A crowdsourcing platform developed by Alibaba Inc.) Given the clothing image, and two synthesized clothed model images by two different methods (with resolution $256\times 192$ and $512\times 384$, respectively), the worker is asked to choose the one is more realistic and accurate in a virtual try-on situation. We choose $500$ testing samples from the MPV dataset. All synthesized results are presented five times to different workers to avoid individual preferences. Quantitative comparisons are summarized in Table 1. As can see from the table, our method gives better results than the other methods at a resolution of $256\times 192$. When the resolution is enlarged to $512\times 384$, our evaluation results are significantly better than the other methods, demonstrating that our method can better maintain texture details for large resolution images.

As shown in Fig. 7, we further analyze and show the visual comparison of warping modules with CP-VTON, CP-VTON+ and ACGPN on MPV dataset. Both CP-VTON and CP-VTON+ use the Spatial Transformation Network (STN) to learn parameters of Thin-Plate Spline (TPS) to warp the clothing image. ACGPN uses TPS with a second-order difference constraint to get less distortion results. As shown in Fig. 7, the results of CP-VTON and CP-VTON+ are often distorted in the sleeve and torso parts, and ACGPN is prone to skewing. This is because the deformation between different clothing parts can affect each other using TPS.

We take this into account in the pre-processing stage. For the parsing definitions of person and clothing, we distinguish between different parts. For example, the left arm, torso and right arm are marked as different parsing categories (See supplementary materials), thus we know which part each mesh belongs to when meshing the clothing. GAT allows overlap between different meshes, and different parts do not affect each other. During the rendering step, the overlapping regions can be handled orderly. Therefore, compared to other methods, our warping results are flatter and rigid, and also better maintain the original silhouette of the garment.

Here we conducted a series of ablation experiments to verify each module and loss’s effectiveness of the proposed methods. For evaluation metric, we calculate the precision by checking the percentage of vertices between prediction and ground truth within a certain threshold $\tau=10^{-2}$.

Table 2 and Fig. 8 show the quantitative and qualitative results of the ablation study. SC-VTON^† is the optimal model trained with UMV dataset. SC-VTON^∗ is the optimal model trained with both UMV and MPV datasets. From Table 2 and Fig. 8, we can see that removing clothing’s image and parsing information made the results worse. Removing $\mathcal{L}_{smooth}$ increases the accuracy, but the silhouette of the clothing becomes jagged. There is also a slight decrease with no iteration during training, demonstrating that iterative training can improve the deformation result. Training with UMV and MPV simultaneously also slightly boosts the performance.

As described in Section 3.2, with the splitting network and the synthesis network, the whole framework can be extended to typical virtual try-on task for the clothed model. At inference time, both the clothed model image $I_{m}$ and a different clothing image $I_{c^{\prime}}$ are sent to the network. We can directly put the warped clothing image to the predicted underwear model to get the final result, or use the Synthesis Network to generate a refined result. More results are given in the supplementary materials. The task of predicting the underwear model is very difficult, hence the visual result of II is not quite good. Even so, adding II can optimize the warping result, while increasing the robustness on new samples.

All warping methods based on 2D clothing images face a common problem: cannot handle the side postures. Most works infer the side textures of the clothing by generators, the results are often blurry and not realistic. Our method also has this problem and this paper only deal with the front view models. For side posture cases, the results are not good, please see Fig. 9 for visual results. We came up with some solutions to solve the problem as our future works: for person modeling, 3D information need to be considered, DensePose (Güler et al., 2018) or SMPL (Loper et al., 2015) model may be two options. A more detailed clothing dataset need to be collected, which should contain at least two clothing images of front and back views. For the warping algorithm, depth information need to be considered during the warping process.