Virtual Accessory Try-On via Keypoint Hallucination

As the main branch of our network, background UNet contains a background encoder $E_{b}$ and a heatmap decoder $D_{hm}$, and is able to generate heatmaps $\hat{H}$ based on the input background image $I_{b}$. Following the structure in [22], the decoder $D_{hm}$ consists of five stages, where the first four stages consist of two to three stacked residual blocks with a deconvolution layer included in the first residual block of each stage, and the last stage consists of only a deconvolution layer and a convolutional layer to obtain the final output. The encoder feature maps from the last three stages in the encoder $E_{b}$ are added to the feature maps in the corresponding stages in the decoder $D_{hm}$ via skip connection. We denote the predicted (resp., ground-truth) $k$-th heatmap as $\hat{H}_{k}$ (resp., $H_{k}$). We omit the subscript $k$ when not specifying a heatmap.

Although the main branch alone can predict the target locations of foreground keypoints, it is challenging to make precise predictions in the absence of foreground information. For example, in the hat try-on task, although the four target locations can be roughly positioned around the top of head, only with the specific shape and the type of hat (e.g., baseball cap, cowboy hat) can they be more precisely decided. Therefore, it is necessary to inject foreground information into the main branch.

We first concatenate the foreground image $I_{f}$ and its mask $M_{f}$, and feed them into the foreground encoder $E_{f}$ to extract the foreground feature map $F_{f}$. Then, we fuse the foreground feature map $F_{f}$ and background feature map $F_{b}$ in the bottleneck to ensure that the heatmap decoder $D_{hm}$ could receive both foreground and background information. Inspired by previous works on attention [33] and feature fusion [9], we propose a dual-attention fusion (DAF) module to fuse two feature maps attentively. The DAF module first applies dual attention (channel attention and spatial attention in [33]) to the summation of two feature maps to predict the fusion weight map $M_{fuse}$ with the same size as $\{F_{f},F_{b}\}$. Then, $F_{f}$ and $F_{b}$ are fused according to the fusion weight map $M_{fuse}$:

where $F_{fuse}$ is the fused feature. The values in $M_{fuse}$ are between 0 and 1 after Sigmoid, enabling our network to make a soft selection between two feature maps. The DAF module is illustrated in Figure 2, in which the dashed line means reversing the fusion weight map.

Due to our design of background-oriented network, it can be easily extended to utilize additional background information. In this work, we take semantic information as an example to investigate how to leverage auxiliary background information in our network. It is reasonable to presume that background semantic information could benefit target location hallucination, because the relative position and distance to different semantic components (e.g., eyes, nose, ears) could help determine the target locations of foreground keypoints. Similar to previous multitask learning works [29, 32, 3, 4], we introduce a semantic decoder $D_{sm}$ to predict background semantic mask. The semantic decoder $D_{sm}$ and the heatmap decoder $D_{hm}$ share the same background encoder $E_{b}$. $D_{sm}$ and $E_{b}$ also have skip connections. To leverage the semantic knowledge learned by $D_{sm}$, the feature map of each layer in the decoder $D_{sm}$ is propagated to the decoder $D_{hm}$ by adding to the corresponding feature map, except the last layer which accounts for heatmap prediction.

We conduct experiments on STRAT dataset [37], which consists of three sub-datasets (STRAT-glasses, STRAT-hat, STRAT-tie). Each sub-dataset contains 2000 training tuples and 1000 test tuples. Each tuple has a foreground image with mask, a background image, and the ground-truth composite image. The foreground/background images in the training set and test set have no overlap. For quantitative evaluation, we adopt local structural similarity index (LSSIM), vertex displacement error (Disp), and intersection over union (IoU) following [37]. We also consider human perception and include user study for more comprehensive comparison. Specifically, we collect the composite images generated by different methods for the whole test set. 20 human raters are asked to select the most reasonable and realistic result for each test tuple. Then, we report the frequency that each method is selected as the best one.

We use PyTorch [26] v1.11.0 to implement our model, and train it on the NVIDIA RTX 3090 GPU. In the STRAT-glasses and STRAT-hat sub-datasets, we resize both foreground and background images to $224\times 224$. For the STRAT-tie sub-dataset, we resize the images to $448\times 448$ because the foreground object only occupies a small proportion of the background, following [37]. We set the Gaussian radius of ground-truth heatmap as $g=20$, and set $\gamma=10$ in Eqn. 3 and $\lambda=0.1$ in Eqn. 8 via cross-validation.

We use adam optimizer with the learning rate initialized as 0.0002 and weight decay set to $1\times 10^{-5}$. The batch size is 32 and our model is trained for 40 epochs with the learning rate decreasing linearly from epoch 10 to 0.00005.

For the semantic classes of the background image, we divide three sub-datasets into two groups. The background images in the first group (STRAT-glasses and STRAT-hat) are human faces and those in the second group (STRAT-tie) are human bodies. Human face has 12 semantic classes: background, hair, hat, eyebrows, glasses, eyes, nose, mouth, skin, neck, ears, and clothing. Human body has 8 semantic classes: background, head, upper, lower, arms, legs, shoes, and bags.

We compare our method with previous virtual accessory try-on methods: CorrelNet [37], ST-GAN [23], RegGAN [18], CompGAN [1], SF-GAN [36], AGCP [21], and representative virtual clothing try-on methods: CP-VTON+ [25], PF-AFN [13].In particular, (+s)(e.g., ST-GAN(+s) and CompGAN(+s)) means adding additional supervision (MSE loss) between generated composite images and ground-truth composite images compared to their original methods, which allows them to utilize the ground-truth annotations more effectively (see [37] for more details). Note that two virtual clothing try-on methods require semantic parsing, and we provide them with ground-truth semantic annotations. As for the other input of these two methods, i.e. the pose heatmaps, we did not use the corresponding information into the input. And we only use the first stage of these two methods to obtain the warped foreground image, and then use the foreground mask as guidance to synthesize the two images, which is also consistent with the process of our method. In addition, since the optical flow-based warping method PF-AFN cannot accurately obtain the final locations of these specified vertices, we calculate the perspective transformation matrix by sampling some matching point pairs, through which we can obtain the final locations of there vertices to approximately estimate the Disp metric.

The quantitative results are summarized in Table 1. Among the virtual accessory try-on methods, the recent method CorrelNet [37] achieves the best results, surpassing other methods by a large margin, which demonstrates the advantage of calculating warping parameters based on selected keypoint pairs. The virtual clothing try-on methods [25, 13] are very competitive and outperform most early virtual accessory try-on methods. We observe that [25, 13] perform admirably on the STRAT-glasses and STRAT-hat datasets, but less ideally on the STRAT-tie dataset. The reason could be that with the foreground tie only occupying a small region on the background, the approach of seeking the global correspondence between foreground and background becomes less effective. This is more obvious in CP-VTON+ because it only establishes such correspondence on high-level features. Its poor performance on STRAT-tie dataset is also attributable to the difficulty in distinguishing between neck ties and bow ties. We refer to our method without semantic information as Ours(w/o sm). It can be seen that Ours(w/o sm) performs better than all previous methods on all metrics, which demonstrates the superiority of our method. After using semantic information, our approach achieves further improvement (Ours v.s. Ours(w/o sm)).

By taking STRAT-hat as an example, we conduct ablation studies to validate the effectiveness of each component in our network, and the results are shown in Table 2. First, we only use the main branch (background encoder $E_{b}$ and heatmap decoder $D_{hm}$). As shown in row 1, the overall results are very poor. On the basis of row 1, we introduce semantic mask and foreground information respectively, and the results are shown in row 2 and row 3 respectively. As can be seen, the additional semantic information brings limited improvement in this case. The addition of foreground information results in substantial improvement, because our model is unable to predict heatmaps accurately without knowing the foreground information (e.g., shape and sub-type of hat). On the basis of row 3, we add the semantic mask to arrive at our full-fledged method in row 9.

Then, we investigate the strategy of injecting foreground information. One simple strategy is adding the feature maps $F_{b},F_{f}$ as fused feature map $F_{fuse}$. The obtained results in row 4 are lower than in row 9, which demonstrates that the DAF module can fuse foreground and background feature maps more effectively. Besides, we also experiment with a simplified version of DAF, which is referred to as DAF(s) in row 5. Specifically, we remove the dual attention [33] and directly output the fusion weight map. Compared with row 9, the features refined by dual attention are more conducive to feature fusion.

Next, we study the impact of various loss functions. We only use MSE loss or AWing loss [31] to supervise the heatmap, as shown in row 6 and row 7 respectively. We observe that AWing loss is more superior in supervising heatmap prediction. In row 8 (resp., 9), we add weighted loss based on row 6 (resp., 7). Whether using MSE loss or AWing loss, assigning higher weights to the positive locations is helpful to address the imbalance issue.

To better test the performance of our model, we perform hyper-parameter analyses and significance test. In the hyper-parameter analyses, we vary the values of three hyper-parameters (i.e., $\lambda$ in Eqn. 8, $\gamma$ in Eqn. 3, and the Gaussian radius $g$ of ground-truth heatmap), and plot the performance variance of our method to observe their impact. The significance test is conducted between our method and the strongest baseline CorrelNet [37]. The detailed results are left to Supplementary due to the space limitation.