Single Stage Warped Cloth Learning and Semantic-Contextual Attention Feature Fusion for Virtual TryOn

The objective of virtual try-on is to produce a try-on image $I_{p\_output}\in\mathbb{R}^{3XHXW}$ by leveraging a person image $I_{p}\in\mathbb{R}^{3XHXW}$ and an in-shop garment image $I_{g}\in\mathbb{R}^{3XHXW}$. The aim is to ensure that the garment in $I_{g}$ seamlessly aligns with the corresponding regions in $I_{p}$, creating a coherent visual outcome. Moreover, the generated $I_{p\_{output}}$ should retain both the intricate elements from $I_{g}$ and the non-garment sections of $I_{p}$. In simpler terms, the individual portrayed in $I_{p}$ should remain unaltered in $I_{p\_{output}}$ except for the addition of the worn garment $I_{g}$.

Our proposed methodology encompasses several key components: An Attention-based Flow Estimator, Semantic-Contextual Fusion Attention Block, and a Joint Learning Tryon Module.

The Inputs to our method are the in-shop garment $I_{g}$ and cloth agnostic input $I_{agnostic}$ of subject person $I_{p}$ for tryon. Additionally, we utilize a keypoint visualization $I_{pose}$ to facilitate target person synthesis.

Our method utilises two feature extractors. The source input is $I_{g}$ whereas the reference input is the channel concatenated $I_{agnostic}$ and $I_{pose}$. A Refined Pyramid Extractor operates by computing features at multiple scales [10].

These features are then employed for learning deformable flow through our innovative Linear Attention Flow (LAF) block. LAF learns both attention and flow across various scales, which is used by our warp module to compute warped features. These warped features are fed into the Semantic-Contextual Feature Fusion block to predict the final tryon image$I_{p\_{output}}$. It’s noteworthy that attention and flow are learned using a multi-scale residual approach similar to Fig2.

In the following subsections, we describe every module in detail.

Transformers, pivotal in NLP and computer vision, rely on Query, Key, and Value (QKV) formulation. Despite enabling intricate dependency modeling, its quadratic complexity poses computational challenges. Addressing this, our proposed Linear Attention Flow Block (LAF) employs efficient linear attention, as in Wang et al. [14]. This alternative minimizes costly matrix multiplications, significantly reducing computation and memory requirements, ensuring LAF’s scalability and efficiency at high spatial resolutions.

The fundamental principle of our proposed linear self-attention involves the incorporation of two linear projection matrices, $E_{i}$ and $F_{i}\in\mathbb{R}^{n\times k}$, during key and value computation. Initially, we project the original $(n\times d)$-dimensional key and value layers, $KW_{i}^{K}$ and $VW_{i}^{V}$, into $(k\times d)$-dimensional projected counterparts. Subsequently, the context mapping matrix $\bar{P}$ of dimensions $(n\times k)$ is derived through scaled dot-product attention, as illustrated by the equation:

The final step entails generating context embeddings for each head_i utilizing $\bar{P}\cdot(F_{i}VW_{i}^{V})$. Importantly, these operations exhibit a time and space complexity of $O(nk)$, which is considerably efficient. This efficiency becomes more pronounced when we opt for a small projected dimension $k$, substantially reducing memory and space requirements. Our LAF block stands as a testament to the balance struck between computational efficiency and robust attention modeling, making it well-suited for our Virtual Tryon pipeline. Finally, the attended values are obtained as $Output=Attention_{scores}\cdot V$.

In this subsection, we propose a novel approach for computing the final try-on result in our virtual try-on pipeline. The baseline feeds pixel-wise summation of the features of the warped garment and the features of the generated reference person to a 2-layer convolutional shallow decoder to compute the target tryon result. This approach lacks the ability to effectively capture the fine-grained relationships between the two sets of features, potentially leading to suboptimal try-on results. To address this limitation, we introduce a Semantic-Contextual Fusion Attention (SCFA) module that encapsulates the attention mechanism’s capability to capture semantic information while considering the context between the garment and the person during the try-on process.

The warping module is the backbone of the tryon methodology and the results obtained from this stage play a crucial role in the tryon module moving forward.

The key idea behind this constraint is to integrate explicit learning of the structural and perceptual complexity of the warped garment into our pipeline. The availability of high quality garment segmentation is one of the key factors that motivated us to formulate a learning constraint with the warped garment directly into the single stage pipeline [10]. This results into a joint learning of the warped garment and tryon results, improving the results in a parallel manner.

The WCLM uses the parser based label in the form of garment segmentation GT as the guiding parameter. The model is optimised utilising an L1 loss.

The loss $L_{warp}$ is a combination of $I_{wgt}$ and $I_{warp}$ and can be written as:

The L1 loss is a combination of $L_{tryon}$ and $L_{warp}$ mathematically formulated as ,

The perceptual loss is based on the VGG network which calculates the L1 distance between the extracted feature maps by the network for both the tryon and warped garment, formulated as ,

The style Loss is the optimised statistical error between the feature maps of predicted and GT , where the gram matrix measures the feature correlation between the two inputs,

where $G_{i}^{\phi}$ denotes the Gram matrix of features.

The overall loss is presented as:

where $\mathcal{L}^{n}$ is the loss of the $n_{th}$ scale, and the scale closer to the output is given a larger weight $n-1$.

The experiments were conducted on two commonly used and publicly available datasets, VITON[5] and VITON-HD[4]. VITON consists of 14221 training images and a test set of 2032 pairs of images. The view for the images is frontal and paired with the in-shop garment cloth for tryon. VITON-HD contains high-resolution paired images of in-shop garments and their corresponding human models wearing the garments consisting of consists of 11647 train images and 2032 test images.

All the experiments have been conducted on two A5000 GPUs using PyTorch. The deformable flow is learned the value of $K=6$. Adam optimizer is used for the batchsize of 8. The model is trained for a learning rate of 0.000035 for 250 epochs. The predefined weights for the loss functions are $\lambda_{L1}=1,\lambda_{prec}=1,\lambda_{style}=100$. The features are extracted for $N=5$ scales with $[64,128,256,256,256]$ number of filters.

Our method was qualitatively evaluated against CP-VTON+ [1], ACPGN [16], PFAPN [17], SDAFN [10] (Fig. 2). While CP-VTON+ and ACPGN are parser-based and prone to misalignment issues, PF-AFN, a parser-free framework, avoids misalignment but suffers from blurred body parts like arms and shoulders.

All methods nearly align the distorted garment with the wearer, but artifacts become more visible as the texture and structural complexity increases. Our model preserves image contrast and skin texture, especially neck region, much better. In Fig. 2, our model exhibits superior preservation in left shoulder garment and realistic arm generation, while ensuring structural integrity with minimal misalignments in sleeves and collar edges.

The baseline of the proposed approach incorporated a simple warp module and flow without an attention module with the computed scores as shown in table2, as an improvement we introduce LAF, SCFA and a Warp loss. The LAF module improves the flow which consecutively improves the garment warping, SCFA module encapsulates the attention mechanism’s capability to capture semantic information while considering the context between the garment and the person during the try-on process, and Finally, the warped loss improves the warping by not allowing the learning of warping in bad garment regions as shown in table2. Proposed method’s assessment on VITON Dataset as shown in Table 1, achieves outstanding results in terms of SSIM and FID with scores of 0.90 and 9.14, respectively, outperforming other methods as shown in Table1. The evaluated scored on VITON-HD dataset also outweigh the performance of proposed method upon comparison with other methods as shown in table3 demonstrating the effectiveness of our proposed single-stage framework.