Towards Scalable Unpaired Virtual Try-On via Patch-Routed Spatially-Adaptive GAN

Since the paired data for supervised training is unavailable for the unpaired virtual try-on task, the synthesis network has to be trained in an unsupervised manner via image reconstruction, and thus takes a person image as input and separately extracts the feature of the intact garment and the feature of the person representation to reconstruct the original person image. While such a training strategy retains the intact garment information, which is helpful for the garment reconstruction, the features of the intact garment entangle the garment style with the spatial information in the original image. This is detrimental to the garment transfer during testing. Note that the garment style here refers to the garment color and categories, i.e., long sleeve, short sleeve, etc., while the garment spatial information implies the location, the orientation, and the relative size of the garment patch in the person image, in which the first two parts are influenced by the human pose while the third part is determined by the relative camera distance to the person.

To address this issue, we explicitly divide the garment into normalized patches to remove the inherent spatial information of the garment. Taking the sleeve patch as an example, by using division and normalization, various sleeve regions from different person images can be deformed to normalized patches with the same orientation and scale. Without the guidance of the spatial information, the network is forced to learn the garment style feature to reconstruct the garment in the synthesis image.

Fig. 3 illustrates the process of obtaining normalized garment patches, which includes two main steps: (1) pose-guided garment segmentation, and (2) perspective transformation-based patch normalization. Specifically, in the first step, the source garment $G_{s}$ and human pose (joints) $J_{s}$ are firstly obtained by applying Gong2019Graphonomy and openpose to the source person $I_{s}$, respectively. Given the body joints, we can segment the source garment into several patches $P_{s}$, which can be quadrilaterals with arbitrary shapes (e.g., rectangle, square, trapezoid, etc.), and will later be normalized. Taking the torso region as an example, with the coordinates of the left/right shoulder joints and the left/right hips joints in $P_{s}^{i}$, a quadrilateral crop (of which the four corner points are visualized in color in $P_{s}^{i}$ of Fig. 3) covering the torso region of $G_{s}$ can be easily performed to produce an unnormalized garment patch. Note that we define eight patches for upper-body garments, i.e., the patches around the left/right upper/bottom arm, the patches around the left/right hips, a patch around the torso, and a patch around the neck. In the second step, all patches are normalized to remove their spatial information by perspective transformations. For this, we first define the same amount of template patches $P_{n}$ with fixed $64\times 64$ resolution as transformation targets for all unnormalized source patches, and then compute a homography matrix $\mathcal{H}_{s\rightarrow n}^{i}\in\mathbb{R}^{3\times 3}$ zhou2019stnhomography for each pair of $P_{s}^{i}$ and $P_{n}^{i}$, based on the four corresponding corner points of the two patches. Concretely, $\mathcal{H}_{s\rightarrow n}^{i}$ serves as a perspective transformation to relate the pixel coordinates in the two patches, formulated as:

$$\left[\begin{array}[]{c}x_{n}^{i}\\ y_{n}^{i}\\ 1\end{array}\right]=\mathcal{H}_{s\rightarrow n}^{i}\left[\begin{array}[]{c}x_{s}^{i}\\ y_{s}^{i}\\ 1\end{array}\right]=\left[\begin{array}[]{ccc}h_{11}^{i}&h_{12}^{i}&h_{13}^{i}\\ h_{21}^{i}&h_{22}^{i}&h_{23}^{i}\\ h_{31}^{i}&h_{32}^{i}&h_{33}^{i}\end{array}\right]\left[\begin{array}[]{c}x_{s}^{i}\\ y_{s}^{i}\\ 1\end{array}\right]$$

(1)

where $(x_{n}^{i},y_{n}^{i})$ and $(x_{s}^{i},y_{s}^{i})$ are the pixel coordinates in the normalized template patch and the unnormalized source patch, respectively. To compute the homography matrix $\mathcal{H}_{s\rightarrow n}^{i}$, we directly leverage the OpenCV API, which takes as inputs the corner points of the two patches and is implemented by using least-squares optimization and the Levenberg-Marquardt method gavin2019levenberg . After obtaining $\mathcal{H}_{s\rightarrow n}^{i}$, we can transform the source patch $P_{s}^{i}$ to the normalized patch $P_{n}^{i}$ according to Eq. 1.

Moreover, the normalized patches $P_{n}$ can further be transformed to target garment patches $P_{t}$ by utilizing the target pose $J_{t}$, which can be obtained from the target person $I_{t}$ via  openpose . The mechanism of that backward transformation is equivalent to the forward one in Eq. 1, i.e., computing the homography matrix $\mathcal{H}_{n\rightarrow t}^{i}$ based on the four point pairs extracted from the normalized patch $P_{n}^{i}$ and the target pose $J_{t}$. The recovered target patches $P_{t}$ can then be stitched to form the warped garment $G_{t}$ that will be sent to the texture synthesis branch in Fig. 2 to generate more realistic garment transfer results. We can also regard $\mathcal{H}_{s\rightarrow t}=\mathcal{H}_{n\rightarrow t}\cdot\mathcal{H}_{s\rightarrow n}$ as the combined deformation matrix that warps the source garment to the target person pose, bridged by an intermediate normalized patch representation that is helpful for disentangling garment styles and spatial features.

Motivated by the impressive performance of StyleGAN2 karras2020stylegan2 in the field of image synthesis, our PASTA-GAN inherits the main architecture of StyleGAN2 and modifies it to the conditional version (see Fig. 2). In the synthesis network, the normalized patches $P_{n}$ are projected to the style code $w$ through a style encoder followed by a mapping network, which is spatial-agnostic benefiting from the disentanglement module. In parallel, the conditional information including the target head $H_{t}$ and pose $J_{t}$ is transferred into a feature map $f_{id}$, encoding the identity of the target person by the identity encoder. Thereafter, the synthesis network starts from the identity feature map and leverages the style code as the injected vector for each synthesis block to generate the try-on result $\tilde{I_{t}}^{\prime}$.

However, the standalone conditional StyleGAN2 is insufficient to generate compelling garment details especially in the presence of complex textures or logos. For example, although the illustrated $\tilde{I_{t}}^{\prime}$ in Fig. 2 can recover accurate garment style (color and shape) given the disentangled style code $w$, it lacks the complete texture pattern. The reasons for this are twofold: First, the style encoder projects the normalized patches into a one-dimensional vector, resulting in loss of high frequency information. Second, due to the large variety of garment texture, learning the local distribution of the particular garment details is highly challenging for the basic synthesis network.

To generate more accurate garment details, instead of only having a one-way synthesis network, we intentionally split PASTA-GAN into two branches after the $128\times 128$ synthesis block, namely the Style Synthesis Branch (SSB) and the Texture Synthesis Branch (TSB). The SSB with normal StyleGAN2 synthesis blocks aims to generate intermediate try-on results $\tilde{I_{t}}^{\prime}$ with accurate garment style and predict a precise garment mask $M_{g}$ that will be used by TSB. The purpose of TSB is to exploit the warped garment $G_{t}$, which has rich texture information to guide the synthesis path, and generate high-quality try-on results. We introduce a novel spatially-adaptive residual module specifically before the final synthesis block of the TSB, to embed the warped garment feature $f_{g}$ (obtained by passing $M_{g}$ and $G_{t}$ through the garment encoder) into the intermediate features and then send them to the newly designed spatialy-apaptive residual blocks, which are beneficial for successfully synthesizing texture of the final try-on result $I_{t}^{\prime}$. The detail of this module will be described in the following section.

Given the style code that factors out the spatial information and only keeps the style information of the garment, the style synthesis branch in Fig. 2 can accurately predict the mean color and the shape mask of the target garment. However, its inability to model the complex texture raises the need to exploit the warped garment $G_{t}$ to provide features that encode high-frequency texture patterns, which is in fact the motivation of the target garment reconstruction in Fig. 3.

However, as the coarse warped garment $G_{t}$ is directly obtained by stitching the target patches together, its shape is inaccurate and usually misaligns with the predicted mask $M_{g}$ (see Fig.4). Such shape misalignment in $G_{t}$ will consequently reduce the quality of the extracted warped garment feature $f_{g}$.

To address this issue, we introduce the spatially-adaptive residual module between the last two synthesis blocks in the texture synthesis branch as shown in Fig. 2. This module is comprised of a garment encoder and three spatially-adaptive residual blocks with feature inpainting mechanism to modulate intermediate features by leveraging the inpainted warped garment feature.

To be specific on the feature inpainting process, we first remove the part of $G_{t}$ that falls outside of $M_{g}$ (green region in Fig.4), and explicitly inpaint the misaligned regions of the feature map within $M_{g}$ with average feature values (orange region in Fig. 4). The inpainted feature map can then help the final synthesis block infer reasonable texture in the inside misaligned parts.

Therefore given the predicted garment mask $M_{g}$, the coarse warped garment $G_{t}$ and its mask $M_{t}$, the process of feature inpainting can be formulated as:

$$M_{align}=M_{g}\cap M_{t},$$

(2)

$$M_{misalign}=M_{g}-M_{align},$$

(3)

$$f^{\prime}_{g}=\mathcal{E}_{g}(G_{t}\odot M_{g}),$$

(4)

$$f_{g}=f_{g}^{\prime}\odot(1-M_{misalign})+\mathcal{A}(f_{g}^{\prime}\odot M_{align})\odot M_{misalign},$$

(5)

where $\mathcal{E}_{g}(\cdot)$ represents the garment encoder and $f_{g}^{\prime}$ denotes the raw feature map of $G_{t}$ masked by $M_{g}$. $\mathcal{A}(\cdot)$ calculates the average garment features and $f_{g}$ is the final inpainted feature map.

Subsequently, inspired by the SPADE ResBlk from SPADE park2019spade , the inpainted garment features are used to calculate a set of affine transformation parameters that efficiently modulate the normalized feature map within each spatially-adaptive residual block. The normalization and modulation process for a particular sample $h_{z,y,x}$ at location ($z\in C,y\in H,x\in W$) in a feature map can then be formulated as:

$$\gamma_{z,y,x}(f_{g})\frac{h_{z,y,x}-\mu_{z}}{\sigma_{z}}+\beta_{z,y,x}(f_{g}),$$

(6)

where $\mu_{z}=\frac{1}{HW}\sum_{y,x}h_{z,y,x}$ and $\sigma_{z}=\sqrt{\frac{1}{HW}\sum_{y,x}\left(h_{z,y,x}-\mu_{z}\right)^{2}}$ are the mean and standard deviation of the feature map along channel $C$. $\gamma_{z,y,x}(\cdot)$ and $\beta_{z,y,x}(\cdot)$ are the convolution operations that convert the inpainted feature to affine parameters.

Eq. 6 serves as a learnable normalization layer for the spatially-adaptive residual block to better capture the statistical information of the garment feature map, thus helping the synthesis network to generate more realistic garment texture.

With the modulated intermediate feature maps produced by the spatially-adaptive residual module, the texture synthesis branch can effectively utilize the reconstructed warped garment and generate the final compelling try-on result with high-frequency texture patterns.

As paired training data is unavailable, our PASTA-GAN is trained unsupervised via image reconstruction. During training, we utilize the reconstruction loss $\mathcal{L}_{rec}$ and the perceptual loss johnson2016perceptual $\mathcal{L}_{perc}$ for both the coarse try-on result $\widetilde{I}^{\prime}$ and the final try-on result $I^{\prime}$:

$$\mathcal{L}_{rec}=\sum_{I\in\{\widetilde{I}^{\prime},I^{\prime}\}}\|I-I_{s}\|_{1}\;\;\;\;and\;\;\;\;\mathcal{L}_{perc}=\sum_{I\in\{\widetilde{I}^{\prime},I^{\prime}\}}\sum_{k=1}^{5}\lambda_{k}\left\|\phi_{k}(I)-\phi_{k}\left(I_{s}\right)\right\|_{1},$$

(7)

where $\phi_{k}(I)$ denotes the $k$-th feature map in a VGG-19 network DBLP:journals/corr/SimonyanZ14a pre-trained on the ImageNet ILSVRC15 dataset. We also use the $L_{1}$ loss between the predicted garment mask $M_{g}$ and the real mask $M_{gt}$ which is obtained via human parsing Gong2019Graphonomy :

$$\mathcal{L}_{mask}=\|M_{g}-M_{gt}\|_{1}.$$

(8)

Besides, for both $\widetilde{I^{\prime}}$ and $I^{\prime}$, we calculate the adversarial loss $\mathcal{L}_{GAN}$ which is the same as in StyleGAN2 karras2020stylegan2 . The total loss can be formulated as

$$\mathcal{L}=\mathcal{L}_{GAN}+\lambda_{rec}\mathcal{L}_{rec}+\lambda_{perc}\mathcal{L}_{perc}+\lambda_{mask}\mathcal{L}_{mask},$$

(9)

where $\lambda_{rec}$, $\lambda_{perc}$, and $\lambda_{mask}$ are the trade-off hyper-parameters.

During training, although the source and target pose are the same, the coarse warped garment $G_{t}$ is not identical to the intact source garment $G_{s}$, due to the crop mechanism in the patch-routed disentanglement module. More specifically, the quadrilateral crop for $G_{s}$ is by design not seamless/perfect and there will accordingly often exist some small seams between adjacent patches in $G_{t}$ as well as incompleteness along the boundary of the torso region. To further reduce the training-test gap of the warped garment, we introduce two random erasing operations during the training phase. First, we randomly remove one of the four arm patches in the warped garment with a probability of $\alpha_{1}$. Second, we use the random mask from liu2018partialconv to additionally erase parts of the warped garment with a probability of $\alpha_{2}$. Both of the erasing operations can imitate self-occlusion in the source person image. Fig. 5 illustrates the process by displaying the source garment $G_{s}$, the warped garment $G_{t}^{\prime}$ that is obtained by directly stitching the warped patches together, and the warped garment $G_{t}$ that is sent to the network. We can observe a considerable difference between $G_{t}$ and $G_{s}$. An ablation experiment to validate the necessity of the randomly erasing operation for the unsupervised training is included in the supplementary material.