XCloth: Extracting template-free textured 3D clothes from a monocular image

The first task is to recover the 3D geometry of each garment from the monocular image. To this end, we adopt the architecture of PeelGAN (jinka2020peeledhuman) and extended it to include two novel decoder branches. More specifically, the commmon encoder of the PeelGAN (jinka2020peeledhuman) now encodes the input image along with peeled SMPL (depth and body part segmentation) priors. This common encoding is fed to four decoder branches namely, $D_{rgb}$, $D_{depth}$, $D_{seg}$, $D_{norm}$. $D_{rgb}$, $D_{depth}$ serve the same purpose as in PeelGAN, i.e. predicting RGB peelmaps and depth peelmaps. $D_{seg}$ branch predicts the semantic labels across all the peeled layers, providing pixel aligned semantic information in 3D in order to differentiate between multiple garments and body parts in the depth peelmaps. In order to train the respective decoder, we minimize the cross entropy loss on the label classes. The classes in ground truth labels correspond with the classes provided by PGN (gong2018instancelevel). Please refer to Section LABEL:sec:data-generation for complete semantic data generation process. It is important to note that that semantic labels should not only help in identifying different clothes, but also improve the depth prediction and corrects the distortion in body parts, as demonstrated in (SnehithICVGIP2021). To further improve the prediction of the deformation in clothing due to underlying body, similar to  (jinka2021sharp), we provide SMPL depth peelmaps as a prior along with input RGB image. Additionally, we also provide peeled semantic labels of the SMPL body parts (see Fig. LABEL:fig:). This helps the network to predict the semantic labels more accurately and also to learn the correct deformation in clothing in presence of the underlying human body.

Another key improvement in architecture is the addition of $D_{norm}$ branch, which estimates peeled normal maps. The normal maps add further details in the surface, by refining the wrinkles and folds present in the garments. Generally, the trend is to predict normal maps for front and back from the input image using off the shelf network (like in PIFuHD (saito2020pifuhd) and ICON (xiu2022icon)) and then passing them as a prior to subsequent generator networks, thereby improving the fidelity of the predicted surface. However, this strictly restricts the network to overcome any noise present in the predicted prior normal maps and leads to over fitting and failure to generalize in case of loose clothing. Thus, our normal peelmaps estimation branch ($D_{norm}$) enable passive regularization for improving common encoding that would eventually help improve the prediction of peeled depth maps. We minimizing the L1 and perceptual loss over the normal peelmaps.

The aforementioned reason is one of the motivations behind using a simple yet efficient multi-branch encoder-decoder architecture. All the different decoder branches provide each other just the relevant information by propagating the gradients via the common encoder, while avoiding interfering directly with each other, resulting in more flexible learning which is robust to the noisy ground truth information.

It is important to note that all the peelmaps ( i.e., depth, rgb, segmentation and normal) are pixel-aligned with each other. The output of the 3D reconstruction module is the point cloud estimated from backprojected depth peelmaps, where each point is also assigned a segmentation label directly from the predicted semantic peelmaps. This labelling information is used to separate garments. The extracted point clouds of the garments are passed to the next module.

The output of the reconstruction module are the point cloud representation of the garments and have missing regions, due to inherent drawback of PeeledHuman representation. These missing regions are cause by surface patches that are tangential to peeling camera rays and hence need to be filled to be able to recover a complete dense surface representation of garments. One common solution is to run PSR (Poisson Surface Reconstruction) on top of point cloud which gives a watertight mesh. But PSR smoothens out the details coming from the depth peelmaps (back-projected point cloud). Instead of naively running PSR, we first meshify the depth peelmaps in image space independently for each peeling layer and merge them while filling only the missing tangential regions by sampling on the Poisson surface reconstruction of the pointcloud. This helps in retaining the fine-grained surface details present in the predicted depth maps. Please refer Fig 2 for geometry refinement. Additionally, meshifying each depth peelmap also allows us to store peeling information, i.e. each vertex belongs to which peeling layer. Storing this information is useful as it is later needed for texture mapping.

While existing garment digitization methods require a predefined parametrized garment template and then learn to transfer texture from multi-view input image(s) to the UV space, our novel approach adopt a method which exploits the predicted depth and RGB peelmaps for UV parametrization as well as texture map filling.

One naive idea is to simply apply flat plane parametrization to the geometry, but it causes overlap as multiple vertices project onto the same UV space. Various parameterization techniques are available such as LSCM (levy2002least), etc., which estimate parameterization from 3D space to 2D space. Most of the UV parametrization methods either suffer from the discontinuity in the texture space or multiple faces overlap in the UV space while parametrizing a (spherical) genus zero surfaces like human body (heckbert1986survey). The former results in visible seams throughout the whole geometry, while the latter faces the issue of projecting wrong texture to the geometry. Discontinuity in the uv space can be partially solved by employing in-painting methods to in-paint the seams, but estimating seams which can produce minimal distortion in the geometry is not straightforward.

We propose to perform automatically find the seams and split the garment mesh into multiple parts guided by predicted peeled depth maps and then apply UV parametrization on individual part meshes. The peelmaps are useful here, as they naturally split the shape into parts and hence can be used to find consistent seams. Instead of projecting to the same UV space, we can project the vertices according the peeling layer they belong to, thereby constructing a layered-UV space which avoids the issue of overlap.

Boundary first flattening(sawhney2017boundary) parametrization which produces flattening with minimal area distortion and virtually zero angle distortion. The initial mapping can be interactively edited using handle-based manipulation to reduce the overlap of vertices and to texture regions that receive more texture. In this work, we exploit depth peelmaps and split the mesh according to the peeling information to handle the overlapping.

The proposed split texture mapping approach has three steps:

1. (1)

   Automated Seam Estimation: Since we have stored the peeling information for each vertex while reconstructing the mesh for each depth peelmap, we know which mesh vertices belong to which peeling layer as every mesh vertex project to a pixel in the peelmap. For vertices belonging to faces that are tangential to peeling camera rays, we employ nearest neighbor approach to find association with depth peelmaps. This yields a automatic definition of seams consist of set of vertices that are equidistance (in terms of mesh connectivity) from respective peelmap pairs. We propose to replicate these seam vertices across the two splits to avoid the artifacts near the split boundary.

2. (2)

   Peeled Paramterization: We parametrize each split part mesh separately using boundary first flattening  (sawhney2017boundary). This gives us 3D to 2D parametrization for each part with very minimal distortion.

3. (3)

   Texture Filling & In-painting: Traditional methods fill texture map by projecting respective 3D points on visible multi-view images and interpolate the RGB colors for corresponding texture map pixel. However, this is computationally expensive and require multi-view images. Instead, we exploit the multi-layer RGB peelmaps for texture filling and our depth peelmap guided split UV parametrization strategy enables us to achieve this. Since every split texture map for a specific garment have an associated RGB peelmaps that are also pixel aligned. Thus, for each pixel in texture map, we take the RGB value from respective RGB peelmap pixel. This enables filling large part of texture map. However, the surface regions that are not visible in the peelmaps remain unfilled and still need a valid texture. We propose to employ off-the-shelf structured image in-painting solution  (huang2014image) for this task as we already know the mask for these unfilled pixels in the texture map.

   Additionally, we empirically observed that predicting high quality texture for unseen areas is a tedious task for any generative models like that of our’s. Thus, if the image has high frequency texture garments, we propose to employ the existing structured in-painting for second or subsequent RGB peelmaps from a known patch in the visible region of the input image or externally provided texture image of the underlying cloth (as shown in Figure LABEL:fig:).

We employ a multi-branch encoder-decoder net- work for XCloth, which is trained in an end-to- end fashion. The network takes the input image concatenated with SMPL peel maps in 512 × 512 resolution. The shared encoder is consist of a convolutional layer and 2 downsampling layers which have 64, 128, 256 kernels of size 7×7, 3×3 and 3×3, respectively. This is followed by ResNet blocks which take downsampled feature maps of size 128×128×256. The decoders for predicting D fused, D rd and R , consist of two upsampling lay- ers followed by a convolutional layer, having same kernel sizes as of the shared encoder. Sigmoid acti- vation is used in D fused and D rd decoder branches, while a tanh activation is used for the R decoder branch. The D rd output values are scaled to a [−1, 0.5] range which is found empirically. We use the Adam optimizer with an expo- nentially decaying learning rate starting from 5 × 10−4. Our network takes around 18 hrs to train for 20 epochs on 4 Nvidia GTX 1080Ti GPUs with abatchsizeof8and to 1, 1, 0.1 and 0.001, respectively, found empirically. We use trimesh (Dawson-Haggerty et al., 2019) library for rendering the peel maps.

In this subsection we explore the available datasets which can be leveraged for cloth digitization. (bhatnagar2019mgn) introduced the Digital Wardrobe dataset which consists of 94 real body scans with textured meshes in simple postures. The dataset covers only five categories of garments which are relatively tight clothing. DeepFashion3D (zhu2020deep) introduces vertex colored point cloud data of various garments with diverse garment categories, however they do not provide the input images which restricts us to use it for reconstruction from monocular images. (bertiche2020cloth3d) proposed large-scale dataset that provides dynamic dataset of garments in motion. However, the dataset is still synthetic in nature and cannot be fruitful for training models to extract clothes on real-world data. [Thuman2.0] proposed a large-scale dataset of around 500 real body scans with relatively tight clothing and diverse poses. Recently [3DHuman] introduced the 3DHuman dataset which contains the high-frequency textured scans of South-Asian popoulation with diverse and loose clothing.
Hence we train our network on 3DHumans, THUmans2.0 and Digital Wardrobe datasets and show superior performance over SOTA methods qualitatively as well as quantitatively.

We show the qualitative results of our method on three different datasets in Figure 5. Here (i) & (ii) corresponds to 3DHumans, (iii) & (iv) to THUmans2.0 and (v) & (vi) to the Digital Wardrobe dataset respectively. Our method can reconstruct high qualiy textured geometry for topwear and bottomwear separately. Due to the incorporating of the texture mapping, the patterns in the input image are preserved in the rendering of mesh as well.

To this end, we showcase our results on in-the-wild internet images rendered in different views (Figure 4). We show that our method can generalize to loose and diverse garments and can recover high fidelity textured mesh. Note that how textured reconstruction can recover the

We evaluate our framework separately for two classes of garments (topwear & bottomwear). To evaluate the reconstruction error, we compute Point-to-Surface distance (P2S) with the ground garment mesh. For evaluating the output of the segmentation branch, we use Intersection-Over-Union (IOU) for each class. Finally, to evaluate the quality of the extracted garment surface, we compute normal reprojection error (NRE) separately for each class (ignoring the background). We report the quantitative results in Table 1.

We train OccNet on 3DHumans and THUman2.0 dataset and compare it with our method in terms of P2S. In OccNet, there is no provision for semantic segmentation for each class, neither the final reconstruction is pixel-aligned with the input image, therefore we skip IOU and NRE during quantitative comparison. Additionally, the training code for other methods is not available so we decided not to do a quantitative comparison with them. We report the quantitative comparison with OccNet in Table 2.

We extended original PeelGan architecture by adding another branch from the shared encoder for predicting normal peelmaps. Depth values in the depth map depend on the distance from the camera, whereas normal maps are pure representation of the surface geometry, therefore incorporating normal peelmaps helps retaining fine wrinkles and other high-frequency surface details in the garments. Moreover, computing loss on normal peelmaps regularizes the depth map prediction and lowers the P2S and NRE as shown in Table 1 and Table 3.

Although the segmentation branch of the module-1 is generalizable to in-the-wild images, but in case of highly challenging poses and complex clothing, the chance of errors in segmentation in high. For such cases, we have modified our framework to take the segmentation map for the first layer using some off-the-shelf semantic segmentation networks [JPPNet, PGN], and predict segmentation maps for the remaining layers. This small augmentation helps in achieving accurate semantic labelling in complex cases as can be seen by improved IOU in table Table 4 as compared to Table 1