MVP-Human Dataset for 3D Clothed Human Avatar Reconstruction from Multiple Frames

Multi-view acquisition approaches capture the scene from multiple synchronized cameras, and build the 3D shape from photometric cues. Stereo based methods [25, 26, 27, 7, 28] achieve remarkable performance by optimizing multi-view stereo constraints from a large number of cameras. The geometric detail can be further improved by active illumination [29, 30, 31]. However, its hardware configuration is inaccessible to general consumers due to its special equipment and complexity. Benefit from deep learning, recent works [32, 33, 34, 35, 36] reduce the camera number to very sparse views. Yi et.al [37] propose a novel multiview method for human pose and shape reconstruction that scales up to an arbitrary number of uncalibrated camera views (including the single view), guided by dense keypoints. StereoPIFu [38] introduces a stereo vision-based network fusing multi-view images to extract voxel-aligned features. However, this work focuses on statistical reconstruction from multi-view images in the same pose. Compared with these methods, our method can be applied to reconstruct 3D avatars from monocular frames in free views and poses without camera calibration.

Most of the methods assume a pre-scanned human template and reconstruct dynamic human shapes by deforming the template to fit the images. Prior methods [39, 40, 7] require a number of multi-view images as inputs, and align the template to observations using non-rigid registration. MonoPerfCap [41] and LiveCap [9] enable posed body deformation from a single-view video by optimizing the deformation to fit the silhouettes. Recently, DeepCap [10] employs deep learning to estimate the skeletal pose and non-rigid surface deformation in a weakly supervised manner, constrained by the multi-view keypoint and silhouette losses. DeepMultiCap [42] further extends DeepCap to the multi-person scenario. Template-based methods need a pre-scanned template and thus focus on pose tracking and deformation, and template-free capture methods aim to estimate poses and reconstruct surface geometry at the same time. However, most approaches [43, 44, 45, 46, 47] are based on depth sensors which provide surface information for reconstruction, and these methods have difficulty in working on outdoor scenarios. Recently, some approaches [14, 15, 48, 49, 50] utilize the technique of implicit function to generate high-fidelity meshes from single or multi-view RGB images. However, the reconstructed results can not be articulated and applied in scenarios where an avatar is needed. We introduce a template-free method which reconstructs an avatar from merely unconstrained RGB frames. Our method shares the same input formulation, single-view RGB video, with some of these methods, but we do not need a pre-scanned template.

Early works [51, 52, 53] employ statistical body models such as SCAPE [54] or SMPL [55] to recover human models which are typically animatable. However, parametric models cover limited shape variations due to the restriction of PCA shape space. Recently, some methods try to recover a clothed 3D avatar from a monocular video in which a person is moving. Alldieck et.al [13] ask the target subject to turn around in front of the camera while roughly holding the A-pose. Then the dynamic human silhouettes are transformed into a canonical frame of reference, where the visual hull is estimated to deform the SMPL model. This method is further improved by [11] on fine-level details reconstruction by introducing more constraints like shape from shading. Octopus [12] encodes the images of the person into pose-invariant latent codes by deep learning, and fuses the information to a canonical T-pose shape, achieving faster prediction. Such methods share the same goal of reconstructing canonical-pose shape from monocular video with our method, but we do not require the subject to perform specific actions.

Recently, neural-network-based implicit function [56] is introduced to represent 3D humans, demonstrating significant improvement in representation power and fine-grained details than voxel [57] and parametric models [55]. PIFu [14] aligns individual local features at the pixel level to regress the implicit field (inside/outside probability of a 3D coordinate), enabling the 3D reconstruction from a single image. PIFuHD [15] extends PIFu to a coarse-to-fine framework so that a higher resolution input can be leveraged for achieving high fidelity. MonoPort[58] proposes a faster rendering method to accelerate inference time. Yang et.al [59] represent the pedestrian’s shape, pose, and skinning weights as neural implicit functions that are directly learned from data. PHORHUM [60] presents an end-to-end methodology for photorealistic 3D human reconstruction given just a monocular RGB image. FloRen [61] initially recovers a coarse-level implicit geometry, which is then refined using a neural rendering framework that leverages appearance flow. ARCH [16, 17] reconstructs a human avatar in the canonical pose from a single image, where the image features are aligned by a semantic deformation field. These methods hold the minimum assumptions, one unconstrained image, on the input, and show state-of-the-art performance, but they suffer from depth ambiguity and unsatisfactory artifacts on the back. It is worth noting that, PIFu can be extended to multiple views [14, 15], but camera calibration is still needed in the scenario.

NeRF has attracted much attention in learning scene representations for novel view synthesis from 2D images. Recently, some methods introduce NeRF in 3D human reconstruction. NeRF– [62] aims to learn a neural radiance field without known camera parameters. AvatarGen [63] and Zheng et al. [64] incorporate NeRF with the human body prior to enabling animatable human reconstruction. FLAG [65] focuses on the pose generation task for avatar animation from sparse observation by developing a flow-based generative model. Marwah [66] presents a two-stream methodology to infer both the texture and geometry of a person from a single image and combines the outputs of the two streams using a differentiable renderer.

Several works have released their 3D clothed human data, summarized in Tab. I. Multi-Garment [1] and THuman 2.0 [24] release a large number of scans with various body shapes and natural poses, which have been well employed in single-view and multiview reconstructions [48, 21]. However, the single scan for each subject is not suitable for our task that depends on multi-pose scans. BUFF [19] and CAPE [22] capture 4D people performing a variety of pose sequences, which are compatible with our task but the subject number is rather limited. THuman [21] consists of $200$ people in rich poses, but the captured 3D meshes lose fine-grained geometry with only $10$k points. HUMBI [23] is a large multiview image dataset of the human body with various expressions. However, the HUMBI dataset only published minimally-clothed SMPL models while the real scans are not released. Instead, our dataset has released both clothed 3D meshes and skinning weights for users to train avatar reconstruction models. MTC [20] provides a social interaction dataset but only image sequences and 3D skeletons are available. Compared with existing datasets, MVP-Human demonstrates particularity by its multi-pose scans of hundreds of people, high-quality scans ($10k$ vertices) and unique multiview real (not rendering) images.

Generally, the MVP-Human provides $15$ poses and $8$-view images for $400$ subjects, proving $6,000$ 3D scans and $48,000$ images in total. To the best of our knowledge, MVP-Human is the largest 3D human dataset in terms of the number of subjects along with high-quality 3D body meshes and multi-view RGB images.

To create the linear blend skinning weights as the supervision of SKNet, we optimize the canonical 3D model with the skinning weights for each subject following the recent SCANimate [71], shown in Fig. 2(c). Under the support of MVP-Human, where every subject has multiple scans in different poses with labeled 3D landmarks, we do not employ the original circle consistency loss and modify SCANimate in two aspects: 1) we directly optimize the chamfer errors brought by reposing, calculated by the distances between the reposed shape and the target-pose scan. 2) The reposing is guided by hand-labeled 3D landmarks rather than detected landmarks. The advanced SCANimate provides more reliable skinning weights, which are not only utilized as the target of SKNet but also used to repose the scanned T-pose shape to the strict T-pose as the target of SRNet. We also cut the edge in self-intersection regions according to [71] and fill the hole using smooth signed distance surface reconstruction [72].

PIFu [14] has demonstrated that the pixel-aligned image feature is the key to reconstructing detailed 3D surfaces. In posed 3D human reconstruction [14, 15], the image feature of a 3D point $(x,y,z)$ can be naturally retrieved on the $(x,y)$ of the image plane. However, in avatar reconstruction, the retrieval of 2D projected coordinates $\mathbf{x}_{1},\mathbf{x}_{2},...,\mathbf{x}_{V}$ for the 3D point $\mathbf{X}$ is not straightforward due to the unknown views and poses of the input frames.

To rebuild the canonical-to-image correspondence, we employ a skeleton-driven deformation method, as shown in Fig. 5. Specifically, each 3D vertex on the body surface can be transformed to any pose using a weighted influence of its neighboring bones, defined as $\mathbf{w}=\{w_{k}\}_{k=1}^{K}$ on $K$ joints $\mathbf{J}=\{j_{k}\}_{k=1}^{K}$, called Linear Blend Skinning (LBS) [18]. For each 3D surface point $\mathbf{X}$ in the canonical space, given the target pose parameters $\mathbf{R}$ represented by relative rotations and the projection matrix $\mathbf{P}$ calculated by the camera external parameters, its projected location is:

where $w_{k}$ is the skinning weight and $\mathcal{G}_{k}(\mathbf{R},j_{k})$ is the affine transformation that transforms the $k$th joint from the canonical pose to the target pose [16]. Thanks to the development of human model fitting, we can estimate the pose $\mathbf{R}$, the corresponding $\mathcal{G}_{k}(\mathbf{R},j_{k})$, and the projection $\mathbf{P}$ by fitting the SMPL to each frame [73] independently. Therefore, the main challenge is the unknown skinning weights $\mathbf{w}$, especially the 3D shape is inaccessible by now.

To address this problem, we propose the SKinning weights Network (SKNet) to learn an implicit skinning field [71] in the canonical space from multiple images. Given a 3D point $\mathbf{X}$, we build the SKNet to regress the skinning weights by:

where for each 3D point $\mathbf{X}$ in the canonical space, the Network $Net^{skin}$ takes the corresponding image feature $F(\mathbf{x}_{v},\mathbf{I}_{v})$ in each frame and regresses the skinning weights through an MLP architecture. The 2D projection on each frame $\mathbf{x}_{v}$ is achieved as in Eqn. 2, where the NN-Skin $w_{k}^{nn}$ is employed. Particularly, the SKNet outputs $K+1$ ($K$ is the joint number) channels, where the first $K$ channels are the skinning weights and the last channel holds the inside/outside probability as auxiliary supervision. The label is $(\mathbf{w}^{*},0)$ for an inside point and $(0,0,...,1)$ for an outside point. The target skinning weights are provided by the new MVP-Human dataset. Compared to NN-Skin, SKNet enables more accurate 2D projection and avoids cracks during avatar animation, which is validated in the experiments.

Transformer [74] is originally proposed to capture the correlations across input sequences, and its self-attention mechanism is naturally extendable to our task. We employ a multi-head attention network as [75] to encode the relevance between features:

where $\mathbf{F}^{I}$ is the fused image feature, $\mathbf{t}_{v}$ is the input token of the transformer, and each token is the concatenation of the retrieved image feature $F(\mathbf{x}_{v},\mathbf{I}_{v})$ and its visibility, represented by the surface normal $\mathop{\mathbf{n}_{v}}\limits^{\rightarrow}$ on the fitted SMPL. The normal can encode visibility because when the z of normal $\mathop{\mathbf{n}}\limits^{\rightarrow}$ is negative, it means the vertex faces inward and it is invisible. Otherwise, if the z is positive, the vertex is visible. We can see that the image feature $F(\mathbf{x}_{v},\mathbf{I}_{v})$ is extracted from the top-level feature map of the image encoder. Even if the skinning weights are sub-optimal, each feature has a receptive field that implicitly aligns the error.

The training and testing of avatar reconstruction models depend on a sophisticated dataset equipped with multi-pose scans, multi-view images, and the strict T-pose shape for each subject, which is described as follows.

The two sub-network follow the same MLP architecture [14] with the intermediate neuron dimensions of $(1024,512,256,128)$ and use the Stacked Hourglass [76] as the image encoder. The SKNet has a $280$-dimensional input including $256$-dimensional averaged image feature and $24$-dimensional position encoding from the 3D coordinate [77]. The output is the $24$-dimensional skinning weights plus an inside-outside probability. The input of SRNet has an additional $24$-dimension spatial features, and the output is occupancy. The transformer for image fusion follows the architecture of [74] with $4$ heads and $2$ layers, which takes $4$ image features as inputs and outputs a $256$-dimensional fused feature. During training, we first train the SKNet, whose parameters are utilized to initialize SRNet, and then jointly train the two sub-networks. The L2 loss is employed for both SKNet and SRNet. The Adam optimizer is adopted with the learning rate of $1\times 10^{-3}$, decayed by the factor of $0.1$ at the $500$th epoch, and the total number of epochs is $800$. Each mini-batch is constructed by $16$ images of $4$ subjects, with $4$ images randomly selected from one subject, and $20,000$ points are sampled around the canonical mesh with a standard deviation of $0.1$ cm.

We split the MVP-Human into a training set of $350$ subjects and a testing set of $50$ subjects. For the training set, we create the rendering images of 3D meshes following [14], where $360$ images are produced by rotating the camera around the vertical axis with intervals of $1^{\circ}$, generating $2,100,000$ images. The ground-truth canonical shape and skinning weights are used for supervision. For the testing set, $4$ images are randomly selected from the image collection of a subject as one testing sample, which is repeated $100$ times to generate $5,000$ testing samples. It is worth noting that, this testing set first enables the quantitative evaluation of real-world images. Besides MVP-Human, we also employ People-Snapshot [13] in qualitative evaluation, where the subjects are asked to rotate while holding an A-pose, whose scenario can be considered as an easier case of ours.

We evaluate the proposed baseline method for clothed 3D avatar reconstruction from multiple unspecific frames on MVP-Human dataset. To perform the comparison, we introduce a method solving a highly related task, and a modification of a state-of-the-art method.

VideoOpt (Video based Optimization) [13] takes a monocular RGB video where a subject rotates in an A-pose and generates the SMPL model with per-vertex offset by aligning rendered silhouettes to observations. This method can be generalized to the inputs without an A-pose assumption but suffers from performance deterioration due to complicated silhouettes in the unconstrained environment.

Octopus [12] fits the SMPL+D model from semantic segmentation and 2D keypoints. Similar to VideoOpt, Octopus also employs the shape offsets to represent personal details, such as hair and wrinkles, but is much faster by employing a deep learning model.

ARCH is denoted as a multi-frame extension of the state-of-the-art method ARCH [16]. The original ARCH learns to reconstruct a clothed avatar from a single image, which is extended by us to take multiple images. The main difference between ARCH and our baseline method is that we try to estimate skinning weights rather than approximating that by the nearest neighbor, and we fuse the image features by a sophisticated model to adapt to the complicated unconstrained environment.

PIFu [14] is a single-image reconstruction method that is designed to reconstruct the human body in its original posed space. However, it faces challenges in handling the MVP-Human dataset due to the lack of prior knowledge about the human body. The error is large because PIFu cannot reconstruct strict T-pose mesh as other avatar reconstruction methods.

ICON [50] performs better than PIFu due to its incorporation of prior knowledge about the human body. However, the absence of a global image encoder in ICON leads to a reduction in surface smoothness, resulting in a slightly bumpy surface for the reconstructed 3D avatar.

For quantitative comparison, we evaluate the accuracy of the reconstructed canonical 3D shape on the testing set of MVP-Human (real images), with three metrics similar to [14], including the normal error, point-to-surface (P2S) distance, and Chamfer error. For fairness, the ARCH is fine-tuned on the MVP-Human training set. As shown in Table II, we achieve the best results.

For qualitative comparison, Fig. 7 shows the reconstructed shapes from rendering and real images of MVP-Human, and Fig. 8 shows the results in People-Snapshot, respectively. It can be seen that Octopus [12] and VideoOpt [13], which attempt to learn SMPL offset, have difficulty in capturing fine-grained geometry due to the limited representation power of the shape space. ARCH retains some distinctive shapes but suffers artifacts like discontinuous surfaces and distorted body parts, which come from misaligned image features.

PIFu [14] cannot well capture poses due to the depth ambiguity. ICON generates 3D shapes with lumpy surfaces, and the overall shapes are similar to SMPL bodies without clothes. In contrast, our method generates more realistic avatars with estimated skinning weights and better image fusion. Besides, our method shows good generalization by reconstructing pants in People-Snapshot even without the clothes in the training set. The reconstruction results on the challenging poses including walking, standing, and running are illustrated in Fig. 9. Existing methods for single-image reconstruction often fail in generating a complete human body, resulting in a backside that appears overly smooth. When introducing multi-view images for reconstruction, current methods typically rely on accurate camera calibration, which may not be practical in real-world scenarios. Our proposed method enables the reconstruction of an animatable human body from multiple images with unconstrained poses and cameras, providing a more practical solution. Besides, We show some failure cases when SMPL fitting is inaccurate in Fig. 10. The primary limitation of our proposed method is the potential risk that inaccurate pose estimation brings errors in mapping 3D points to different image planes for feature fusion. This may cause irrelevant features to be fused, which could potentially deteriorate the quality of the 3D reconstruction.

We also evaluate the effectiveness of better skinning weights in animation. As shown in Fig. 11, compared with the common Nearest Neighbor Skinning weights (NN-Skin) [16], we learn continuous skinning weights which can articulate a mesh smoothly while retaining coherent geometric details.