LiveHPS: LiDAR-based Scene-level Human Pose and Shape Estimation
in Free Environment

Optical motion capture technology has advanced from initial marker-based systems [52, 53, 40] that rely on camera-tracked markers to reconstruct a 3D mesh, to markerless systems [24, 36, 2, 7, 14, 46, 47, 41, 50] which track human motion using cameras through feature detection and matching algorithms. Despite they can get high-accuracy results, these systems are often expensive and require elaborate setup and calibration. To mitigate these challenges, monocular mocap methods using optimization [20, 32, 6, 30] and regression [25, 26, 28, 67], along with template-based, probabilistic [60, 17, 59, 18, 19], and semantic-modeling techniques [29], have emerged to address monocular system limitations. Nonetheless, these approaches still suffer from inherent light sensitivity and depth ambiguity. Some strategies[48, 3, 57, 66, 16] incorporate depth cameras to resolve depth ambiguity, yet these cameras have a limited sensing range and are ineffective in outdoor scenes.

Unlike optical systems, inertial motion capture systems [58] are not affected by light conditions and occlusions. They generally need numerous IMUs attached to form-fitting suits, a setup that can be heavy and inconvenient, motivating interest in more sparse configurations, such as six-IMU setting [54, 21, 63, 64] and four-IMU setting [45]. However, these methods suffer from drift errors over time, cannot provide precise shape and global translation, and require wearable devices, not practical for daily-life scenarios.

With precise long-range depth-sensing ability, LiDAR has emerged as a key sensor in robotics and autonomous vehicles [10, 69, 70, 65, 42]. LiDAR can provide precise depth information and global translation in expansive environments, remaining uninfluenced by lighting conditions, enabling robust 3D HPS. Recently, PointHPS [9] provides a cascaded network architecture for pose and shape estimation from point clouds. However, it is applicable for dense point clouds rather than sparse LiDAR point clouds. LiDARCap [33] employs a graph-based convolutional network to predict daily human poses in LiDAR-captured large-scale scenes. MOVIN [23] presents a generative method for estimating both pose and global translation. However, these methods cannot predict full SMPL parameters (pose, shape, and global translation) and are fragile for complex real scenarios with occlusion and noise.

Data-driven 3D HPS have gained traction in recent years benefiting from extensive labeled datasets. Indoor marker-based datasets like Human3.6M [22] and HumanEva [49] use multi-view camera systems to record daily motions. AMASS [37] unifies these datasets, providing a standardized benchmark for network training. Marker-less datasets such as MPI-INF-3DHP [38] and AIST++ [34] capture more complex poses without constraint of the wearable devices, all above datasets are still confined to indoor settings. Outdoor motion capture datasets like PedX [27] and 3DPW [55] capture motions in the wild but lack accurate depth information, hindering scene-level human motion research. HuMMan [8] constitutes a mega-scale database that offers high-resolution scans for subjects, and MOVIN [23] provides motion data from multi-camera capture system with point clouds, but both datasets are limited in short-range scenes.  [12, 13, 62] are proposed for human motion capture in large-scale scenes using environment-involved optimization, but they are limited in a single-person setting. Recently, LiDARHuman26M [33] and LIP [45] provide LiDAR-captured motion dataset in large scenes, but both datasets exclusively provide pose parameters of SMPL in single-person scenarios. In contrast, we propose a large-scale LiDAR-based motion dataset with full SMPL parameter annotations. It comprises a variety of challenging scenarios with occlusions and interactions among multiple persons and objects, which has great practical significance.

LiveHPS takes a consecutive sequence of single-person point clouds with $T$ frames as input. As raw point clouds have various numbers of points at different times $t$, we implement normalization process by resampling each frame to a fixed $N_{fps}=256$ points utilizing the farthest point sampling algorithm (FPS) and subtracting the average locations $\mathbf{loc}(t)\in\mathbb{R}^{3}$ of the raw data. $\mathbf{P}(t)\in\mathbb{R}^{3N_{fps}}$ denotes the pre-processed input at time $t$.

We define $N_{J}$ as the number of body joints and $N_{V}$ as the number of body vertices on SMPL mesh; $\hat{\mathbf{J}}(t),\ \widetilde{\mathbf{J}}^{GT}(t)\in\mathbb{R}^{3N_{J}}$ as predicted and ground-truth root-relative joint positions at time $t$, repectively; $\hat{\mathbf{V}}(t),\widetilde{\mathbf{V}}^{GT}(t)\in\mathbb{R}^{3N_{V}}$ as predicted and ground-truth vertex positions. Our network prediction consists of $\hat{\mathbf{\theta}}(t)\in\mathbb{R}^{6N_{J}},\hat{\mathbf{\beta}}\in\mathbb{R}^{10}$ and $\hat{Tr}(t)\in\mathbb{R}^{3}$, the pose, shape, and global translation parameters of SMPL. $\mathbf{\theta}^{GT}(t),\mathbf{\beta}^{GT}$ and $Tr^{GT}(t)$ are corresponding ground truth. We use 6D-rotation-based pose representation.

For the input of our pre-processed consecutive point clouds, we extract the point-wise feature following the PointNet-GRU structure proposed by LIP [45] and regress the human body joint positions with an MLP decoder. Considering the irregular distribution of LiDAR point clouds vary across different capture distances and devices, and are also effected by occlusion and noise (Fig. LABEL:fig:teaser), we design a Vertex-guided Adaptive Distillation (VAD) mechanism to unify the point distribution to facilitate the training of the network and improve the robustness. Because the vertices of SMPL mesh have relatively regular representation, we make diverse point distributions aligned with the mesh vertex distribution in high-level feature space by distillation, as Fig. 1 shows.

Firstly, we use the global translation $Tr^{GT}(t)$ to align the LiDAR point cloud $\mathbf{P}(t)$ with the ground truth mesh vertex $\widetilde{\mathbf{V}}^{GT}(t)$ and utilize k-Nearest-Neighbours (kNN) algorithm to sample the corresponding vertices, defined as $\widetilde{\mathbf{V}}^{GT}_{pc}(t)$. Then, we use $\widetilde{\mathbf{V}}^{GT}_{pc}(t)$ to pre-train a vertex body tracker to regress the joint positions $\hat{\mathbf{J}}_{v}(t)$. We use the mean squared error (MSE) loss ${L}_{mse}(\hat{\mathbf{J}}_{v})$ for supervision:

$\displaystyle\mathcal{L}_{mse}(\hat{\mathbf{J}}_{v})=\sum_{t}\parallel\hat{\mathbf{J}}_{v}(t)-\widetilde{\mathbf{J}}^{GT}(t)\parallel_{2}^{2}.$

(1)

Subsequently, we input sequential point clouds $\mathbf{P}(t)$ and their corresponding vertex data $\widetilde{\mathbf{V}}^{GT}_{pc}(t)$ into two independent body trackers to obtain point-wise features $F_{p}(t)\in\mathbb{R}^{k}$ and $F_{v}(t)\in\mathbb{R}^{k}$, respectively, where $k=1024$. Notably, two point-based body tracker networks share distinct weights and we freeze the pre-trained parameters of the vertex body tracker during training. To align real point distributions with the regular vertex distribution, we employ a pose-prior consistency loss $\mathcal{L}_{pc}$ to minimize the high-level feature distance between LiDAR point clouds and guided vertices. The distillation procedure enables our feature extractor to own the ability to maintain insensitivity under vastly differentiated data distribution. Finally, we leverage an MLP decoder to predict the joint positions $\hat{\mathbf{J}}_{p}$. A combined loss $\mathcal{L}_{prior}$ consisting of $\mathcal{L}_{mse}(\hat{\mathbf{J}}_{p})$ and $\mathcal{L}_{pc}$ is utilized to train the network, which is formulated as below

$\displaystyle\mathcal{L}_{mse}(\hat{\mathbf{J}}_{p})=\sum_{t}\parallel\hat{\mathbf{J}}_{p}(t)-\widetilde{\mathbf{J}}^{GT}(t)\parallel_{2}^{2},$

(2)

$\displaystyle\mathcal{L}_{pc}=\sum_{t}F_{v}(t)\log(\frac{F_{v}(t)}{F_{p}(t)}),$

(3)

$\displaystyle\mathcal{L}_{prior}=\lambda_{1}\mathcal{L}_{mse}(\hat{\mathbf{J}}_{p})+\lambda_{2}\mathcal{L}_{PC},$

(4)

where $\lambda_{1}$ and $\lambda_{2}$ are hyper-parameters, and we set $\lambda_{1}=1$ and $\lambda_{2}=10^{3}$ in our experiments. During inference, the VAD process is not required.

We have already obtained the joint positions of human poses from the point-based body tracker. Considering that human motions are coherent at time sequence and different joints of the human body usually execute the action with relative dynamic constraints, we propose a Consecutive Pose Optimizer (CPO) (Fig. 2) to refine the body skeleton using consecutive joint-wise geometry features and relationship features in temporal and spatial spaces, which can further reduce the effect of incomplete and noised point clouds.

Specifically, we utilize the concatenation of point-wise feature $F_{p}(t)\in\mathbb{R}^{k}$ and the predicted joint positions $\hat{\mathbf{J}}_{p}(t)$ as the initial joint-wise feature input. To capture the motion consistency in sequence, we use linear transformations to generate $Q(t)$, $K(t)$, and $V(t)$ in each frame and conduct temporal interaction to learn the motion-consistent feature $F_{t}(t)\in\mathbb{R}^{k_{2}}$ for each joint, where $k_{2}=256$. This temporal interaction process guides the estimation of more reasonable continuous human motions, especially for occluded situations. Then, we use the dynamic and geometric constraints among joints to further enhance the joint feature via spatial feature interaction. The input $F_{j}(n_{j}\in N_{J})\in\mathbb{R}^{k+k_{2}+3}$ consists of the point-wise feature $F_{p}(t)\in\mathbb{R}^{k}$, temporal interaction feature $F_{t}(t)\in\mathbb{R}^{k_{2}}$, and each joint feature $\hat{\mathbf{J}}_{p}(n_{j}\in N_{J})\in\hat{\mathbf{J}}_{p}(t)$. We generate $Q(n_{j})$, $K(n_{j})$ and $V(n_{j})$ with linear mapping for each joint and conduct the spatial joint-to-joint interaction to get the enhanced feature $F_{ts}(t)\in\mathbb{R}^{k_{3}}$, where $k_{3}=512$. The feature interaction matrix can be formulated as:

$\displaystyle\mathcal{F}_{interaction}=\operatorname{softmax}\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right)V.$

(5)

Finally, we regress the refined joint positions $\hat{\mathbf{J}}_{refine}(t)$ from the enhanced feature and the loss function is

$\displaystyle\mathcal{L}_{mse}(\hat{\mathbf{J}}_{refine})=\sum_{t}\parallel\hat{\mathbf{J}}_{refine}(t)-\widetilde{\mathbf{J}}^{GT}(t)\parallel_{2}^{2}.$

(6)

In the last stage, we propose an attention-based multi-head solver to regress the SMPL [35] parameters $\hat{\mathbf{\theta}}(t)$, $\hat{\mathbf{\beta}}$, $\hat{Tr}(t)$ from refined joint positions and the input point cloud. Because the pose and the shape reflect local geometry of human body, they can be determined by root-relative joint features obtained in last stage. We utilize the same network structure as CPO as the pose solver and shape solver to get $\hat{\mathbf{\theta}}(t)$ and $\hat{\mathbf{\beta}}$. However, the global translation could be obtained only from the root-relative local geometry features. Previous methods [33, 45] usually take the average position of the body point cloud as the global location or directly regress the global translation. However, due to the interference of occlusion and noise, their predicted results are unstable in consecutive frames. In contrast, we simplify the task of predicting global translation to predict the bias between the average position of point cloud and the real 3D location. Thus, we propose a Skeleton-aware Translation Solver underpinned by a cross-attention architecture, which intelligently integrates skeletal and original point cloud data to get more accurate translation estimation. We employ point cloud $\mathbf{P}(t)$ and refined root-relative joint positions $\hat{\mathbf{J}}_{refine}(t)$ as the input, utilizing the cross-attention to match the geometric information of joints with the point cloud. We generate the $Q(t)$ from refined joint positions and $K(t)$, $V(t)$ from point cloud. The feature interaction matrix can be formulated as Eq. 5. The decoder outputs the bias, which can be added to the average location $\mathbf{loc}(t)$ of raw point cloud to get the global translation $\hat{Tr}(t)$. Finally, we use SMPL model to generate the human skeleton joint positions and mesh vertex positions as below.

$\displaystyle\hat{\mathbf{J}}_{smpl}(t),\hat{\mathbf{V}}_{smpl}(t)=\operatorname{SMPL}(\hat{\mathbf{\theta}}(t),\hat{\mathbf{\beta}},\hat{Tr}(t)).$

(7)

The loss function for the multi-head solver is formulated as:

	$\displaystyle\mathcal{L}_{solver}=$	$\displaystyle\lambda_{3}\mathcal{L}_{mse}(\hat{\mathbf{J}}_{smpl})+\lambda_{4}\mathcal{L}_{mse}(\hat{\mathbf{V}}_{smpl})$		(8)
	$\displaystyle+$	$\displaystyle\lambda_{5}\mathcal{L}_{mse}(\hat{\mathbf{\theta}}(t))+\lambda_{6}\mathcal{L}_{mse}(\hat{\mathbf{\beta}}$
	$\displaystyle+$	$\displaystyle\lambda_{7}\mathcal{L}_{mse}(\hat{Tr}(t))+\lambda_{8}\mathcal{L}_{SUCD},$

where $\lambda_{3}$, $\lambda_{4}$, $\lambda_{5}$, $\lambda_{6}$, $\lambda_{7}$ are hyper-parameters with $\lambda_{3}=\frac{100}{N_{j}}$, $\lambda_{4}=\frac{100}{N_{v}}$, $\lambda_{5}=\frac{1}{5}$, $\lambda_{6}=1$, $\lambda_{7}=1$ and $\lambda_{8}=10^{3}$.

Because the raw point cloud contains the real pose, shape, and global translation information, it can be taken as an extra supervision which is ignored by previous methods. In particular, we introduce a novel scene-level unidirectional Chamfer distance (SUCD) loss by calculating the unidirectional Chamfer distance from the raw point cloud to the predicted mesh vertices. It provides a comprehensive evaluation for all predicted SMPL parameters, denoted as

$\displaystyle\mathcal{L}_{SUCD}=\sum_{t}\frac{1}{|\mathbf{P}(t)|}\sum_{x\in\mathbf{P}(t)}\min_{y\in\hat{\mathbf{V}}_{smpl}(t)}|x-y|_{2}^{2},$

(9)

Considering that the indoor multi-camera panoptic studio can provide high-precision full SMPL parameter annotations (pose, shape and global translation) and outdoor scenes are large-scale and suitable for real applications, we have two capture systems as shown in Fig. 3. For the first one, we set up a 76-Z-CAM system to obtain SMPL ground truth and three OUSTER-1 LiDARs at varied distances to get LiDAR data. Notably, we arrange other performers outside the studio to simulate occlusions in real-life scenarios. For the second one, we built three sets of LiDAR-camera capture devices, including a 128-beam OUSTER-1 LiDAR and a monocular Canon camera for each, in different locations to capture multi-view and multi-range visual data, and provide the global translation ground truth. The performer is equipped with a full set of Notiom equipment (17 IMUs) to obtain the pose ground truth. Particularly, the shape parameters of outdoor performers are captured in panoptic studio in advance. The capture frequencies for the LiDAR, Z-CAM, Canon camera, and IMU are set at 10Hz, 25Hz, 60Hz, and 60Hz, respectively. All the data are calibrated and synchronized.

The detailed comparison with existing public datasets is presented in Tab. 1. FreeMotion has several distinctive characteristics and we summarize three main highlights below.

Free Capture Scenes. Diverging from previous datasets focused on single-person HPS, FreeMotion is captured in real unconstrained environments, which involves diverse capture scales, multi-person activities (sports, dancing shows, fitness exercises, etc.), and human-object interaction scenarios. The large-scale human trajectories, occlusions, and noise in our data all bring challenges for precise human global pose and shape estimation, thereby promoting the envelope of HPS technology for real-life applications.

Diverse Data Modalities and Views. FreeMotion offers multi-view and multi-modal capture data, including LiDAR point clouds, RGB images, and IMU measurements, providing rich resources for the exploration of single-modal, multi-modal, single-view, multi-view HPS solutions.

Complete Scene-level SMPL Annotations. Existing LiDAR-based motion datasets usually provide pose annotations using dense IMUs and lack annotations for accurate human shape and global translation. FreeMotion remedies this by providing full SMPL parameters annotations(pose, shape, translation), as shown in Fig. 3. We capture a variety of natural human motions across long distances, involving 20 individuals with varying body types engaging in 40 types of actions. Details are in appendix. Accurate and complete annotations in rich scenarios can comprehensive evaluation for algorithms and benefit many downstream applications.

To enrich the dataset with various poses and shapes for pretraining, we follow LIP [45] to create synthetic point clouds from SURREAL [51], AIST++ [34], and portions of AMASS [37], including ACCAD and BMLMovi. We simulate occlusions by randomly cropping body parts of point clouds. It consists of $2,378$k frames and $3,118$ body shapes. Note that statistics in Tab. 1 do not include the synthetic data. Detailed process is shown in appendix.

We adhere to privacy guidelines in our research. The LiDAR point clouds naturally protect privacy by omitting texture or facial details. Additionally, we mask faces in RGB images to uphold ethical standards in our dataset.

We build our network on PyTorch 1.8.1 and CUDA 11.1, trained over 200 epochs with batch size of 32 and sequence length of 32, using an initial learning rate of $10^{-3}$, and AdamW optimizer with weight decay of $10^{-4}$. The process was run on a server equipped with an Intel(R) Xeon(R) E5-2678 CPU and 8 NVIDIA RTX3090 GPUs. For training, we used clustered and manually annotated human point cloud sequences from raw data, while for testing, we employ sequential point clouds of human instances processed by a pre-trained segmentation model [56]. As for the dataset splitting, we take training set of FreeMotion, Sloper4D, and synthetic dataset including training set of SURREAL, AIST++, ACCAD, and BMLMovi for training.

We evaluate LiveHPS against other state-of-the-art (SOTA) LiDAR-related methods [33, 45, 23] on FreeMotion and several public datasets [51, 13, 33, 45, 62, 4, 61] to demonstrate its superiority in capturing human global poses and shapes in large-scale free environment, even with severe occlusions and noise. Our LiveHPS achieves SOTA performance as shown in Tab. 2. The J/V Err(P) and Ang Err only relate to the pose parameter estimation, we surpass LiDARCap [33], LIP [45] and MOVIN [23] by an obvious margin. For fair comparison, we only use the LiDAR branch of LIP. As the pioneer to fully estimate SMPL parameters (pose, shape, global translation) for LiDAR-based HPS, we develop a shape regression head with the same architecture of their pose regression head for fair comparison with other methods [33, 45, 23], the translation prediction of LiDARCap is the average of point cloud. Visual comparisons in Fig. 4 further highlight our method’s superiority in global pose and shape estimations, yielding results that closely mirror ground truth. Other methods struggle in situations with occlusions and noise, as exemplified in challenging scenes from Sloper4D [13] and FreeMotion in Fig. 4. MOVIN [23] estimate translation based on velocity regression, it is not applicable on synthetic data SURREAL without real trajectories. Our LiveHPS demonstrates robust performance against noise like carried objects, as demonstrated in FreeMotion’s left case in Fig. 4.

Tab. 3 illustrates our cross-dataset evaluation to validate the generalization capability of LiveHPS by directly testing on other datasets. LiDARHuman26M [33] and LIPD [45] only offer pose parameters. CIMI4D [62] provides pose, shape, and translation, but the translation is not that precise as shown in the third row of Fig. 5. SemanticKITTI [4] and HuCenLife [61] are large-scale datasets for 3D perception, not providing SMPL annotations. Thanks to our VAD module’s ability to harmonize diverse human point cloud distributions and CPO module’s ability to model geometric and dynamic human features, our method can achieve SOTA performance on these cross-domain datasets, even in challenging cases with extreme occlusions, as Fig. 5 shows.

We first validate the effectiveness of each module in LiveHPS. Then, we evaluate inner designs of each module to verify the effectiveness of detailed structures.

Tab. 4 shows the performance of our method with different network modules, demonstrating the necessity of our vertex-guided adaptive distillation (VAD) and consecutive pose optimizer (CPO) modules. We also illustrate ablation details of attention-based temporal and spatial feature enhancement in CPO, showing that the combination of temporal and spatial feature interaction performs best. We also conduct experiments to validate our attention-based multi-head SMPL solver. Our pose and shape solver, using the same network as CPO, outperforms ST-GCN from LiDARCap [33] and GRU from LIP [45] by fully utilizing the global temporal context and local spatial relationship existing in consecutive body joints. For the translation solver, the average of point cloud can reflect the coarse translation but it is very unstable with the distribution of point cloud changes. Compared with global velocity estimation utilized in MOVIN [23], our skeleton-aware translation solver directly estimates translations without error accumulation. Moreover, unlike GRU-based pose-guided corrector in LIP [45] which overlooks relationship between the skeleton and point cloud, our approach performs better by considering the relationship and more spatial information.

We assess the generalization capability of LiveHPS across varying lengths of input point cloud sequences and across different point numbers on human body in each frame, as Tab. 5 shows. Our method performs better with increasing sequence length but maintains good accuracy even with short inputs. In addition, our method performs relatively robust even for the situation with 100 points on the human body, which means far distance (about 15 meters) to LiDAR or severe occlusion. Fig. LABEL:fig:teaser and Fig. 6 show our method is practical for in-the-wild scenarios, capturing human motion in large-scale scenes day and night with real-time performance up to 45 fps. This strongly demonstrates the feasibility and superiority of our method in real-life applications. More application results are in appendix.

When we capture data in both capture systems, we divide the capture processing into three stages. In the first stage, the actor performs different kinds of actions alone without occlusion and noise. In the second stage, we arrange other persons to interact with main actor and perform specific actions depending on the scene characteristics, such as basketball court, meeting room, etc. The interactions bring real occlusions. In the last stage, the main actor interacts with some objects, such as the balls, package skateboard, etc. The objects bring real noise.

Point Cloud. The raw point clouds are captured from 128-beam OUSTER-1 LiDAR. The LiDAR features a 360° horizon field of view (FOV) $\times$ 45° vertical FOV. For the training dataset, we first record the point cloud of static background, and then set a threshold to delete the background points to get the point clouds of actors. For the processed point clouds, we use DBSCAN [15] cluster to get instance point cloud of each person. Then we employ the Hungarian matching algorithm for point cloud-based 3D tracking. Moreover, to promise high-quality annotations of our dataset, we review the segmentation point cloud sequence of each person. For the challenge and complex scene, such as the meeting room, which contains many extra objects, we use manually annotation for each point cloud sequence. For the testing dataset, we train the instance segmentation model [56] in our training set and inference in our testing set while the tracking processing is the same by above methods.

SMPL annotations. For the first capture system, we use multi-camera method [1] to generate the ground-truth SMPL parameters(Poses and translations). The predicted shape parameters is not accurate by this method. As the Figure 7 shows, we use the multi-camera method [43] to reconstruct the human mesh and fit the human mesh to the SMPL model in template pose for more accurate shape parameters. For the second capture system, we use full set of Noitom [39] equipment(17 IMUs) to get the SMPL pose parameters and the shape parameters of performer are captured in the panoptic studio in advance. In outdoor capture scenes, we follow  [13] to fit the SMPL mesh to the point cloud for accurate translation parameters.

Synchronization The synchronization among various-view and modal sensors is accomplished by detecting the jumping peak shared across multiple devices. Thus, actors are asked to perform jumps before the capture. We subsequently manually identify the peaks in each capture device and use this timestamp as the start for time synchronization.

Calibration Our capture systems contain LiDARs, Cameras and IMUs. For the three LiDARs calibration, we select the static background point clouds in the same timestamp, and manually to do the point cloud registration. For multi-camera calibration, we use  [43] to calibrate all cameras. The calibration of IMUs is the algorithm of Noitom [39]. For calibration of each sensor, in the first capture system, we generate the SMPL human mesh from multi-camera data, the coordinate of human mesh is the same with the multi-camera coordinate, then we follow LIP [45] to utilize the ICP [5] method between the segmentation human point cloud and the human mesh vertices for LiDAR-camera calibration. In the second capture system, we utilize the Zhang’s Camera Calibration Method [68] for LiDAR-camera calibration, and the human SMPL mesh’s coordinate is the same with the coordinate of IMUs. We use the above ICP method for LiDAR-IMU calibration.