QuestSim: Human Motion Tracking from Sparse Sensors with Simulated Avatars

Generating 3D full body poses based on camera images can be used for motion tracking. This image can be used to predict parameters for a parameterized and differentiable statistical human body model such as SMPL (Loper et al., 2015; Rong et al., 2021; Kanazawa et al., 2019; Xu et al., 2019). Oftentimes, camera pixels are preprocessed to extract body keypoints (Cao et al., 2019) or body correspondences from the image (Güler et al., 2018). One of the difficulties when using monocular cameras is the depth ambiguity, e.g. a short person closer to the camera looks identical to a taller person further away. This depth ambiguity can cause reconstructed 3D poses to accurately match the camera image when viewed from the same angle, whereas different views reveal unnatural leaning, scale and pose. Furthermore, if every frame is reconstructed individually, continuity between frames can be difficult and can cause poses to jitter. Constraining the poses through physics-based priors can help to mitigate these issues (Rempe et al., 2021).

Another popular approach is to use sensors mounted on a user’s body, for example inertial measurement units (IMUs). They are small, lightweight and don’t exhibit occlusion, since they are based on acceleration signal, versus vision. This allows them to be used across rooms, as well as outdoors and makes them agnostic to light and weather conditions. An offline method using IMUs was proposed by Marcard et al. (2017), where the system optimizes the pose parameters of the SMPL model so that it matches the sensor’s signal. The system performs best when it can access the entire sensor signal trajectory. In order to overcome this offline constraint, deep learning based approaches using mocap data paired with IMU signals were proposed. These models produce a local joint angle pose in realtime  (Huang et al., 2018; Nagaraj et al., 2020). Methods that estimate the full pose, including global 6D root from IMU signal were explored by (Yi et al., 2021; Jiang et al., 2022b). Since kinematic pose prediction can be jittery or drift, these methods use predicted foot contacts, Stationary Boundary Points, or terrain prediction to improve the synthesized poses. Apart from IMUs, sensors based on electromagnetic (EM) fields have also been used to reconstruct full-body poses (Kaufmann et al., 2021). However, most of the above approaches require sensors attached to the lower body.

As Head Mounted Devices (HMDs) for AR/VR are becoming more widely available, methods that generate full-body poses only from HMD and controllers are being explored. Dittadi et al. (2021) proposed a framework based on a variational autoencoder (VAE), where a VAE is first learned with all poses in the dataset, then the decoder is combined with another encoder which uses IMU signals as input. Aliakbarian et al. (2022) utilized a flow-based model instead of VAEs, where the invertible nature of flow-based models enables learning a shared latent space in a single learning phase. Jiang et al. (2022a) uses a Transformer architecture to predict the global skeleton state from only the HMD and controller poses. However, since these approaches are kinematic and don’t enforce physical constraints, they can suffer from artifacts such as foot-skating and jitter.

Many of the previous approaches are kinematic, meaning that the model has no notion of masses, inertias or forces as it synthesizes the poses. However, obeying physical laws makes synthesized motions more believable. This physics prior is especially important when the problem is under-constrained, where many solutions fulfill the sensor constraints but only a few are desirable. This is the case when using sparse sensor signal (head and controllers) to reconstruct physically accurate full-body poses.

Shimada et al. (2020) optimized the output kinematic motion predicted from a monocular video where physics laws are used as soft constraints. Yi et al. (2022) added a physics-based optimizer as a refinement process to further improve motions generated by kinematic methods such as TransPose (Yi et al., 2021), showing impressive results for IMU-based pose reconstruction. We show that we can generate motions of comparable quality without sensors on the lower body.

However, many methods still allow physics to be slightly violated, e.g. they are not enforced as hard-constraints. This can cause the posture of the characters to be abnormally tilted or for generated motions to appear to be floating in the air. We build much of our work on RL-based imitation learning, which adds a physics simulator as the final step before the pose generation, thereby enforcing physics as hard-constraints that cannot be violated (Peng et al., 2018a). The controller outputs either target joint angles or torques which are used by the physics simulator to generate the next pose of the simulated character. This stream of research traditionally focused on imitating motions from dense, full-body user observations. Here, a variety of methods have explored how a single policy can imitate diverse motions, such as walking, jogging, break-dancing etc (Won et al., 2020; Park et al., 2019; Bergamin et al., 2019; Fussell et al., 2021; Chentanez et al., 2018). This becomes especially relevant for motion tracking, where we cannot know in advance what type of motion the user will perform, but require the policy to be able to track it. Another line of research is training on large mocap datasets to generate fundamental ”building blocks” of motions, that can later be reused to achieve other downstream tasks  (Peng et al., 2021; Merel et al., 2020; Won et al., 2021; Peng et al., 2019). In the context of motion tracking these insights could be used to track the motion using a specific style.

Reinforcement Learning has also been used to imitate kinematic motions generated from sparser observations, such as monocular video (Peng et al., 2018b; Yu et al., 2021). It has been successfully combined with egocentric video to synthesize poses and object interaction (Luo et al., 2021) and predict future motion (Yuan and Kitani, 2019). Luo et al. (2021) learned a universal controller and a kinematic policy and used it to generate simulated motions from a single-view egocentric video. In contrast to these video-based approaches, we show motion synthesis using only the 6 DoF state of the headset and controllers. We use a simple architecture, consisting of a single MLP trained end-to-end. Since our physics simulation doesn’t use additional non-physical forces, our approach generates high-quality and believable motions.

We use reinforcement learning to train the character. The goal is to find a policy $\pi_{\theta}$ to maximise the expected discounted return

where $\theta$ represents the weights of a neural network, $\gamma\in[0,1]$ is the discount factor, and $\tau=(s_{0},a_{0},\dots,s_{T+1})$ represents trajectories collected by the policy interacting with the environment. The probability for a specific trajectory $\tau$ when using the current policy $\pi_{\theta}$ is given by $P(\tau|\theta)=P(s_{0})\prod_{t=0}^{T}P(s_{t+1}|s_{t},a_{t})\pi_{\theta}(a_{t}|s_{t})$, where $P(s_{0})$ is the initial state distribution and $P(s_{t+1}|s_{t},a_{t})$ represents the dynamics of the physics simulator. This probability is used to calculate the expectation $\mathds{E}_{\tau\sim\pi_{\theta}}$ which quantifies the expected return. To learn the policy weights $\theta$, we use the proximal policy optimization (PPO) algorithm (Schulman et al., 2017).

The goal is to reconstruct the full body pose of a user from sparse user observations $o_{\text{user}}$ as shown in Figure 2. This output pose $s_{t}$ is generated by a physics simulator. The simulator is driven by joint torques, which are produced by a neural network termed ”Policy”. We use Reinforcement Learning, together with an imitation objective (Peng et al., 2018a), to train the policy to produce torques that track a user. During training we use a scalar reward $r_{t}$ that captures the goal of producing a pose $s_{t}$ that is as close as possible to the corresponding ground-truth pose $s_{t,\text{gt}}$ from the motion database.

We hypothesize this architecture works well with sparse input data because of the high quality full body supervision signal during training, as well as because of the feedback of the simulated state back into the policy. Since the user observations are sparse and don’t provide much information about the user, considering the current state of the physically simulated character significantly reduces ambiguity on possible next poses. This allows the policy to use this internal state to decide on optimal actions. For kinematic tracking approaches, this feedback loop is conceptually similar to Recurrent Neural Networks (RNNs) or ”Pose Priors”. However, complex architectures such as RNNs or Transformers are not required in our case and a 3-layer MLP policy is sufficient.

The policy requires observations as input in order to determine the torque at every simulation step. The observations can be split into three sections discussed in the following: the observations coming from the simulated character, the sparse observations coming from the sensors worn by the user, and the scale of the user.

The simulated avatar has 33 degrees-of-freedom (DoF). It is fully observable, allowing the policy access to any values that are helpful for determining the torques. We use the joint angles $o_{\text{sim},q}\in\mathbb{R}^{33}$ and joint angle velocities $o_{\text{sim},\dot{q}}\in\mathbb{R}^{33}$. Even though redundant, we also give the policy access to the Cartesian positions and orientations of each link, which speeds up training. All the Cartesian positions $o_{\text{sim},x}\in\mathbb{R}^{16\times 3}$ are expressed with respect to frame $S$, which is located on the floor below the avatar and rotates according to its heading direction (see Figure 2). This allows the policy to learn torque mappings independent of the heading direction. The link orientations $o_{\text{sim},R}\in\mathbb{R}^{16\times 6}$, also in frame S, are encoded by the first two columns of their rotation matrices. This has been shown to be a favorable orientation representation for neural networks compared to quaternions or angle axis, which can introduce discontinuities. We also observe dynamic quantities such as contact forces of each foot $o_{f}\in\mathbb{R}^{2\times 3}$, which allows the policy to reason about contact states when determining the torques. The policy also observes the simulated avatar’s linear and angular velocity of each of the links, which is necessary in order to take the inertia of the character into account when producing torques. In total the simulated avatar is observed by $o_{sim}\in\mathbb{R}^{312}$.

Training the policy requires sensor data of the user paired with ground-truth poses $s_{t,\text{gt}}$ from which the reward is computed. Instead of recording a new dataset capturing the full-body pose of a user (e.g. with a marker-based setup) while wearing a headset, we synthetically generate this paired data. We offset the ground-truth head and wrist joints to emulate the position and orientation of a headset and left and right controllers as if the subjects were equipped with the devices. Since the real sensor signal is sufficiently clean, noise was not added to the offsets.

Our in-house Mocap data consists of  8 hours of motion clips of 172 subjects. Specifically, the dataset contains 130 minutes of walking, 110 minutes of jogging, 80 minutes of casual conversations with gestures, 90 minutes of whiteboard discussion and 70 minutes of balancing. Existing datasets lacked either subject diversity or motion diversity.

The sensor data, either coming from the real headset or synthetically generated for training, is given by the position and orientation of the headset $h$, the left controller $l$ and the right controller $r$. Like the Cartesian observations of the simulated character, all user positions and orientations are relative to frame $S$ as shown in Figure 2. The orientations ${}_{S}R_{i}\in\mathbb{R}^{6}$ denote the first two columns of the rotation matrix of the head, left and right controller relative to frame $S$ (see Sec. 3.3 for details).

The policy also has access to 6 future user observations. This gives the simulated avatar the ability to better anticipate and thereby track a user’s motion by knowing what they will do next. The trade-off is that relying on future poses introduces a 160ms latency, which makes real-time tracking less responsive. In total the pose observations coming from the user are $o_{\text{user}}\in\mathbb{R}^{6\times 3\times(3+6)}\in\mathbb{R}^{162}$.

Many imitation learning approaches generate policies that are specific to one character of a particular scale. In order to deliver a general motion tracking solution, we want to be able to track users of any scale (tall or short). One approach is to generate individual policies per user scale and then blend them during inference. The downside is that this requires a unique policy for every user scale and there might not be sufficient mocap training data available for each user. So instead, we learn a single policy which generalizes to various user scales. To achieve this we give the policy access to the user scale in form of a scalar value, which specifies the user’s height in meters.

This allows the policy to learn to adjust torques based on the user, for instance apply larger torques when tracking taller users with larger masses and inertia.

During training, when imitating a clip recorded by one of the 172 individual subjects, we use the first pose in the clip, which is an A-pose, to extract the height of the subject. This single scale value is used to initialize a simulated character with approximately the same scale. During inference, when a user wears the headset, we require the user to initially stand upright and use the height estimated by the headset to initialize their avatar.

The reward $r_{t}$ is used to generate a desired behavior of the character. The goal for the simulated character is to imitate a user’s motion as close as possible. We build on the imitation reward introduced by Peng et al. (2018a). Since during training the sparse observations are synthetically generated from the full-body mocap pose, the corresponding full-body pose $s_{t,\text{gt}}$ is known and can be used to formulate this reward. This way the policy has dense supervision during training while requiring only sparse data during inference. Our reward function is given by the dot product

where the terms $r(q)$ and $r(\dot{q})$ quantify the difference of joint angles and joint angle velocities between the simulated avatar and the ground truth mocap pose and $r(x)$ and $r(\dot{x})$ quantify the difference in Cartesian positions and velocities of each joint. Each term is expressed using a Gaussian kernel $r(s)=\exp({-k_{s}\sum_{j}\lVert{s}_{sim}-{s}_{\text{gt}}\rVert_{2}^{2}}),$ where ${s}_{sim}$ and ${s}_{\text{gt}}$ are the particular representation of the simulated character and the ground truth pose respectively and $k_{s}$ is the sensitivity of the kernel for each representation. Weights $\mathbf{w}$ and kernel sizes $\mathbf{k}$ can be found in the Appendix.

Using only the above terms when no lower-body user observations are available results in the simulated avatar taking short, high-frequency steps, which look unnatural. The additional reward term $r_{f}=\exp(k_{f}\sum_{i=L,R}\max(0,f_{y,i,{\text{prev}}}-f_{y,i}))$ reduces this, by penalizing an abrupt decrease in vertical contact force $f_{y}\in\mathbb{R}$, where $L$ and $R$ denote the left and right foot, and $k_{f}$ the sensitivity of the Gaussian kernel. This encourages the avatar to first naturally unload a leg before lifting it, but still allows to forcefully step down to make contact.

The policy directly outputs torque values instead of PD target angles, which tracked stably and reduced code complexity compared to e.g. Stable PD (Tan et al., 2011) or other more involved controller formulations. We run the policy and simulation at 36 frames per second (fps), which we found was the largest stable timestep. This also facilitated downsampling of the HMD sensor data during inference, which is provided at 72 fps. As a physics simulator we use Nvidia’s PhysX, which has been wrapped by the RL training framework IsaacGym (Makoviychuk et al., 2021). This allows us to simulate physics on the GPU and gives access to the simulation data through PyTorch tensors (Paszke et al., 2019). We combine IsaacGym with the open-source, GPU-ready PPO implementation rsl_rl (Rudin, 2021; Rudin et al., 2021). This allows us to train 4000 characters in parallel on an NVIDIA GTX 3080, which trains to 14 billion policy steps, equivalent to 13 years of human experience, in 48 hours. This is roughly 2 orders of magnitude faster than CPU-based approaches. For a detailed description of RL algorithm parameters see Appendix.

The tracking accuracy for a variety of motions is summarized in Table 1. We evaluate our trained model on synthetic input generated from the Lafan dataset (Harvey et al., 2020) as well as on real data recorded while wearing the headset. We evaluate poses reconstructed using the HMD and controllers (H+2C), and using only the 6 DoF headset pose (H). We compare our metrics to a state-of-the-art solution (Yi et al., 2022), which uses 6 IMUs, two attached to the legs and evaluated on the TotalCapture (TC) dataset (Trumble et al., 2017).

As metrics we use MPJRE (Mean Per Joint Rotation Error) in degrees and MPJPE (Mean Per Joint Position Error) in centimeters. We also compute the global root error in centimeters. To compare to (Yi et al., 2022), we calculate the ”SIP” error, which represents the mean orientation error of the upper arms and legs in the global space in degrees. Jitter represents the smoothness between poses, where smaller is more smooth. This is quantified by the jerk, the derivative of the acceleration, and calculated as done in (Yi et al., 2022). Finally, we compute the MHPE/MHRE (Mean Headset Position/Rotation Error), the error between the the simulated avatars head/hands (+offset) and the global position and orientation of the Quest devices.

When comparing reconstructed poses in Figure 3, we observe that the framework can distinguish different types of motion. To better understand how the model is achieving this, we plot the real sensor input driving these animations in Figure 4. This specific clip contains three different motion types: writing, walking and jogging. Starting with the writing section, we notice that the vertical head position dips, hinting that the user is crouching and must be bending the knees. Other than that there is little correlation between controllers and head. During the walking section, the controllers oscillate in the horizontal directions (red and blue) relative to the head. This might be a pattern the model uses to identify a walking gait. In contrast to walking, jogging motions have no oscillation in the x direction (red), but displays a very unique and high frequency oscillation of the vertical HMD (green).

We show that a single policy can track users of different scale. The user in Figure 3 is 180 cm tall, whereas the user in Figure 5 is 167 cm. The pose reconstruction is done by the same policy, without retraining. The user scale during inference is determined based on the initial height of the HMD, which causes the appropriately scaled avatar to be initialized (see Sec. 3.6 for details).

Apart from tracking users of different scale, we can also use avatars of different scale to track the same user. Figure 5 shows the same user being tracked by a larger avatar, which is a naive form of retargeting. Since the avatar is larger, but tries to track the position of e.g. the head, it naturally results in a more crouched position than the user. But generally the avatar imitates the users motion fairly well. More importantly, due to the physics simulator, the motion doesn’t violate any physical laws, such as foot skating or floor penetration that often appear when retargeting motions. With this naive baseline, future work could investigate how to further decouple users from avatar representation, allowing them to be embodied by whatever avatar they choose.

Since the characters are physically simulated (compared to kinematically tracked), their pose is influenced by collisions with the simulated environment (floor, objects). This adaptation of the tracked pose by physics can compensate for missing sensor data and convincingly integrate characters with their virtual world. Two scenarios are shown in Figure 6.

In the top scenario a ball and a rigid object are loaded into the physics simulator. As a user is tracked, we observe a two-way interaction: The avatar influences the external objects (ball kick), and the objects can influence the pose of the avatar (trip hazard). This poses new research questions, for instance, how much should this semi-autonomous avatar be controlled by physics, and when should physics be violated to closely track a user?

The bottom scenario demonstrates an avatar tracking a motion on rough terrain, while the user and sensor signal is recorded on flat ground. This example demonstrates how the reconstructed motion adapts to unseen virtual worlds to create believable animations, for instance the foot adapts to different slopes in the terrain. A similar application would be a user siting on a chair, in which case the avatar’s mesh should not penetrate the virtual chair. It might be difficult to measure the user to this accuracy with sensors. However, the simulated collisions and gravity can correct the tracked avatars pose to avoid penetration or hovering.

While this method can produce high quality tracking results for some motions, it can also fail to track others entirely. This is different than kinematic-leaning approaches (Yi et al., 2022), which, on difficult motions, might perform worse (e.g more jitter, less accuracy), but nonetheless still track the motion. But since our motions are simulated without additional non-physical forces, moving the root of the character to a desired position requires a precise sequence of joint torques. Furthermore, physics simulation does not allow teleportation, so as the character drifts further from the user, it can become increasingly difficult to catch up. For these reasons the simulated character can fall when attempting to imitate a dynamic out-of-distribution motion for which it hasn’t yet learned the torque controls (e.g. break-dance, jumping).

Another difficulty comes from uncorrelated upper-lower body motions, resulting in different motions being represented by the same upper body sensor data. In this case, the policy will synthesize a natural and physically-valid lower body pose, but this might not match the user’s pose.