DEMOS: Dynamic Environment Motion Synthesis in 3D Scenes via Local Spherical-BEV Perception

(a) We take human start information and scene point clond as inputs where the scene is further perceived through the Spherical-BEV projection.

(b) GoalNet was designed to first sample the future goal position and orientation of the human body given current human and scene information.

(c) Then, we take RouteNet to predict a scene-aware route consisting of trajectory and root orientation sequence in the short-term future.

(d) Based on the start/goal information and route, we can generate the appropriate pose sequence according to the scene geometry using PoseNet.

(e) Later, we iteratively predict future motions and utilize newly predicted motions to update latent motion through blending for dynamic environment adaptation.

We will finish the process or sample a new goal once the current goal is achieved. Our iterative motion synthesis method will finally generate long-term motions in dynamic environments where the surrounding point cloud may change over time.

Concretely, we estimate the future goal position distribution based on the scene structure via a CVAE GoalNet in a Bird’s Eye View (BEV), and utilize the elevation map to filter out inaccessible areas. Then, we sample the goal position $\hat{t}_{G}$ and adjust the height according to the position elevation with state hint. We also adjust the orientation $\hat{r}_{G}$ according to a possible route obtained from the elevation map. Given current human&scene information at time $T$ and sampled goal, we will predict the route in the future $k$ frames $\hat{\mathcal{R}}_{T}=\hat{R}_{1+T:k+T}=(\hat{t},\hat{r})_{1+T:k+T}$ according to the environment point cloud $\mathcal{E}_{T}$. We will then generate scene-aware pose sequence $\hat{\mathcal{P}}_{T}=\hat{P}_{1+T:k+T}=(\hat{\theta}_{p},\hat{\theta}_{h})_{1+T:k+T}$ which also depends on the start-goal information and predicted route. The motion sequence consisting of routes and poses from $T+1$ to $T+k$ is marked as $\hat{\mathcal{S}}_{T}=(\hat{\mathcal{R}},\hat{\mathcal{P}})_{T}$, termed hypothesized motion in this paper. For long-term motion synthesis, we iteratively predict $\hat{\mathcal{S}}_{T}$ and blend it with previous latent motion $\ddot{\mathcal{S}}_{T-1}$ to obtain the updated latent motion $\ddot{\mathcal{S}}_{T}$, and finally synthesize long-term motion $\tilde{\mathcal{S}}_{0:F}=[\ddot{S}_{0}[0],\cdots,\ddot{S}_{F}[0]]$ where $F$ is the total number of pre-defined frames and the execution frame $\ddot{S}_{i}[0]$ is the route and pose in the first frame of $\ddot{S}_{i}$. As for the initialization, we set $\ddot{\mathcal{S}}_{0}=\hat{\mathcal{S}}_{0}$. Here, we set $k$ to $60$ frames, i.e. , $2$ seconds for a video of 30 FPS.

Based on the human motions from PROX [24] and GTA-IM [1] datasets, we attempt to categorize all the poses into four states. For each pose from these two datasets, we first obtain the SMPL-X body mesh and check the contact [17] and movement [2] information of foot, gluteus, and back vertices. Thus, we can define the state of each pose according to the contact information.

Specifically, as shown in Figure 3, we define bodies with only two feet as anchors to be idle, bodies with only one foot as the anchor to be in locomotion, bodies with gluteus as the anchor (and back is not an anchor) to be sitting, bodies with back as the anchor to be lying. Poses will be invalid and not be used for training/testing if none of the foot, gluteus, and back are anchored body parts. To leverage the prior knowledge of periodic motion like locomotion [43], we will also automatically calculate the phase based on the foot anchor information, while the phases in other states are set to zero. Thanks to these annotations, we can utilize them to guide the scene-aware PoseNet to predict states&phases that correspond to local scenes, and later rational poses. We can also know the anchored parts given the predicted states, which can also be used to avoid skating at contacted parts between consequent poses.

To make the synthesized motion more consistent with the surroundings, we propose two projection-based local scene perception (PBLSP) methods to extract the spherical depth and BEV elevation information from environment point clouds.

For spherical angular perception, we first extract the local scene point clouds $\mathcal{E}_{T}^{l}$ consisting of $K_{1}$ nearest points in environment point cloud $\mathcal{E}_{T}$ from human root position. Then, these $K$ points are projected onto a unit sphere with pre-defined $L_{1}$ longitude bins and $L_{2}$ latitude bins to obtain the angular depth representation of the local scene. In this paper, we simply take $L_{1}=36$ longitude bins and $L_{2}=18$ latitude bins, and $K_{1}=2500$ nearest points from the local scene are selected for the angular projection. This real-time point cloud projection is implemented in parallel by extending the Pytorch implementation of z-buffer [44, 45]. Specifically, we calculate the longitude and latitude bin for each point and then scatter the distance value into the sparse matrix $\mathbf{M}_{1}\in\mathbb{R}^{L_{1}\times L_{2}\times K_{1}}$ with default value $\infty$. Then, the angular depth at different directions can be easily obtained through finding the minimum among $K_{1}$.

After that, we choose to design a lightweight spherical convolution rather than sophisticated ones like KTN [46] (the comparison is available in Section 4.5) to extract the angular depth distribution pattern from the local scene point cloud. Each layer of our spherical convolution consists of a longitude convolution and a latitude convolution. Unlike the common 2D convolution, the longitude convolution with a ring structure should be cyclic. So, we first warp the feature map horizontally and apply 1D convolution on the longitude. As for the latitude convolution without ring structure, we can directly utilize 1D convolution. After a few layers of spherical convolution and angular pooling, we utilize several fully connected layers to extract the angular depth distribution feature $f_{a}$.

Thus, we attempt to take a Bird’s Eye View (BEV) to extract the elevation map surrounding the current human body. In our setting, we mainly care about the elevation map of the local scene within a body-centered square with side length $S$. This square is also discretized into $M\times M$ bins, and $K_{2}$ surrounding points from the scene are selected for BEV depth rendering. In this paper, the side length is commonly set to $8.2$ meters which is further split into $M=41$ bins, and $K_{2}=20,000$ nearest points are taken for BEV perception. Different from point projection into unit sphere, there will be some points outside all the bins, and these points can not be ignored in batch operation. So, we pad the square map with one bin on each side, i.e. , $L_{3}=M+2$ bins in total (see the bins in Figure 4 (b) without color), and all residual points are rendered to the nearest padding bins. By now, all rendered points can be scattered to another sparse matrix $\mathbf{M}_{2}\in\mathbb{R}^{L_{3}\times L_{3}\times K_{2}}$, and then we can obtain the BEV depth map $\mathcal{D}\in\mathbb{R}^{M\times M}$ by calculating the minimum depth value among $K_{2}$ and discarding the padding bins. After that, the BEV elevation map can be extracted from $\mathcal{B}=C_{0}-\mathcal{D}$ where $C_{0}$ is a constant hyper-parameter and set to $0.8$ in this paper. Finally, we build a small network consisting of several 2D convolution and pooling layers to extract the BEV elevation features $f_{b}$.

By now, we can concatenate the scene points feature $f_{p}$ extracted from MLPs with the spherical angular pattern and BEV elevation feature to obtain the scene feature $f_{s}=f_{p}\oplus f_{a}\oplus f_{b}$. Such Spherical-BEV perception can supply the local scene cues essential for instant scene-aware motion synthesis given any input scenes (demonstrated in Section 4.5).

The fundamental architectures of the networks bear resemblance to those employed in previous works [2, 3] for motion synthesis. In this work, our networks focus more on how to model the correlation between human embedding and scene hints extracted from the PBLSP, and later attempt to sample feasible goals in the scene and infer rational future motion sequence under the guidance of designed anchor-based states with human-scene contact information.

For training of GoalNet, we pursue reconstructing the goal position and orientation and pushing the goal latent space close to standard normal distribution, and the supervision is formulated as:

where $KL$ represents the KL divergence between these two normal distributions, and thus $\mathcal{L}_{KL}^{g}$ can be simplified as:

In the motion synthesis stage, we also utilize the elevation map to filter out impossible areas in BEV, which works as a filter $\mathcal{F}$. Specifically, we build a graph based on the elevation map with each node representing a horizontal position and recording the elevation. Given the current state and position node, we can easily judge the state transition according to the relative elevation between neighboring nodes. For instance, the transition between idle to sit is usually accompanied by an elevation transition of 0.3-0.6 meters. We can then extract the possible area $\mathcal{F}$ according to the graph nodes with a possible state.

We then sample the final root position from the distribution given by a multiplication of the normal distribution $\mathcal{N}(\hat{t}_{g},(\sigma^{t})^{2}\mathbf{I})$ and the filter $\mathcal{F}$, and $\sigma^{t}$ is a hyper-parameter that is commonly set to $2$ in our setting. Finally, we adjust the z-axis position according to the elevation map to obtain $\hat{t}_{G}$ and adjust z-axis root orientation according to a possible path from $t_{0}$ to $\hat{t}_{G}$ to obtain $\hat{r}_{G}$.

In the inference stage, RouteNet can directly predict the future route within $k$ frames based on current human-scene information and sampled goal.

Along with latent code $z^{p}$ and human-scene condition $f_{c}^{p}$, we choose to take an additional LSTM to extract frame-wise route features as position query, and then predict the state probability $\hat{s}_{1:k}$ at each frame accordingly. As for locomotion, we choose the relative transformations between consequent routes rather than routes themselves as input to predict the phase difference $\hat{\omega}_{1:k}$, and infer the phase sequence $\hat{p}_{1:k}$ via cumulative summation.

Later, we use a mixture of expert networks to infer the pose sequence following previous works [19, 20]. The mixed network weight $\alpha_{i}$ at each frame can be controlled by the predicted probability $\hat{s}_{1:k}$ of state $c$ at frame $i$ via $\alpha_{i}=\sum_{c=1}^{C}\hat{s}^{c}_{i}\alpha^{c}.$ Condition feature $f_{c}^{p}$, latent code $z^{p}$, initial pose $P_{0}$, route $R_{1:k}$, and inferred phase $\hat{p}_{1:k}$ are fed into the mixed network to predict the entire pose sequence $\hat{P}_{1:k}$.

For the training of PoseNet, we supervise the network by anticipating the intermediate state&phase and reconstructing the input pose sequence, where loss can be formulated as follows:

where $\mathcal{L}_{KL}^{p}$ is KL Divergence for pose latent code distribution. $\mathcal{L}_{joint}$, $\mathcal{L}_{vert}$ and $\mathcal{L}_{\theta}$ are the L1 loss for body joint, mesh and parameter reconstruction. $\mathcal{L}_{state}$ is the cross entropy loss in state classification and $\mathcal{L}_{phase}$ is the L1 loss in phase step prediction. The explicit formulation of these losses can be defined as:

Here, $\mu^{p}$ and $(\sigma^{p})^{2}$ are the predicted mean and variance value of the latent code $z^{p}$ in PoseNet. $\mathcal{J}$ and $\mathcal{M}$ indicate the SMPL-X [9] body joints and mesh vertices formed by the translation, orientation, and body shape and pose. $\hat{\mathcal{P}}_{i}$/$\mathcal{P}_{i}$ contains the predicted/P-GT body pose and hand pose parameter $(\hat{\theta}_{p},\hat{\theta}_{h})_{i}$/$(\theta_{p},\theta_{h})_{i}$. $\hat{s}_{i}^{c}$/$s_{i}^{c}$ is the predicted/P-GT probability of state $c$, while $\hat{\omega}_{i}$/$\omega_{i}$ is the predicted/P-GT phase step.

Given current human-scene information and goal, we can predict the motion in future $k$ frames, which is reasonable for a static scene. However, in dynamic scene point clouds where other characters or vehicles move around, the future motion should be updated accordingly. So, we choose to iteratively predict future motion $\hat{\mathcal{S}}_{T}$ and use it to update previous latent motion $\ddot{\mathcal{S}}_{T-1}$ (derived from $\hat{\mathcal{S}}_{0}$, $\cdots$, $\hat{\mathcal{S}}_{T-1}$) to both benefit from the stability of prior-based motion synthesis and responsiveness of iterative methods when handling dynamic 3D environments.

Here, we introduce a time-variant motion blending to iteratively update the latent motion using newly hypothesized motions, and a continuous space for motion is necessary for this motion blending. Specifically, we choose the required continuous space formed by the Cartesian product of trajectory in Euclidean space, root orientation in 6D continuous space [49] and pose feature in SMPL-X prior space [9]. For initialization, we directly set the latent motion as $\ddot{\mathcal{S}}_{0}=\hat{\mathcal{S}}_{0}$. Here, we first consider the latent route update. Given previous latent route $\ddot{\mathcal{R}}_{T-1}$ and current predicted route $\hat{\mathcal{R}}_{T}$, we can update the latent route as shown in Figure 2 (e) through blending for motion continuity:

where $\tau\in[0,\cdots,k-1]$, $\alpha=(k-\tau-1)/k$ is the time-variant blending coefficient, and $\nu$ is a hyper-parameter that equals to $4$ to make sure the responsiveness in our implementation. For $\ddot{\mathcal{R}}_{T-1}[k]$ which works as a placeholder in the formulation will be set to a zero vector. Inspired by previous works [20, 3] which utilize the $A^{*}$ algorithm to help path planning, we take the trajectory $\mathcal{T}^{*}$ given by the $A^{*}$ algorithm to adjust our predicted route. Unlike directly blending the route, we find that the low-frequency component of velocity given by $\mathcal{T}^{*}$ can help avoid obstacles. Therefore, we first calculate the velocity sequences $v^{*}$ and $\ddot{v}$ from $\mathcal{T}^{*}$ and $\ddot{\mathcal{T}}_{T}$ (trajectory of $\ddot{\mathcal{R}}_{T}$). Then, we utilize the Fast Fourier Transform to obtain the velocity spectrum:

After that, we blend the velocity sequence into the spectrum space through $\bar{f}=lf^{*}+(1-l)\ddot{f}$ where $l$ is a low-pass filter. The desired trajectory $\bar{\mathcal{T}}$ is derived from the blended velocity $\bar{v}=Real(IFFT(\bar{f}))$ via

Here, we can calculate the z-axis angular difference $M_{rot}$ between current trajectory and desired trajectory at the $m$-th frame in the future where $m$ is set to $15$. Then, the fused route can be adjusted in the matrix representation $\ddot{\mathcal{R}}_{T}=M_{rot}\ddot{\mathcal{R}}_{T}$. As for the poses, the latent pose sequence $\ddot{\mathcal{P}}_{T}$ is updated the same as Eq. 12, while the latent motion is updated as $\ddot{\mathcal{S}}_{T}=(\ddot{\mathcal{R}}_{T},\ddot{\mathcal{P}}_{T})$.

The final synthesized motion from $0$ to $F$ is extracted as $\tilde{\mathcal{S}}_{0:F}=[\ddot{\mathcal{S}}_{0}[0],\cdots,\ddot{\mathcal{S}}_{F}[0]]$. We will terminate the generation or sample the next goal when the distance between goal and current position is smaller than a tolerance $\epsilon$.

Thanks to the anchor-based state and phase prediction, we can introduce a no-skating operation for contacted parts during iterative latent motion updates. Take locomotion for instance, we can know which part of the body should be in contact with the scene after checking current state $\hat{s}_{T}$ and phase $\hat{p}_{T}$. We will calculate the mean position of anchored body part $\mathbf{x}_{T-1}$ from the execution frame of previous latent motion $\ddot{\mathcal{S}}_{T-1}[0]$, and derive the new position $\mathbf{x}_{T}$ from current latent motion $\ddot{\mathcal{S}}_{T}[0]$. Then, the current latent motion will be corrected for no-skating through a translation of $\kappa(\mathbf{x}_{T-1}-\mathbf{x}_{T})$, where $\kappa$ is set to $0.6$ in our setting.

In order to align the data format of two datasets and unify the evaluation, we extend MMHuman3D [50] to fit the SMPL-X parameters using skeleton joints from GTA-IM where SMPL-X parameters of consecutive frames are constrained to ensure continuity, and we take these fitted parameters as pseudo-ground truth (P-GT). We also build a pipeline based on the Open3D [51] to obtain the scene mesh, point cloud, as well as Signed Distance Function (SDF) of each scene from the RGB-D images. We refer to the training/testing split given by Wang et al.  [52] where $70$ sequences in $6$ scenes are used for training and $30$ sequences in $3$ scenes are used for testing. Similar to the data format of PROX, each sample contains current information, goal position&orientation, and motion in the future 2 seconds, and all starting frames are sampled with a stride of 5 frames.

For detailed parameter settings, the spherical angular depth is fed into a small network consisting of several layers of light-weight spherical convolution and fully connected layers to extract the spherical angular pattern $f_{a}\in\mathbb{R}^{256}$. On the other side, the BEV elevation feature $f_{b}\in\mathbb{R}^{128}$ is extracted from the BEV elevation map through a small 2D convolution network.

In the GoalNet, the dimension of the goal condition feature $f_{c}^{g}$ is set to $128$, and the mean and variance value of the goal latent code $z^{g}$ is set to $\mu^{g},(\sigma^{g})^{2}\in\mathbb{R}^{32}$. The start pose feature $f_{sp}\in\mathbb{R}^{56}$ from the SMPL-X prior and the goal-based moving feature $f_{g}\in\mathbb{R}^{256}$ are used in the RouteNet. As for the PoseNet, there are additional route feature $f_{r}\in\mathbb{R}^{64}$ to supply route hints, while the dimension of pose sequence latent code $z^{p}$ is set to $64$.

In the training procedure, $\lambda_{1,\cdots,7}$ are set to $\{0.3$, $0.7$, $10$, $10$, $1$, $1$, $1\}$, respectively, for both PROX and GTA-IM datasets. GoalNet, RouteNet, and PoseNet are trained independently by Adam [53] optimizers with learning rates {$10^{-3}$, $10^{-3}$, $3\times 10^{-6}$}. All these networks are trained for $20$ epochs.

Based on these settings, all our networks can be trained on a machine with 16G RAM and 11G VRAM, and the total networks’ inference time is $4.9$ms on a single GTX 3090 after training.

Users are required to enter their username to fetch the first batch of samples given by different methods/P-GT or continue the rating procedure. These samples with the same initial information are randomly permuted and then presented to users. They are asked to give a score between $1$ to $5$ for each sample after watching all videos within the batch. After submitting their scores, they can get a new batch of samples if they have not rated all motion samples. They can also terminate the rating at any time because we will record the results once they submit the scores and usernames can be used to continue the procedure. Finally, we will collect all the results and calculate the mean scores and variance values of different methods for both PROX [24] and GTA-IM [1].

In addition, we utilize Fréchet Distance (FD) as a perception metric like previous works [20, 3] to evaluate the perception similarity between the synthesized motions and P-GT motions in motion feature space.

We also present the visual results of previous works and the proposed method in Figure 8. As can be seen in the first row, the turning around during locomotion given by our method is more continuous than the competitors. Meanwhile, the proposed method achieves a higher level of naturalness in the transition between locomotion and sitting, as indicated in the second row. In addition, our predicted sitting motions in the following three rows are more similar to the pseudo-ground truth and more compatible with surrounding scenes thanks to the specifically designed local scene perception. The generated sitting poses can change continuously according to the distance to the chair, and this is attributed to the geometry perception from the PBLSP module which guides the PoseNet to control scene-aware state coefficients and finish the sitting procedure.

The visual comparison of different methods on GTA-IM is presented in Figure 9. Given the scene point cloud and initial human body information, all methods generate 4-second future motions with the highest probability. From the first row of Figure 9, we can see that the motion predicted by our method is more likely to circumvent the column compared to other methods due to the perception of obstacles. According to the second row, the phase information encoded by the discrete cosine transform utilized by HMP [22] can only provide limited phase hints for short-term motion, while the phase step information predicted by our method can better guide the locomotion and make sure the continuity for long-term motion synthesis. From the last few examples, we can see the proposed method can better reconstruct future motion with scene-aware actions given a goal and start human bodies in 3D scenes. That indicates the recognition of the local geometry in our method can help predict the correct anchor-based state to synthesize plausible motion.

We also report the Fréchet Distance in motion feature space in Table IV which indicates the perception similarity between synthesized motions and original ones. We can see that the synthesized motions given by our method are more similar to motions in the original datasets than other competitors.

In this section, we conducted more experiments to check the effectiveness of the proposed DEMOS pipeline and prove our claims. Without loss of generality, our ablation studies are mainly conducted on the GTA-IM dataset.

We report all the results in Table V. The results indicate that guiding the network to predict the state and phase of procedure can help synthesize motion with correct and unambiguous action. Moreover, recognizing the surrounding scene point cloud via Spherical-BEV perception and leveraging its correlation with human start information can enhance the precision of the predicted route and motion. Specifically, the SAP focuses more on route prediction and we think that is because SAP can better recognize the objects in different directions. On the other hand, the BEP is more effective in pose sequence prediction because the body-centered elevation recognition can further provide height hints of surrounding objects for state prediction. These functional components can help the networks to predict scene-aware motions instantly via better recognizing the surrounding environment and exploiting the correlation between the scene and the human body.