Diverse Dance Synthesis via Keyframes with Transformer Controllers

Recently, deep learning has gained remarkable success in the field of both CV and CG. To synthesis realistic and natural human motion, there has been a surge in modeling human motion with neural networks. For example, Holden et al. [HSK16] propose a deep convolutional network to perform human motion synthesis and resolve the ambiguity via incorporating foot contact information. However, the proposed framework concentrates on simple character movements regarding walking, running, and punching. Inspired by the image inpainting techniques, Hernandez et al. [RGM19] reformulate motion prediction as an inpainting task to complete the masked joints in spatiotemporal volumes. It is well-known that recurrent neural networks (RNNs) have inherent advantages in modeling sequential data. Hence recent work employs RNNs to model human motion. Martinez et al. [MBR17] make several modifications to the standard RNN models and develop a sequence-to-sequence architecture with residual connections for short-term human motion prediction. Lee et al. [LLL18] present a multi-layer RNN conditioned to handle spatiotemporal constraints and structural variabilities for interactive character animation. Wang et al. [WHSZ21] formulate a new spatiotemporal RNN framework to investigate the motion manifold. The proposed network avoids the generation of average posture and eliminates the need for a separate disambiguous network. Also, Wang et al. [WCX21] construct an RNN-based generator for human motion synthesis and utilize a refiner network with adversarial training loss to refine motion sequences. In our work, the RNN is utilized for sequence modeling and long-term motion synthesis. The constrained conditions in our method is modeled using two additional Transformers, which enables the network to synthesize diverse dance motion sequence.

Dance motion synthesis can be viewed as a typical conditional motion generation task. Due to the high correlation between dance and music, extensive works have been dedicated to the music-oriented dance generation. In the early research, dance-to-music is regarded as the problem of template matching, which attempts to generate dance movements according to musical similarity [SNI06, LLP13]. By analyzing the rhythmic patterns of motions, Kim et al. [KPS03] facilitate the rhythmic motion generation synchronized with an input sound signal. However, the above methods are limited by their capacity of the provided dataset. Recently, neural networks dominate dance motion synthesis. Tang et al. [TJM18] design an LSTM-autoencoder model to extract the mappings between music and motion, which can largely enhance choreography in accordance with the music. Lee et al. [LYL^∗19] and Ye et al. [YWJ^∗20] attempt to synthesize dance from music through a two-stage procedure, which firstly learn basic dance units and then organizes the basic units into dance sequence. Nevertheless, two-stage generation methods lack enough flexibility and scalability. To alleviate error accumulation of autoregressive model in long-term motion generation, Huang et al. [HHW^∗21] formulate the music-conditioned dance generation as a sequence-to-sequence learning problem and utilize the curriculum learning strategy to enhance the training process. Considering that the motion manifolds of classical convolutional and recursive neural models are non-Euclidean geometry, Ferreira et al. [FCG^∗21] design a novel method based on graph convolutional networks to synthesize human motion from music. The goal of our research is different from the above work. We attempt to synthesize dance motion sequences conditioned on temporal sparse keyframes with user-specified root trajectory and velocity factor. It is a rarely explored but rather valuable topic for its significance in character animation and entertainment.

In computer animation, there have been intensive investigations on motion transition generation. We limit the task as synthesizing intermediate movements between user-specified keyframes. It is quite challenging as significant motion gaps must be filled under sparse temporal constraints. Pioneering approaches are mainly based on retrieval paradigms, which search the matched motion clips from a database and blend them, such as motion graphs [KGP02, ZS09]. After that, methods based on probabilistic models have been widely adopted for human motion. Chai et al. [CH07] and Min et al. [MCC09] formulate motion synthesis as maximum a posterior (MAP) problem. Wang et al. [WFH08] apply Gaussian process dynamical models (GPDMs) for learning models of human pose and motion. However, the above methods are designed for designated actions, making the algorithms look like pre-arranged scripts and rules that will fail in confronting complex human movements. Because of the impressive scalability and expressiveness of deep neural networks, recent studies tend to use deep learning for motion transition prediction. Gaisbauer et al. [GLSR19] present a fully-connected feed-forward neural network for generating feasible postures from given input postures. Zhang et al. [ZvdP18] formulate an autoregressive recurrent neural network (ARNN) that is conditioned on the target keyframes for motion-aware animation with fixed interpolated frames. Harvey et al. [HP18] propose the recurrent transition networks (RAT) based on LSTM to synthesize missing data between keyframes. However, the method can still only generate motion transition sequences with definite lengths and is limited to periodic routine actions. Later, Harvey et al. [HYNP20] improve the RAT method [HP18] by adding the time-to-arrival embedding to the network, which allows the method generates variable transition lengths. They verify the method on a periodic dance dataset and obtain good animation results. In summary, existing methods mainly focus on the motion transition of periodic actions with relatively short-term synthesis, while our work concentrates on the long-term and impromptu contemporary dance generation. For diverse motion synthesis, besides the conventional root trajectory constraint, we introduce the velocity factor for the first time in motion modeling, which preserves the naturalness and diversity of the generated dance movements.

The dance dataset we use comes from[dat, AZS^∗17]. We remove the finger joints, and the resulted human skeleton contains $23$ joints. We preprocess the dataset by rotating the postures at different timesteps toward the positive direction of $z$-axis and extract the related information based on the rotated postures. The definition of the symbols used in this paper is listed in Table 1.

We use $\textbf{X}_{t}=\{\textbf{p}_{t},\textbf{o}^{r}_{t},\textbf{c}_{t},\textbf{v}_{t}\}$ to represent the ground truth information at time $t$ and $\tilde{\textbf{X}}_{t}=\{\tilde{\textbf{p}}_{t},\tilde{\textbf{o}}^{r}_{t},\tilde{\textbf{c}}_{t},\tilde{\textbf{v}}_{t}\}$ to denote the prediction results of the network at time $t$. The given inputs and conditions can be expressed as $\{\textbf{p}_{k_{1}},\textbf{o}^{r}_{k_{1}},\textbf{p}_{k_{2}},\textbf{o}^{r}_{k_{2}},\hat{\textbf{p}}^{r},\hat{\textbf{f}}\}$. The output of the model contains the position sequence $\{\tilde{\textbf{p}}_{k_{1}+1},\cdots,\tilde{\textbf{p}}_{k_{2}}\}$ of all joints, the rotation sequence $\{\tilde{\textbf{o}}^{r}_{k_{1}+1},\cdots,\tilde{\textbf{o}}^{r}_{k_{2}}\}$ of the root joint, the contact label sequence $\{\tilde{\textbf{c}}_{k_{1}+1},\cdots,\tilde{\textbf{c}}_{k_{2}}\}$ of the toe joints and heel joints, as well as the velocity sequence $\{\tilde{\textbf{v}}_{k_{1}+1},\cdots,\tilde{\textbf{v}}_{k_{2}}\}$ of all joints.

The proposed network aims to receive historical information and control signals, then produce smooth and diverse dance sequences. As shown in Figure  2, we demonstrate the generation process in one timestep. Given two keyframes $k_{1}$ and $k_{2}$, the problem is how to generate a realistic motion sequence between the two keyframes with a natural transition. In essence, this is a time-series-prediction problem. Inspired by [HYNP20], we utilize the LSTM unit to predict the motion sequence.

Overview of the network.

The whole framework consists of three encoders, three controllers, and two decoders. The three encoders are comprised of a state encoder, a velocity encoder, and a position encoder. Specifically, the state encoder receives the posture information, i.e., the root-relative positions $\textbf{p}^{b}_{t}$, the root’s global position $\textbf{p}^{r}_{t}$, the root’s rotation angle $\textbf{o}^{r}_{t}$, and the foot contact information $\textbf{c}_{t}$. The position encoder takes the next frame’s root position as input to make the network aware of the temporal embedding. The velocity encoder projects the dynamics of the movement to our network. The three controllers are composed of a root trajectory controller, a velocity factor controller, and a target controller. The root trajectory controller takes the positions of the root’s trajectory as input to ensure the generated motion moves towards the specified trajectory. The velocity factor is proposed to achieve fine-grained motion control by specifying the moving speeds of different body parts. The target controller receives the posture information of the keyframe. The goal is to make the network perceive the distance between the predicted frame and the target keyframe. The decoders contain a root decoder and a state decoder. The root decoder is in charge of decoding the root position, rotation, and velocity information, while the state decoder predicts the relative positions and velocities of other joints except for the root, as well as the foot contact states at time $t+1$. In summary, the three encoders, two decoders, and the target controller are constructed based on the fully connected layer (FC-based) with different depths, while the root trajectory controller and the velocity factor controller are formulated based on the Transformer structure (Transformer-based).

The root trajectory controller.

To make the generated motion consistent with the given trajectory, it is crucial to let the network capture the temporal context information from the past to the future. Since Transformer [VSP^∗17] can well model long-range dependency by leveraging the query-key correlation to different tokens, it has shown outstanding results in natural language processing [MZZ^∗19], neural machine translation [BDK20], and various vision tasks [ZZT^∗20, CMS^∗20]. Therefore, we construct the root trajectory controller based on the Transformer mechanism.

We illustrate the detailed structure of the root trajectory controller in Figure 3, where $\hat{\textbf{p}}^{r}_{t+1,u}$ refers to the target root trajectory segment with frame $t+1$ as the center and the window size of $u$. $\check{\textbf{p}}^{r}_{t+1,u}$ is the modification of $\hat{\textbf{p}}^{r}_{t+1,u}$, that is, the data before the frame $t+1$ is replaced by the predicted root trajectory. Considering that the root trajectory predicted by the network cannot be completely consistent with as the target trajectory, and the current predicted position is not only required to smoothly connect with the past predicted results, but also move along the future target trajectory, it is not enough to only encode the given target trajectory. Therefore, we introduce the $\check{\textbf{p}}^{r}_{t+1,u}$ and define it as the mixture root trajectory segment. The encoders and decoders in the Transformer are the standard combinations of the multi-head self-attention (MHSA) and the feed forward networks (FFN). The MHSA comprises multiple self-attention blocks and explicitly models the interactions between all entities of a sequence. The FFN is used to perform information transformation. The residual add and layer norm are implemented after both the MHSA and the FFN.

For the constraint coding process, the controller first accepts $\hat{\textbf{p}}^{r}_{t+1,u}$ as input, projects it into the embedding space, and then extracts the trajectory representation with two encoders. We also project $\check{\textbf{p}}^{r}_{t+1,u}$ into the feature space and then decode it with two decoders. Skip connections are used between the encoders and the decoders to increase the flexibility of information flow. As pointed out in [VSP^∗17], the Transformer has a permutation invariance. We follow its procedure, incorporating the positional encoding into $\hat{\textbf{p}}^{r}_{t+1,u}$ and $\check{\textbf{p}}^{r}_{t+1,u}$ to make the controller aware of the trajectory segments’ positions. The technical details can be found in the literature above. Finally, the decoded result is transformed by a linear layer to acquire the representation of the root trajectory constraint.

The velocity factor controller.

To achieve fine-grained control, the velocity factor controller is introduced. As the speeds of different body parts can vary significantly when dancing, we divide the human body into five parts: the trunk, the left and right arms, the left and right legs. We get five velocity factors by weighted average over the moving speeds of the joints related to the body parts. Figure 4 (a) displays the structure diagram of body division, where different colors represent different body parts. For different joints, the weights are set according to their distances from the end effector. The numbers in the circle of Figure 4 (a) are the weights used in this paper. In general, the end effectors are more influential in describing the dynamics of a movement, so joints closer to them have larger weights. The velocity factor of body part $i$, $i\in\{1,2,3,4,5\}$ at frame $t$ can be calculated as:

where $J_{i}$ is the joint set of part $i$, $w^{ji}$ is the weight of joint $j$ in part $i$, and $\textbf{v}_{t}^{ji}$ is the velocity of joint $j$ in part $i$ at time $t$. The obtained velocity factor sequences of different body parts for a given motion sequence are illustrated in figure 4 (b).

Considering human motion at consecutive moments should be continuous without sudden changes, it is important for the velocity factor controller to also perceive the long-range context information, so as to learn the smooth temporal context representation. Therefore, we construct the velocity factor controller with the same structure as the trajectory controller. The differences are that the inputs $\hat{\textbf{p}}^{r}_{t+1,u}$ and $\check{\textbf{p}}^{r}_{t+1,u}$ are replaced by the $\hat{\textbf{f}}_{t+1,u}$ and $\check{\textbf{f}}_{t+1,u}$. Here, the $\hat{\textbf{f}}_{t+1,u}$ is a sequence segment of the given velocity factor with the time $t+1$ as the center and the window size $u$, and the $\check{\textbf{f}}_{t+1,u}$ replaces the data of $\hat{\textbf{f}}_{t+1,u}$ before time $t+1$ with the network predicted results. Similar to $\check{\textbf{p}}^{r}_{t+1,u}$, $\check{\textbf{f}}_{t+1,u}$ is defined as the mixture velocity factor segment.

The LSTM backbone.

If we directly send all the coding formation of the encoders and the controllers to the LSTM network, the information received by the network will be too much and too complex. However, one LSTM unit may not be able to process so much data at one time. Therefore, we employ two LSTM units hierarchically to receive and process the encoded information. The first LSTM receives the outputs of the state and velocity encoders, as well as the root trajectory and velocity factor controllers. After the information is processed by the first LSTM, we concatenate its output with the keyframe embedding of the target controller and send the concatenated results to the second LSTM for further processing. The sequential processing inputs and control conditions make the network achieve better results. We will validate it in the ablation study. In experiments, we employ a scheduled sampling mechanism to select the input of LSTM in each timestep. Specifically, we first define a sampling probability rate $\beta$. When predicting actions in different timesteps, we select the ground truth as the input with the probability of $\beta$. At the beginning, $\beta$ is set as 1. With the progress of the training, $\beta$ decays in an exponential decay manner. The learning strategy can make the convergence of the network more stable and make the motion synthesis and prediction more smooth.

When the pose information of the target keyframe is directly sent to the network, the network cannot know the temporal distance between the frame to be predicted and the target keyframe. Referring to the insight in Transformer, we encode the position information of the frame to be predicted and add it with the input of the second LSTM unit. We use one fully connected layer to embed the position information. Specifically, as pointed out in [ZJJ^∗20], the 3D joint coordinate is a kind of natural position coding. We define the root’s position as the absolute position (AP) representation. The root joint of frame $t+1$ extracted from the root trajectory constraint is used as input of the position encoder. Since the trajectory sequence of the root joint may have the same position at different times, we design an additional relative position (RP) representation $a_{t}$, which is calculated as follows:

At last, the state decoder and the root decoder are used to predict the posture information of the next movement. However, when all the information is decoded by the state decoder, the predicted position sequence of the root joint is discontinuous with relatively large fluctuation. Inspired by the 3D human body pose estimation algorithms [ZFS^∗20, LL20], we separate the information of the root joint and use the root decoder to predict its state information. This practice can make the generated trajectory sequence smoother.

According to the network structure and goal, we define several loss functions, including the reconstruction loss $\mathcal{L}_{rec}$, posture consistency loss $\mathcal{L}_{con}$, root trajectory smooth loss $\mathcal{L}_{root}$, keyframe consistency loss $\mathcal{L}_{key}$, and velocity factor consistency loss $\mathcal{L}_{vfac}$. The complete loss function is defined as follows:

where $w_{rec}$, $w_{con}$, $w_{root}$, $w_{key}$, $w_{vfac}$ are the corresponding loss weights. We set the loss weights as 0.3, 0.2, 0.15, 0.2, and 0.15 respectively in training phase. These parameters are obtained through a number of experiments. We give the details of each loss below.

Reconstruction loss.

The mean square error (MSE) loss is taken to construct the reconstruction loss, which can force the network to generate motion sequences satisfying the designed control constraints. The reconstruction loss $\mathcal{L}_{rec}$ can be expressed as:

where $N$ is the sequence length. The terms of $\mathcal{L}_{rec}$ include the reconstruction of joint positions, root joint rotation angles, foot contact labels, and joint velocities at each moment.

Posture consistency loss.

When the network is trained only with individual joints, the correlations between the connected joints are neglected to a certain extent. Hence, we introduce the bone length consistency loss $\mathcal{L}_{bone}$ to force the network to generate posture consistent with the ground truth bone length. In the meantime, the foot contact labels and the joint velocities can be inferred from the predicted posture information. Thus, we also introduce the foot contact consistency loss $\mathcal{L}_{contact}$ and the joint velocity consistency loss $\mathcal{L}_{velocity}$ to avoid the contradiction of prediction information. Therefore, the consistency loss $\mathcal{L}_{con}$ consists of $\mathcal{L}_{bone}$, $\mathcal{L}_{contact}$ and $\mathcal{L}_{velocity}$, and they are expressed as follows:

where $\mathcal{B}$ in $\mathcal{L}_{bone}$ is the index set consisted of all natural connected joint pairs in the human skeleton. $l_{ij}$ is the original length of the bone segment formed by joint $i$ and $j$, the length of which can also be obtained by calculating the distance between $\tilde{\textbf{p}}_{t}^{i}$ and $\tilde{\textbf{p}}_{t}^{j}$. We penalize the length inconsistency between the ground truth and the inferred to force the correctness of the predicted posture information. In $\mathcal{L}_{contact}$, $\mathcal{F}$ is the index set of the foot contact joints, $\tilde{c}_{t}^{i}$ is the predicted contact label for foot joint $i$ at frame $t$, $\tilde{c}_{t}^{i}=1$ if there is foot contact and $0$ otherwise. We impose a penalty on the product of the L2-norm of $\tilde{\textbf{v}}_{t}^{i}$ and the corresponding contact label $\tilde{c}_{t}^{i}$ to force the consistency of the predicted foot contact label and the predicted velocity [WHSZ21]. In $\mathcal{L}_{velocity}$, $\mathcal{J}$ is the index set of all joints. The velocities of different joints at frame $t$ can be inferred by subtracting skeleton positions in the previous frame from the current frame $t$. The information consistency can be effectively guaranteed by punishing the differences between the inferred velocities and the predicted velocities of our network.

Root trajectory smooth loss.

We refer to the long horizon loss function in [WHSZ21] and extract the loss term of root joint to form our root trajectory smooth loss $\mathcal{L}_{root}$:

We minimize the differences between the root’s spatial positions and rotation angles at frame t and t -1 to enforce temporal consistency. When we impose a similar constraint on other joints, the network tends to overfit, and the final result converges to an average posture. Therefore, we only use the joint trajectory smooth loss to constrain the natural smooth transition for the root joint.

Keyframe consistency loss.

One of the main tasks of the network is to build natural connections of the generated dance with the given keyframes, which means that we need to ensure the continuity of the predicted movements near keyframes and at keyframes. Hence, the keyframe consistency loss $\mathcal{L}_{key}$ is introduced to achieve the goal:

where $m$ is the number of frames affected by the keyframes. When $N>2m$, we impose constraints on the $m$ frames near the keyframe $k_{1}$ or keyframe $k_{2}$; When $N\leq 2m$, the in-between $N$ frames are constrained to be affected by the mixture results of the keyframes $k_{1}$ and $k_{2}$. We use $\frac{t-k_{1}}{N}$ as the impact factor to determine the influence weights of two keyframes on the predicted pose at time $t$. However, when we only use the information of keyframes to calculate the keyframe consistency loss, there is a significant discontinuity at keyframes. By also imposing a temporal consistency constraint on the postures near the keyframes, the generated dance sequence can achieve smooth transitions at the given keyframes. In the training process, we set $m=5$. In experiments, we will validate the superiority of the proposed loss function.

Velocity factor consistency loss.

In order to make the velocity factors of the synthesized dance sequences consistent with the given control condition, the velocity factor consistency loss $\mathcal{L}_{vfac}$ is proposed. After obtaining the network’s outputs, we calculate the velocity factor $\tilde{\textbf{f}}_{t}$ for each frame of the generated dance motion. We formulate the velocity factor consistency loss as follows:

where $\hat{\textbf{f}}_{t}$ is the constrained velocity factor given by the user.

The dance database we use contains a total of 123 pieces of contemporary dance. Among them, $80\%$ of the data set, a total of 98 dance segments, 93347 frames are used for training, and the remaining $20\%$ of data, a total of 25 segments, 20897 frame samples are used for testing. For the network, the state encoder, the velocity encoder, and the target controller are all composed of two fully connected layers. The hidden units for the two layers are set as 512 and 256, respectively. The position encoder consists of one fully connected layer with 512 hidden units. For the Transformer-based controllers, i.e., the root trajectory controller and the velocity factor controller, the temporal window size $u$ is 7, the head number and the embedding dimension for the MHSA block are 8 and 32 respectively, and the dimensions for the three linear layers are set as 256. For the two LSTM blocks, the hidden units are set to 256. Both the state decoder and the root decoder consist of three fully connected layers, where the dimensions for the two hidden layers are 512 and 256, respectively. We use the Parametric Rectified Linear Unit (PReLU) [HZRS15] as the activation function for all the encoders, decoders, and the target controller, while for the two Transformer-based controllers, we follow the same activation function and structure as the typical Transformer does [VSP^∗17].

We implement our model using PyTorch, and all models used in this paper are trained on a GeForce RTX 2080 Ti GPU. When training the model, we use the Adam optimizer [KB15] and set the learning rate to 0.0001 and the batch size to 128. It takes 43 hours to train the proposed network and 0.0085 seconds to generate a single frame during testing. Since our model can generate variable-length motion transition sequences, we also use variable-length dance sequences to train the network. The minimum sequence length $min\_len$ used in training is $5$ and the maximum sequence length $max\_len$ is $70$. For each epoch, we set the current $min\_len$ and $max\_len$ to represent the sequence length range of the current epoch. At the early stages of training, both $min\_len$ and $max\_len$ are set to $5$, and the $max\_len$ increases by 1 after each epoch. In this way, the amount of training data in each round gradually increases as the training progresses. To prevent repeated learning on dance sequences of small lengths, we let $min\_len$ increase by $4$ every five epochs. When $max\_len$ is greater than $70$, the training process is terminated.

Our model can achieve fine-grained control for dance movement generation through the skillfully designed control constraints. We conduct experiments to verify the effects of different control constraints with only one of the constraints changed at a time.

Different root trajectories.

In order to verify the effect of the root trajectory, we visualize the generated root trajectory and the specified control trajectory to compare the difference between them in Figure 5. It can be observed that the motion sequence generated by our model can well fit the root trajectory conditions specified by the user. Meanwhile, we also defined a quantitative index to evaluate the accuracy of the predicted root trajectory. The quantitative index is explained in the ablation experiment (Section 4.2).

When the user provides different root trajectories or different root heights, it should produce different actions. For example, if the root joint reaches a high position, a jump action should be generated, and a squat action should be produced if the root joint is low. The phenomenons can be observed in our results (Figure 6), which demonstrates the close correlations between the generated motions and the height of the root joint. Furthermore, we conduct experiments with totally different root trajectories while maintaining the same keyframes and the velocity factors. The results are illustrated in Figure 7. We can observe that various transition movements are generated, and the diversity of dance motion is significantly enhanced under the control of the root trajectory. The above experiments validate that the root trajectory constraint can well control the global position of joints, affect the types of actions, and promote the diversity of dances.

Different velocity factors.

When other conditions are the same, that is, the keyframes are the same, and the root trajectory constraint is also the same, we change the values of velocity factors to generate different motion sequences. When using different velocity factor sequences, motion sequences with different speeds will be generated. In order to enable users to control the overall speed better, we allow users to use a constant value to achieve multiple changes to the velocity factor sequences. Figure 8 illustrates the generated dance sequences with the velocity factors varied according to the multiples of 1.0, 0.5, and 1.5. We visualize the generated motion movements every five frames. From Figure 8 and the supplementary video, we observe that when the velocity factors become half of the original, the overall speed of the motion sequence becomes very slow, and the variety of actions is significantly reduced. Besides, the motion movements tend to go directly from one keyframe to another keyframe. However, when the velocity factors increase 1.5 times, the overall movements become faster and change dramatically. Because of the enlarged velocity factors, the generated action will not slide directly from one keyframe to another keyframe, and the diversities of intermediate movements have been promoted to a large extent.

The above experimental results illustrate that the same multiple changes are applied to the five velocity factors. Beyond that, users can assign different velocity factors to different body parts to achieve more fine-grained motion control. We implement experiments to change the velocity factors of different body parts. The generated dance movements are displayed every five frames in Figure 9, where Figure 9 (a) is the synthesized motion sequence with the original velocity factors, while Figure 9 (b) is the generated dances with the blue body parts (right arm and left leg) higher velocity factors. It can be observed that the blue parts generate more active results compared with the yellow parts. The experiments further validate that the diversities of synthesized dances can be enhanced by varying the velocity factor constraint.

Motion transitions with different lengths.

To verify the generation ability of our model and the diversities of generated motions, we use the same keyframes to generate long-term motion transition sequences of 100, 150, 200, and 250 frames, respectively. We visualize the four generated dance sequences in 25 frames in Figure 10. It turns out that the intermediate actions are still diversified when the length of the generated sequence is 200 frames. When the length exceeds 200 frames, repeated meaningless actions begin to appear, which can be seen in the last row of Figure 10 and the supplementary video. The results further reflect the importance of the two control conditions, i.e., the root trajectory and the velocity factor, to the generation of the action. The existence of the two terms can make the synthesized action more diversified and realistic.

We conduct ablation experiments to verify the superiority of different modules of our network. We define three quantitative metrics: the evaluation criterion of joint position, the evaluation criterion of velocity factor, and the evaluation criterion of root trajectory. The units of the three metrics are cm, cm/frame and cm, respectively.

Inspired by the L2P criteria [HYNP20], we propose the average L2 distances of root-relative position (LRP) to measure the deviation between the predicted motions and their ground truth:

where $\mathcal{D}$ is a test set, $\mathcal{|D|}$ is the number of $\mathcal{D}$, and $c$ is a transition sequence of $\mathcal{D}$. $\tilde{\textbf{p}}_{t}^{b}(c)$ refers to the root-relative positions of skeleton joints at time $t$ in sequence $c$, and $\textbf{p}_{t}^{b}(c)$ is the related ground truth. The smaller the distance error is, the more accurate the predicted movements are.

We use the accuracy rates for the evaluation indicators for both velocity factor and root trajectory. We define two fault-tolerant thresholds, $\delta_{v}$ (in cm/frame) and $\delta_{r}$ (in cm), to calculate the corresponding accuracies. We calculate the difference between the ground truth and the inferred result from the predicted motion sequence for the velocity factor. If the difference is less than $\delta_{v}$, the result is correct; otherwise, it is considered to be out of the range of correct values. The average accuracy of the velocity factor (AVF) can be obtained by:

where $g_{v}(c)$ is the deviation of the velocity factor for dance sequence $c$, $\tilde{f}_{t}^{i}(c)$ is the predicted value of body part $i$ at timestep $t$ in sequence $c$, and $\hat{f}_{t}^{i}(c)$ is the corresponding ground truth.

The average accuracy of the root trajectory (ART) can be obtained similarly as the AVF:

where $g_{r}(c)$ indicates the deviation of root trajectory for dance sequence $c$, $\tilde{\textbf{p}}_{t}^{r}(c)$ is the predicted root joint position at timestep $t$ of sequence $c$, and $\hat{\textbf{p}}_{t}^{r}(c)$ is the ground truth. During the experiment, we take $\delta_{v}$ as 1.0 and $\delta_{r}$ as 7.0.

We report the ablation results of the three quantitative metrics under different model settings in Table 4, Table 4, and Table 4. Specifically, the “One LSTM” model represents that the generation model contains one LSTM unit to deal with postural information and control conditions. “Condition-FC” refers to replace the Transformer-based controllers with two fully connected layers to encode the constraints of root trajectory and velocity factor. “One decoder” means that the state decoder decodes the posture information of all skeleton joints, including the root joint. “Without Velfac constraint” stands for the model without the velocity factor constraint. “Without $\mathcal{L}_{key}$” denotes we train the whole model without the keyframe consistency loss $\mathcal{L}_{key}$. “Whole model” is the overall framework learned with all loss functions. “Interpolation” is the model that utilizes the interpolation strategy to synthesize transition action between keyframes. At last, Harvey’s method [HYNP20] in terms of LRP in Table 4 has also been reported.

Effectiveness of two LSTM units.

To verify the effectiveness of using two LSTM, we conduct experiments by learning the model with one LSTM. It turns out that the dance sequence generated with one LSTM performs less well than the “Whole model”, leading to much larger LRP errors. The results can be observed in Table 4. For the velocity factor, the accuracy produced by the “Whole model” is much higher than the “One LSTM”, as shown in Table 4. As for the root trajectory, when the sequence length is less than 100, the accuracy of the “Whole model” far outstrips the model learned with one LSTM. However, when the predicted length starts larger than 150, the “Whole model” is slightly worse than “One LSTM”. The reason may be that one LSTM unit cannot reach a balance between various control conditions. Instead, it tends to dominate the control of the root trajectory. Hence the results of posture and velocity factor become worse. On the contrary, by processing different information hierarchically and sequentially, the whole model with two LSTM reaches a good balance between different constraints and achieves better results on the three criteria.

Effectiveness of Transformer controller.

Due to the multi-head self-attention layer mechanism in Transformer, the Transformer-based controllers can well leverage the context information in the time window of the frame to be predicted to encode the control signals. We use the fully connected layer to replace the Transformer to validate its importance on constraint control. By comparing the “Condition-FC” with the “Whole model” in Table 4, Table 4, and Table 4, we observe that the performance of the complete model is far better than that of the model constraint with the fully connected layer in terms of the joint position error, velocity factor and root joint prediction accuracy, demonstrating the superiority of Transformer structure in constraints modulation.

Effectiveness of the root decoder.

Existing methods usually employ the state decoder to predict the posture information of all skeleton joints [HYNP20, HP18]. However, for dance, the action space of dance movements is vast. Only one decoder may cause large jitter in the generated root trajectory. To verify the effectiveness of the root decoder, we conduct experiments to obtain the prediction results with only the state decoder. Comparing the experimental results of “One decoder” and “Whole model” in Table 4, Table 4, and Table 4, we can observe that the LRP and the AVF of “One decoder” are similar to those of the complete model, but the ART is inferior. We also illustrate the synthesized dance sequences and the corresponding root trajectories of “One decoder” and “Whole model” in Figure 11. It can be observed that the root trajectory obtained by using one decoder has a larger deviation from the target root trajectory, while the result obtained by using two decoders is more consistent with the target root trajectory, which validates the advantages of the root decoder.

Effectiveness of velocity factor constraint.

We implement experiments to verify the affect of the velocity factor constraint in motion synthesis by removing the corresponding controller. It can be seen from Table 4 when velocity factors are not used for control, the AVF evaluation metric is $56\%$. After adding the velocity factor constraint, the AVF performance has improved to $74\%$. In addition, the whole model has significantly improved the model “Without Velfac constraint” in terms of the evaluation criteria of LRP and AVT as illustrated in Table 4 and Table 4. Three quantitative experimental results confirm the importance of the velocity factor constraint.

Effectiveness of keyframe consistency loss.

Keyframe consistency loss aims to constrain the motion sequence transited natural and smooth near keyframes. To verify its importance, we conduct experiments by learning the network without the $\mathcal{L}_{key}$ function. In the experiment, we find that the performance of the three quantitative evaluation indexes of the model is greatly improved by adding the keyframe consistency loss. The detailed results are illustrated in Table 4, Table 4, and Table 4. The achieved significant gains demonstrate that the keyframe consistency loss has a positive effect on the optimization of the network.

Experiments on dance dataset.

We also compare our model with two related methods to validate its superiority. Specifically, we make a comparison with Harvey’s method [HYNP20] and the results obtained by interpolation strategy. When training Harvey’s model [HYNP20], we employ the same learning strategy on the dance dataset we use. It turns out that the generated motion sequences are far from satisfactory, many synthesized actions are unnatural, and the continuity at the keyframes is poor. The main reason for the results may be that the network is mainly designed for walking, running, and other simple movements, while the dance movements are relatively complex. Its variability makes it impossible to uniquely describe the movements using only keyframes, which may easily cause ambiguity. Therefore, the final synthesized results are performed poorly. For the interpolation strategy, we take the root trajectory and keyframes as the control conditions. We spherically interpolate the quaternions between the keyframes to complete the motion transition.

We also calculated the LRP errors of the two methods. The comparison results are reported in Table 4. It can be observed from Table 4 that within short-term motion transition, e.g. $N\leq 10$, the interpolation method shows a decisive advantage. It is partly because the motion becomes almost linearly in a sufficiently short timescale. However, with the increasing length of the prediction sequence, our method obviously exceeds the interpolation-based strategy. In addition, the results in Table 4 also prove that our method is always better than Harvey’s model [HYNP20] under different prediction sequence lengths and far exceeds it by a large margin.

We show the visual comparison of the dance sequences generated by our algorithm and the above two algorithms, as well as the ground truth (GT) sequences in Figure 12. It can be seen that the motion generated by Harvey’s method [HYNP20] are noisy and unnatural, and there are obvious discontinuities at the keyframes. The results obtained by interpolation lose the diversity of actions, and the steps are always floating. In contrast, our method improves the problem above and can generate more continuous and smooth motion movements. The qualitative and quantitative comparative experiments demonstrate that our model has better generation ability in complex dance synthesis and can achieve fine-grained control through the root trajectory and velocity factor constraints.

Experiments on cyclic motion dataset.

We select 97,123 frames related to walking and running from the dataset used by Harvey. To make a fair comparison, we train our model and Harvey’s method [HYNP20] under the same environments using the selected motion samples. The generated results are illustrated in Figure 13, as well as in the supplementary video. Since Harvey’s method does not control the global position, we modify the root joint’s position and rotation of the synthesized results to avoid the influence of global information on the visual effect. From Figure 13 and the supplementary video, we observe that Harvey’s method can generate natural and smooth motion transitions between keyframes for running motion. In contrast, our method has achieved comparable performance to Harvey’s method on the cyclic motion dataset from the visual effect. However, when it comes to the challenging non-cyclic dance synthesis, our model is far better than Harvey’s method in the naturalness and variety of dance movements, as shown in Figure 12 and the supplementary video. The experimental results validate that our model can not only control complex non-cyclic dance generation, but is also suitable for simple cyclic locomotion synthesis. Therefore, our approach demonstrates good robustness on a variety of datasets.