DiffusionPhase: Motion Diffusion in Frequency Domain

The stochastic nature of human motion has driven the application of generative models in motion generation tasks. Prior works have shown impressive results in leveraging generative adversarial networks (GANs) to sequentially generate human motion [4, 6]. Variational autoencoder (VAE) and transformer architectures have also been widely adapted for their capability of improving the diversity of generated motion sequences [12, 10, 11, 19, 3]. Most recently, [30] proposed T2M-GPT, a framework that learns a mapping between human motion and discrete codes in order to decode motion from token sequences generated by a Generative Pre-trained Transformer. Large Language Models have also been leveraged for text-to-motion tasks to boost zero-shot capabilities [33, 16, 17]. Diffusion models, with their strong ability to model complex distributions and generate high-fidelity results, have become increasingly popular for motion generation, especially for context-conditioned schemes [28, 31, 8, 18]. With a transformer-encoder backbone, [26] presented a light-weight diffusion model that predicts samples instead of noise for efficient text-to-motion generation. [32] enhanced the quality of generation for uncommon conditioning signals by integrating a retrieval mechanism into a diffusion-based framework for refined denoising. Although diffusion-based approaches for motion generation have achieved outstanding results, the synthesis of long, arbitrary-length sequences remains limited in quality and computationally expensive. Recently, [7] greatly improved the efficiency of the diffusion process by introducing a motion latent space, providing an avenue for addressing computational challenges. However, similar to prior approaches, limited data still restricts the capability of generating naturally aligned movements for arbitrary time sequences.

Previous works in the field of computer graphics have developed approaches using periodic signals to enhance movement alignment over time, incorporating phase variables that represent the progression of motion. Phase-functioned Neural Networks [15], a notable instance of such approaches, adjust network weights based on a phase variable, defined by interactions between the foot and the ground during leg movement. This principle has been expanded to include additional types of complex movements [29, 23], and for complex movements involving multiple contact points [24]. Periodic signal analysis of such complex movements have shown to be important in real-world motion control tasks, contributing to the optimization of complex movements for robotic systems [22, 9]. Building on the idea of periodic representations, [25] developed a neural network structure that is capable of learning periodic characteristics from unstructured motion datasets. However, this model can only be trained on one type of motion data at a time. In addition, the phase information obtained is not well aligned with semantic signals, such as textual descriptions, limiting intuitive motion generation. Inspired by these observations, we propose DiffusionPhase, a framework that incorporates periodic motion representations with diffusion models to achieve text-driven generation of a wide range of periodic natural movements of arbitrary lengths.

Most sequences in the dataset include separate segments at the beginning and end, showing the initiation and conclusion of the movement. These segments do not reflect the essence of the intended action itself. We enhance our method by excluding these segments from being encoded into the periodic phase manifold, while using the linear signals for encoding them. Hence, we focus on encoding the primary motion into periodic phase manifolds, and then proceed to extend them during the testing phase.

To efficiently and robustly detect main segments of motion, we first process the whole motion sequence from a motion dataset into a latent space, where each frame $i$ of the sequence has a feature vector $d_{i}$ that contains temporal and spatial features of the pose at the frame. We tested two choices of embedding the motions. Please refer to our Suppl. Materials for more detail. Then we are able to select the start pose $t_{s}$ and end pose $t_{e}$ through:

$$\underset{t_{s},t_{e}}{\mathrm{arg\,min}}\ \lambda_{1}\cdot||d_{t_{s}}-d_{t_{e}}||_{2}-\lambda_{2}\cdot\frac{t_{e}-t_{s}}{t_{T}}-\lambda_{3}\cdot||d_{t_{e}}-d_{t_{T}}||_{2}$$

(1)

where $t_{T}$ is the actual length of this motion clip; $\lambda_{1}$, $\lambda_{2}$ and $\lambda_{3}$ are the coefficients. The first term encourages the poses at the selected start and end time to be similar, aiding us in encoding the primary motion as a motion circle. The second term ensures that the primary motion is of adequate length, while the third term excludes the separate segments.

Now, we aim to train an encoder network $\mathcal{E}$ that can encode the latent representation parameters from motion sequences and a convolution-based decoder network $\mathcal{D}$ that is trained to reconstruct the input sequences from the encoded parameters. Our latent representation $X=\{a_{i},p_{i},o_{i}\}^{M}_{i=1}$ is itself the parameters of $M$ discrete periodic phases. In particular, $\mathcal{E}$ encodes the input human motion sequence to the amplitudes $a_{i}$, phase shifts $p_{i}$, and offsets $o_{i}$ parameters of a frequency decomposition of this periodic latent representation. Then with frequencies provided, $X$ can be transformed to its time-domain phase signal $S=\{s_{i}\}_{i=1}^{M}$ via Inverse Fourier. Prior work [25] trained a periodic latent representation by extracting frequencies from motions, which were predominantly in the low-frequency range, and it specialized only in one activity type such as walking, dancing, or dribbling. Since we aim to generate a large variety of motion sequences for many different activities conditioned on unconstrained text descriptions, we need to process a training dataset with many types of actions. Hence we provide a set of positive integer frequencies $F=\{f_{i}\}_{i=1}^{M}$ and let an encoder network predict the $a_{i}$, $p_{i}$, and $o_{i}$ for each of the phases. Then the phase signal can be calculated as:

$$\footnotesize s_{i}(k)_{periodic}=[a_{i}\cdot\sin(2\pi(f_{i}\cdot k+p_{i}))+o_{i},\>a_{i}\cdot\cos(2\pi(f_{i}\cdot k+p_{i}))+o_{i}]$$

(2)

We set the lowest frequency value $min(F)$ to 1: this enables embedding the non-repeatable parts of the motion. Next, we set the highest frequency $max(F)$ to be a common multiple of all frequencies in $F$, so that the pattern of the signals can repeat smoothly forever as $k$ increases. This thus permits generating sequences of any desired length $T$ simply by setting $k=1,2,...,T$. We can regard the encoded motion sequence as one motion circle, and can smoothly transition to a new extrapolated motion circle.

Our improved encoding design boasts two primary advantages: 1) it facilitates the synthesis of motions with lengths that exceed the scope of training motion distribution, thereby broadening the spectrum of possible motions; and 2) it fosters a continuous motion embedding in the temporal domain, promoting arbitrary-length motion synthesis and seamless motion transitions.

After training the encoder $\mathcal{E}$ and the convolution decoder $\mathcal{D}$, we fix the weights of these two networks and use them as a foundation to train our diffusion module that produces $X$ from a text prompt $c$ and a conditional pose $j$. Providing our DiffusionPhase model with a variety of conditional poses can improve comprehensive coverage of diverse character motions during training, and enhance its capability to generate human motion sequences from different starting pose. It also offers a clear and coherent starting point when concatenating motions. Facilitated by our periodicity-based data augmentation, we select a $t_{s}$ from the selection pool and use the pose at $t_{s}$ to help generate this periodic signal, and this conditional pose can be randomly masked during the training for performing prediction conditioned on $c$ only.

We denote $X^{n}$ as the periodic parameters at noising step $n$. For each $J$ and a pair of its $t_{s}$ and $t_{e}$ in the training set, we derive $X^{0}=\mathcal{E}(J^{t_{s}:t_{e}})$. The diffusion noising step is a Markov process that follows:

$$q(X^{n}|X^{n-1})=\mathcal{N}(X^{n-1};\sqrt{\alpha_{n}}X^{n},(1-\alpha_{n})I)~{},$$

(5)

where $\alpha$ is a small constant that could approach $X^{n}$ to a standard Gaussian distribution. According to [14], this diffusion process can be formulated as:

$$q(X^{n}|X^{0})=\sqrt{\bar{\alpha}_{n}}X^{0}+\epsilon\sqrt{1-\bar{\alpha}_{n}}~{},$$

(6)

where $\bar{\alpha}_{n}=\Pi_{i=0}^{n}\alpha_{i}$ and $\epsilon\sim\mathcal{N}(0,I)$. For the reverse step of this diffusion, we need to denoise $\hat{X}^{0}$ from $X^{n}$. We follow the idea in [21, 26] where we directly predict $\hat{X}^{0}$ using our denoising network $\mathcal{T}(j,c,n,X^{n})$ conditioned on pose $j$, text prompt $c$ and noising step $n$ during training with the simple objective [14].

$$L_{simple}=E_{X_{0}\sim q(X_{0}|c,j),n\sim[1,N]}[\|X_{0}-\mathcal{T}(j,c,n,X^{n})\|^{2}_{2}]$$

(7)

where $N$ is the total number of denoising steps.

To evaluate the quality of $\hat{X}^{0}$, we need to process it to achieve the time-domain signals $\hat{S}$, and then apply $\mathcal{D}(\hat{S})$ to reconstruct the motion sequences $\hat{J}$. We use the loss function to constrain $\hat{X}^{0},X^{0}$ and $\hat{J},J$ in training this part of the network.

Having trained our DiffusionPhase model as above, we now discuss how to generate motion sequences. In the test stage, DiffusionPhase is given $c$ and hyper-parameter $t_{s}$, $t_{e}$. For generating the first period of signals, we can start with $\emptyset$ or a random pose as the conditional pose, and we can use the end pose of the last period for smooth transitions between the periods. The sample stage of diffusion is similar to the approach seen in [14, 26]. We start with $X^{N}\sim\mathcal{N}(0,I)$ iteratively execute the prediction $\hat{X}^{0}=\mathcal{T}(j,c,n,X^{N})$, and then this is diffused back to $\hat{X}^{n-1}\sim q(\hat{X}^{n-1}|\hat{X}^{0})$ during the noising step $n$. The whole process iteratively decreases from $N$ to 1. Following [13], we apply classifier-free guidance, where we randomly mask out $c$ and $j$ during the training and then set a trade-off between $\mathcal{T}(j,c,n,X^{N})$ and $\mathcal{T}(\emptyset,\emptyset,n,X^{N})$ during sampling.

There are two operations for synthesizing new composited motions in our setting: (a) phase repetition; and (b) transition between two phases.

Dataset   We evaluate our approach on the two most widely used datasets for text-driven motion generation tasks: KIT Motion Language (KIT-ML) [20] and HumanML3D [11]. KIT-ML includes 3,911 unique human motion sequences alongside 6,278 distinct text annotations. The motion sequences are at the frame rate of 12.5 FPS. HumanML3D is a more comprehensive dataset of 3D human motions and their corresponding text descriptions. It comprises 14,616 distinct human motion capture data alongside 44,970 corresponding textual descriptions. All the motion sequences in KIT-ML and HumanML3D are padded into 196 frames in length.

For this comparison experiment, we select 7 related state-of-the-art methods: Language2Pose [2], Text2Gestures [5], Dance2Music [1], MOCOGAN [27], Generating Diverse and Natural 3D Human Motions from Text [11], MDM, [26], MotionDiffuse [31], T2M-GPT [30] and MLD [7]. The quantitative comparison results on the HumanML3D test set and the KIT-ML test set are shown in Table 1. Please refer to Suppl. Materials for visual results.

Our method is evaluated in two settings. Default Length: We consider the 196-frame motion length used in previous works as the default length setting. $3\times$ Default Length: To further evaluate the quality of generated motion for extended lengths, we generate motions of $3\times$ Default Length. Considering that the ground truth data is limited to a length of 196 frames, we select the last 196 frames of the extended motion sequence for evaluation. If a motion is extended effectively, the last 196 frames of the generated sequence should still exhibit similar motion to the ground truth.

In particular, we test two different operation modes for generating longer sequences using our method. For the Repetition mode, we apply our diffusion model once to produce one set of parameters that determine a repetition period and repeat the sequence to form an extended sequence with $3\times$ Default Length frames. For the Generative mode, we call our diffusion model multiple times until the length of the (accumulated) generated sequence reaches $3\times$ Default Length.

We compare our approach with the top performing methods in the $3\times$ Default Length setting that are capable of specifying the length of generated motion, i.e., [11], MDM [26], MotionDiffuse [31], and MLD [7]. From Table 1 , we can see that our method demonstrates comparable performance in the Default Length setting and a significant advantage over the baselines in the $3\times$ Default Length setting.

Furthermore, to better examine the effect of the generated sequence length on the generation quality, we conduct the same experiments on the HumanML3D test set for length settings from 1 to 5 times the default length setting and plot the Top-3 R-Precision, FID, MultiModal Distance, and Diversity scores as a function of motion length. As shown in Figure 4, compared to other methods that exhibit immediate performance deterioration once the length setting increases, the quality of the sequence generated by our method is nearly length-invariant. This indicates that our method can generate sequences of arbitrary lengths with stable quality.

Here we analyze why the baseline methods have difficulties in generating long sequences. For those methods that use diffusion directly, i.e. [26, 31], long motion sequences are outside of the training distribution, and the quality is impacted immediately since the denoising module must handle the whole sequences at once. Recurrent network-based methods, i.e. [11], have the ability to handle variable lengths using an auto-regressive approach, but typically fail for other reasons; for example, the quality of generated outputs deteriorates slowly over time. Our method generalizes much better to the generation of motion sequences with longer lengths which are far from the training motion distribution.