MotionRL: Align Text-to-Motion Generation to Human Preferences with Multi-Reward
Reinforcement Learning

Many works have attempted to generate motion from text, primarily because language input is very convenient for users. Recent mainstream research has employed stochastic models for motion generation. For instance, the diffusion method (Zhang et al., 2024; Chen et al., 2023; Tevet et al., 2022) describes motion generation as a denoising process conditioned on text. Some BERT-type masked generative models (Guo et al., 2023) have also successfully tackled this task. However, these approaches face a significant limitation: they require the motion length as input to guide generation, which can lead to significant motion quality degradation (Pinyoanuntapong et al., 2024). In contrast, GPT-type generative models (Zhang et al., 2023a; Guo et al., 2022c) solve this issue by first learning a VQVAE (Van Den Oord et al., 2017) to map continuous motion into discrete motion tokens, and models like Zhang et al. (2023a) and Mao et al. (2024) employ a transformer decoder to perform next token prediction.

Human evaluation of generated motion should be highly correlated with motion quality and is arguably more important than traditional metrics such as FID or R-Precision(Heusel et al., 2017). However, in the text-to-motion domain, few studies incorporate human perceptual priors. Recent works, such as Voas et al. (2023), have established datasets of human motion ratings, and others, like Wang et al. (2024), use contrastive methods to improve the robustness of human evaluations. However, these approaches directly integrate ratings into the training process, which can negatively impact other metrics and fail to capture subtle human perceptual differences.

Reinforcement Learning from Human Feedback (RLHF) (Nguyen et al., 2017) has achieved tremendous success in fine-tuning generative models, whether for text generation(OpenAI, 2024) or image generation(Fan et al., 2024). RL effectively aligns model outputs with human evaluations and offers significant advantages over explicit alignment methods for complex optimization objectives. OpenAI’s technical report (OpenAI, 2024) noted that fine-tuning pretrained GPT models with RL helps models better understand human perceptions. However, in the text-to-motion domain, few studies use RL to align with human perceptions. For example, Mao et al. (2024) uses RL to guide generation, but without incorporating any human perceptual priors, focusing only on enhancing text-to-motion alignment.

A common method (Zhang et al., 2023a) to design a motion generator is using vector quantized variational autoencoder (VQ-VAE) (Van Den Oord et al., 2017) and generative pre-trained transformer (GPT). In T2M-GPT, the trained VQ-VAE maps motion sequences $m=[x_{1},x_{2},\dots,x_{T}]$ to a sequence of indices $S=[s_{1},s_{2},\dots,s_{T/l},End]$ which are indices from the learned codebook $C$. Note that a special $End$ token is added to indicate the end of a motion. The VQ-VAE uses a standard CNN-based architecture and is trained following the details in T2M-GPT (Zhang et al., 2023a).

With a trained VQ-VAE, the motion generation task can be described as a next-token prediction task. In this stage a transformer is designed to autoregressively predict the next index $s_{i}$ based on previous indices set $S_{<i}$ and text condition $c$. The optimization goal in this stage is to maximize the distribution $p(S|c)=\prod_{i=1}^{|S|}p(s_{i}|c,S_{<i})$. We train the GPT with cross-entropy loss $\mathcal{L}_{CE}$.

Some relevant works introduce a sequence-level semantic supervision to help motion-text alignment. Specificlly, pre-trained motion encoders and text encoders like TMR encoders (Lu et al., 2023) can measure the similarity between the input text embedding and output motion embedding. The text embedding from the encoder aligns better with human motions than that from CLIP (Radford et al., 2021) or other LLMs. It also provides a way to directly supervise the text-motion alignment. So we calculate an additional loss $\mathcal{L}_{align}$ for supervising the alignment.

Lin et al. (2022) introduced the use of preference information in multi-objective optimization. In MotionRL, reward-specific identifiers are employed, which allow the model to differentiate between various types of rewards during multi-objective optimization. The details about sampling and token design and are in the Appdix B.

Regarding the design of the reward models, we define three specific objectives to fine-tune the model’s output: text adherence, motion quality, and human preferences. The first two rewards are derived from encoders trained on paired motion and text data, while the latter is obtained from a model trained on real human preference data.

For text adherence and motion quality, we use the paired text encoder and motion encoder from Guo et al. (2022a) and Lu et al. (2023). The training objective of the encoder in Guo et al. (2022a) is:

where $\mathbf{f}_{t}$ and $\mathbf{f}_{m}$ denotes the embeddings extracted by text and motion encoders, respectively. $y$ is a binary label indicating the matched text-motion pairs and $m$ is the manually set margin.

The training objective of the encoder in Lu et al. (2023) is:

where $\tau$ is learnable temperature parameter and $B$ is the batch size.

Now we have pretrained motion encoders and text encoders, we can define the text adherence reward as:

where $\mathbf{f}_{m_{\mathrm{pred}}}$ is the generated motion contioned on $t$ , $i$ represents different encoders and $\lambda$ represents their respective weights, used to constrain the values to a similar magnitude. Higher rewards indicating a better match between the generated motion and the input text.

Similarly, the motion quality reward is defined as:

where $m_{\mathrm{gt}}$ is the ground truth motion sequence. Higher rewards signify that the generated motion is closer to the ground truth.

We also need a model to align the human preferences. Giving an input motion sequence $m$, an implicit perception model $\mathcal{P}$ is assumed, where a higher rate indicates that the motion has better quality. To align the generated motions with human perception, we need a computational perception model $\mathcal{C}$ that best aligns $\mathcal{P}$. We use the model from Wang et al. (2024) as our human perception model. The pairwise comparison annotations from the collected dataset $\mathcal{D}$ can be used to calculate the training loss:

where $m^{(h)}$ is the better motion and $m^{(l)}$ is the worse one. After that, the model $\mathcal{C}$ is able to map the high-dimension motion to a reward $r_{p}$ as the motion rating. Therefore the human preference reward is defined as:

where $g$ is a function that converts human motion from a 3D coordinate parameterized form to an SMPL (Loper et al., 2015) parameterized form. Here we train a simple neural network to achieve this goal rather than traditional methods. Please refer the Appendix A.

We normalized three different types of rewards to constrain them within the same order of magnitude, ensuring that the numerical differences between the various rewards do not vary significantly. For specific normalization methods, please refer to Appendix C.

Lin et al. (2022) pointed out that using batchwise Pareto-set learning and selecting good samples in a batch can approximate Pareto-optimality across multiple objectives. In the preliminary stage, for $k$ types of rewards, we generated $k$ texts with special tokens, and each text sampled $N$ motions respectively. Therefore, MotionRL will find the corresponding Pareto-optimal set in this $N$ batch denoted as $\mathcal{M}$. For the different sampled motions $m_{i}$, the dominance relationship is defined as follows:

Let $m_{a},m_{b}\in\mathcal{M}$, $m_{a}$ is considered to dominate $m_{b}$, denoted as $m_{a}\succ m_{b}$, if and only if $r_{k}(m_{a})\geq r_{k}(m_{b}),\forall k\in\{1,\dots,K\}$ and $\exists j\in\{1,\dots,m\}$ such that $r_{j}(m_{a})>r_{j}(m_{b})$.

In order to enable the model to approximate Pareto optimality during the training process, we designed the algorithm as Algorithm 1.

We employ reinforcement learning to optimize the alignment between motion sequences and textual descriptions, human perception preferences, and motion quality. A Pareto-based multi-reward objective guides this process, balancing the different rewards.

The actor model $\pi_{\theta}$ generates motion sequences by selecting motion tokens at each time step based on the input text, while the critic model $V_{\phi}(s_{t})$ estimates the value of the current state. The objective function is:

In the above equation, $r(t_{k},m_{k})$ represents the multi-objective reward function evaluating the alignment between the textual description $t_{k}$ and the generated motion sequence $m_{k}$, taking into account factors like semantic consistency, human preference, and motion quality. $\pi_{\theta}(m_{k}\mid t_{k})$ is the actor model that predicts the motion $m_{k}$ based on the input text $t_{k}$, and $\pi_{\text{ref}}(m_{k}\mid t_{k})$ is the reference model serving as a constraint to regulate how much the updated policy can deviate from the pre-trained model. The term $\beta$ controls the strength of this regularization.

We employ Proximal Policy Optimization (PPO) (Schulman et al., 2017) for training, where the advantage function is computed as:

Here, $A_{t}$ measures the relative improvement of the selected action at time $t$ compared to the expected value. $G_{t}$ denotes the return, or the total reward accumulated from time step $t$, and $V_{\phi}(s_{t})$ is the value function estimated by the critic, representing the expected return at state $s_{t}$.

The actor’s loss function is given by:

where the clipping function ensures that the policy update is constrained to prevent large deviations from the previous policy. The critic model is updated by minimizing the squared error between the predicted value and the actual return:

This framework allows both models to effectively learn from multiple reward signals, balancing semantic alignment, human preferences, and motion quality in the generated sequences.

We select InstructMotion as our baseline model. The text-to-motion transformer is composed of 18 layers, each with a hidden size of 1,024 and 16 attention heads. For the PPO algorithm, we use a mini-batch size of 32 during training. We run the PPO algorithm for 2 epochs, using the AdamW (Loshchilov & Hutter, 2017) optimizer with $\beta_{1}=0.9$ and $\beta_{2}=0.99$. The pre-trained text-to-motion generator is fine-tuned over 40k iterations, with a learning rate of 5e-6. All experiments are performed on 4 NVIDIA RTX 3090 GPUs. The specific method for reward normalization is detailed in Appendix C, while the process and results of the user study are elaborated in Appendix D.

Tables 1 provides an evaluation of our MotionRL framework on the widely used datasets: HumanML3D. The results indicate that MotionRL outperforms the baseline models, T2M-GPT and InstructMotion, by a significant margin in key quantitative metrics such as R-Precision and FID. This demonstrates that the motion sequences generated by our model exhibit stronger alignment with the corresponding textual descriptions and better quality of motion.

It is important to note that ^§ indicates reliance on the ground-truth sequence length for generation. All methods that depend on the ground-truth motion length tend to perform better on the FID metric because they ensure that the motion lengths are consistent with the test set prior to inference. However, Pinyoanuntapong et al. (2024) points out that superior FID scores for these methods do not necessarily imply higher motion quality. To demonstrate the superiority of MotionRL, we utilize the scores from the motion perception model provided in Wang et al. (2024) alongside real human user study ratings.

Figure 3(a) illustrates the output scores from the motion perception model in Wang et al. (2024). It is evident that our model achieves higher perceptual scores compared to other models, indicating that it effectively captures human preference information embedded in the perceptual model, leading to motions that align more closely with human perception. Figure 3(b) presents the scores given by real human evaluators for the model-generated motions. We compared the success rates of MotionRL against other models, further demonstrating the overall superiority of our model in terms of motion quality.

To validate the superiority of MotionRL in generating high-quality actions, we compared MotionRL with other well-performing models, including T2M-GPT, InstructMotion, and Momask, based on several prompts from the test set. As shown in Figure 4, our method generates actions that align with the text, whereas the other methods fail to produce accurate actions. For example, the other methods did not generate the motion of pushing the arms forward and could not comprehend the motion of walking diagonally two steps to the right. In contrast, our method produces high-quality motions that correspond to the text. More details can be found in the supplementary materials.

Reward Design: Our framework employs a multi-objective optimization approach with three distinct rewards. To verify that our method effectively captures human perceptual preferences, we conducted numerical experiments on the HumanML3D test set. By using different reward combinations, we assessed our method’s performance in terms of both traditional metrics and human perceptual model scores. We observed that the perception reward we designed effectively enhanced the output of the human perception model. Additionally, the rewards we developed for motion quality and text adherence significantly improved performance on both FID and Top-1 Precision metrics.

Pareto Optimality: Instead of transforming the multi-objective optimization into a single reward through weighted summation, MotionRL approximates Pareto optimality within each batch of samples. To verify the superiority of our method, we compared the effects of using batch-wise Pareto selection and different reward-specific tokens on model rewards. As shown in Figure 3, the use of Pareto optimization effectively improved the overall reward values of the model. Additionally, employing different reward-specific tokens allowed effective control over the model’s output. It is important to note that comparisons of reward values across different optimization objectives are meaningless because our normalization process during training constrains the rewards to the same scale. However, the physical meaning of the values for different objectives varies, and only the relative magnitudes of rewards within the same optimization objective are meaningful.