MAAIG: Motion Analysis And Instruction Generation

Several research studies on Motion2text (Guo et al., 2022) (Jiang et al., 2023). In their work (Guo et al., 2022), the author introduce a novel approach in which they transform a sequence of motions into a tokenized motion representation within a Code Book, employing the VQ-VAE (van den Oord et al., 2017) framework. This process generates a compact yet semantically rich representation for 3-D motions. Leveraging quantified embedding vectors (namely motion tokens ), the authors employ a Transformer model to build a mapping from human motions to textual descriptions. This approach proves effective in connecting quantified motion tokens to quantified text tokens, enhancing the text generated by model can better describe the sequence of motion. However, it is worthy noting that the aforementioned research primarily focus on generating text which can capture the semantic essence of an entire motion sequence. When it comes to the task ”motion to instruction”, it is still too difficult to capture the detailed information of the sequence of motions.

In (Jiang et al., 2023), MotionGPT employs T5 as its underlying language model with motion-aware capabilities. The research involves two stage process: initial train models with pretraining and then instruction tuning. This approach results in improvements across various tasks and enhances the model’s performance, particularly when faced with previously unseen tasks or prompts.

In (Zhao et al., 2022), the framework for providing exercise feedback relies on Graph Convolutional Kernel. The author employee a classifier model to predict action labels (namely instruction labels). It is worthy of noting that instead of utilizing the 3D joint positions, the author take the DCT (Ahmed et al., 1974) coefficients of joint trajectories as input of the classifier model. Furthermore, the predicted action labels are input to a feedback model, subsequently converted into the tensor. This resulting tensor is then fed into a correction model along with the DCT coefficients of joints. This mechanism enable the correction model to improve the accuracy of the corrected motion. This exercise feedback framework can generate the instruction label. We believe that the output text would be more valuable if it could provide precise instructions rather than just labels. In this way, the framework can offer more meaningful guidance for exercise routines.

The motion to instruction model takes skeleton-based data as input, which is obtained from pose estimation algorithms from videos proposed by HybrIK model (Li et al., 2020), which estimates 3D pose via the twist-and-swing decomposition. Because our motion to instruction model is dedicated to assisting figure skaters in improving their athletic postures, and figure skating involves a multitude of airborne rotations and body twists, we choose the HybrIK (Li et al., 2020) model to assist us in constructing the skeletal structure from the video.

Additionally, it’s essential to note that figure skating takes place on an ice rink and our goal aims to provide accurate instruction. Therefore, we also endeavor to construct a 3-D skeleton to provide the motion to instruction model with distance-related information.

The input data consists of a sequence of frames, with each frame containing a set of 3-D pose joint coordinates. These coordinates are relative to the human body-part in the SMPL (Loper et al., 2015) format with dimension ( 22x3 ). ”22” means the number of joints and ”3” means the x, y, z coordinates for each joints.

Here we describe the proposed model in detail. The inputs to the model consist of 3-D skeleton-based data, with joint dimensions represented as ( 22x3 ). Additionally, we apply linear transformation to convert our 3-D skeleton, which initially has a dimension of 66, into 512-dimensional embedding.

Our model is built upon the T5 (Text-to-Text Transfer Transformer) (Raffel et al., 2019) architecture introduced by Google Research in 2019, which represents a significant advancement in the field of natural language processing (NLP). T5 is a variant of the Transformer (Vaswani et al., 2023) architecture, notable for its pioneering self-attention mechanism that enables the model to process positional information across input sequences simultaneously. This capability is particularly valuable for capturing long-range dependencies in text. The fundamental idea behind T5 is to frame various NLP tasks as text-to-text transformations. It adopts an end-to-end approach where both inputs and outputs are treated as text sequences. This implies that tasks spanning question answering, text classification, text generation, summarization, translation, and more can all be cast as processes that map one text sequence to another. T5’s architectural design comprises two essential components: the encoder and the decoder, which are the core constituents of the Transformer architecture. The encoder is responsible for encoding input text into an intermediate representation, while the decoder deciphers this intermediate representation into the desired output text. During the training process, T5 learns its parameters by minimizing the disparity between predicted output text and the actual target text. This enables the model to generate high-quality outputs during inference. Whether the task at hand involves answering questions, classifying text, generating text, summarizing, or translating, T5’s text-to-text framework offers a unified approach. This versatility empowers T5 as a robust and adaptable NLP model, obviating the necessity to design distinct models for individual tasks. Instead, it requires only adjustments to input and output text specifications.

To tokenize instructions, we used the Tokenizer from mT5 (multilingual Text-to-Text Transfer Transformer). mT5 is a multi-language NLP model based on the Transformer architecture, known for its excellent cross-language processing capabilities. This tokenizer facilitates the tokenization, processing, and representation of textual instructions in a manner compatible with our model’s architecture. Thus, the model can simultaneously handle 3D skeletal data and textual instructions to perform the required tasks.

Furthermore, to equip the model with the capability to handle 3-D skeletal data and generate language, we conducted pretraining using the HumanML3D dataset (Plappert et al., 2016) through T5. HumanML3D is a pretrained three-dimensional human pose recognition model designed to integrate three-dimensional skeletal data with tasks related to human actions. This model focuses on understanding and interpreting three-dimensional human poses, which hold significant value in various applications such as sports analysis, motion capture, human-computer interaction, and virtual reality. Key features and functionalities of HumanML3D include processing three-dimensional skeletal data, pretrained representations, multi-task learning, and real-time performance. It can accept three-dimensional skeletal data captured from deep learning cameras, sensors, or other devices as input, identify and track the positions, orientations, and movements of human joints, and reconstruct three-dimensional human poses. Through large-scale pretraining, HumanML3D can acquire a universal representation of human poses applicable to various tasks and datasets.

Given the lack of existing motion and instruction pair dataset, We collect our own figure skating dataset for this specific task by our self-developed annotation instruction system. Experts can select specific time segments from the database’s videos and annotate them with instruction. An example of the selected motion clip and its corresponding instructions are shown in Figure 2. In this section, we outline the process for acquiring the YourSkatingCoach, which involves three distinct phases: Motion Collection, Instruction Collection, and Dataset Generation.

Motion Collection. We obtain video footage featuring two skaters performing the Axel jump, a maneuver in skating that involves transitioning from the forward outside edge of one skate to the backward outside edge of the other, encompassing one (or more) and a half turns in mid-air. Subsequently, we categorized these videos into two distinct groups: ”Axel” for single jump performances and ”Axel_com” (Axel combo) for sequences comprising one or more consecutive jumps. Nevertheless, due to the inadequate collection of data in the Axel dataset, we have opted to exclusively utilize the Axel_com subset in this experiment. At first, there were 89 videos of athletes performing these two motions.

Instruction Collection. To acquire the instructions for the motions we collected, we hire a professional skater to annotate the video tapes we collected. The annotation process works as follows: firstly, the skater watch the video taken from the last phase and select a ”start time” and an ”end time” of the clip and leave an instruction on how the performers can improve their Axel moves. In the end, we clip out 164 motion clips from the original 89 videos, and divide them to 90/10 for training and testing.

Dataset Generation. With the motion dataset obtained from the Motion Collection and the instruction dataset acquired through the Instruction Collection, we can now construct a comprehensive dataset composed of pairs comprising motion clips and their respective instructions. This process involves segmenting the videos based on the ”start time” and ”end time” chosen by the annotator, followed by merging them with the corresponding annotations provided by the annotator. In scenarios where multiple instructions are assigned to the same time interval within a single video, we unify these instructions into a coherent directive by connecting them using a designated separator symbol.

Initially, we embark on model development from the ground up using T5 architecture. However, this approach yields subpar results owing to the inherent limitations of our YourSkatingCoach Dataset, which comprises a relatively modest collection of just 164 video clips. The constraints imposed by this dataset made it unfeasible to train a sufficiently compact Large Language Model capable of processing textual content effectively.

In response to these constraints, we turn to the HumanML3D dataset (Petrovich et al., 2022), a valuable resource encompassing a substantial repository of 44,970 textual descriptions corresponding to 14,616 3D human motions. This dataset serves as a pivotal pretraining source for our model, empowering it to attain a holistic comprehension of motion representations.

However, it has come to our attention that there exists a difference in the skeletal data between YourSkatingCoach and HumanML3D. Specifically, the data in YourSkatingCoach is represented in a local coordinate system, while HumanML3D utilizes the world coordinate system. As a result, we have explored an alternative setting where we remap HumanML3D into the local coordinate system to address this discrepancy.

In addition to our exploration of the T5 architecture, we extended our experimentation to include the Transformer architecture. Our experiments cover the following six distinct settings:

1. (1)

   Train from scratch using Transformer.

2. (2)

   Fine-tune from HumanML3D pretrain in world coordinate system using Transformer.

3. (3)

   Fine-tune from HumanML3D pretrain in local coordinate system using Transformer.

4. (4)

   Train from scratch using T5.

5. (5)

   Fine-tune from HumanML3D pretrain in world coordinate system using T5.

6. (6)

   Fine-tune from HumanML3D pretrain in local coordinate system using T5.

In Table 1, we present the results of our experiment. Two key observations emerge from this table: (1) Pretraining on a large dataset helps mitigate the limitations of a small dataset. (2) Converting the coordinate systems further enhances performance.

Comparing With or Without Pretraining. The table clearly illustrates the substantial performance improvement achieved by using pretraining across both the Transformer and T5 architectures. This demonstrates the efficacy of pretraining on a large dataset in overcoming the limitations of a smaller dataset, thereby enhancing the language understanding within our model. Notably, all scores exhibit a significant elevation when HumanML3D is utilized as the pretraining source.

Comparing With or Without Converting Coordination. The table reveals that performance is notably enhanced when employing the HumanML3D dataset in the local coordinate system compared to the world coordinate system. This observation underscores the critical role of data compatibility in transferring language understanding from one dataset to another. Remarkably, when YourSkatingCoach and HumanML3D share the same coordinate system, they yield the optimal result in the T5 architectures in the setting of Fine-tune from HumanML3D pretrain in local coordinate system.

While pretraining on the large dataset with T5 significantly boost the performance, there were still parts where the model performed poorly. For example we observe that the model cannot distinguish left from right and vice versa. We suspect that this can be attributed to the scarcity of the annotation regarding left and right. Secondly, the vanilla transformer model have failed to generate fluent sentences, some of the common failures are: omitting verbs, repetitive words, and violating grammar, as shown in Figure 3. Though these issues were all largely mitigated when we switch to the T5 implementation, each evaluating score still remains rooms to improve.

In the future, we aim to initially convert the video actions into world coordinates system by estimating implicit and explicit camera parameters to let model learn left and right. After converting human pose from camera coordinate system to world coordinate system, we will train our ”MAAIG” model in world coordinate system, enabling it to learn the direction-related information. Subsequently, we can also take the kinematic information of skeleton in world coordinate system, e.g.: angular velocity, linear velocity, and rotation as the input of ”MMAIG” model. Nonetheless, we will employ Part-of-speech (POS) tagging on the instruction, which can ensure the output of our model is grammarly correct. Also, it is non-trivial to study the effect of different movement speed in terms of the model performance, we plan to set off a variety of experiments to ensure our model’s stability under arbitrary situations.