Remembering What Is Important: A Factorised Multi-Head Retrieval and Auxiliary Memory Stabilisation Scheme for Human Motion Prediction

3D Skeleton-based human motion prediction is considered a fundamental research topic that benefits a myriad of application areas including intelligent surveillance, autonomous driving, and human-robot interaction. A variety of human motion prediction methods have been proposed that range from traditional machine learning-based methods [5, 6, 7] to deep learning-based methods [8, 9, 10, 3, 11, 4]. Most recent success within this domain has been achieved using Graph Neural Networks (GNNs), with which researchers have exploited the relationships and constraints between different body components. Specifically, the Dynamic Multiscale Graph Neural Networks (DMGNN) [3] architecture models the human body in a multi-scale graph in which nodes are body components at various scales, and edges represent pairwise relations between those components. The multi-Scale Graph Computational Unit (MGCU) is the main processing component within the DMGNN architecture in which single-scale graphs extract features at their respective scales, and these extracted features are passed through different scale graphs to connect body components across two scales. Graph connections are initialized using predefined physical connections and are adaptively adjusted during model training. Then a graph-based GRU (G-GRU) is employed to predict future poses using the extracted representations. The MGCN [11] architecture follows a similar structure to DMGNN where the authors have utilised a Scale Interactional Module (SIM) to encode the human pose at multiple scales. Similarly, the MSR-GCN [4] architecture is motivated by the concept of coarse to fine-grained prediction generation. The descending GCN blocks are used to abstract the human pose at four levels of 12, 7, and 4 joints respectively, after which ascending GCN blocks reconstruct future poses at increasingly fine-grained scales. More recently, a multi-stage prediction framework named Spatial Dense Graph Convolutional Networks (S-DGCN) is proposed in [12]. In this progressively evolving architecture, the observed pose sequence and the initial guess of the future pose, which is predicted by the previous stage, are taken as the input. The authors show that this recursively updating approach leads to the progressive improvement of the initial guess predicted by the first stage.

In a different line of work, [13] proposed a Geometric Algebra-based Multi-view Interaction network (GA-MIN) leveraging Geometric Algebra tools to mitigate feature similarity issues when aggregating high-dimensional features using a deep and multi-stage GCN. Specifically, the authors use a Graph Spectrum Self-interaction module to discover the repeated motion within the input sequences, and use a Graph Spectrum Global-interaction module to extract informative motion representations within spectrum bands. The architecture of PK-GCN [14] is motivated by the observation that sequence interpolation is easier than extrapolation. As such two networks, the named InTerPolation learning Network (ITP-Network) and Final Prediction Network (FP-Network), are proposed. In contrast to an approach where we directly extrapolate the relationship between the observed sequence and the target, the ITP-Network learns to encode the input and a privileged sequence to interpolate the in-between frames in the predicted sequence. The FP-Network receives the encoded input but the privileged sequence is not visible to it. It uses a PK-Simulator that distills the privileged sequence based on the observed sequence. As such the FP-Network is able to imitate the interpolation process. The Dynamic Pattern-based collaborative modeling network (DPnet) [15] considers preserving dynamic information of the joints. The authors show that global modeling of joint relationships could lead to the introduction of undesired trajectory constraints. To address this issue the authors propose a keyframe-enhanced module that augments the extracted temporal features by encoding the input into different-length sub-sequences. The dynamic patterns of different joints are discriminated using a dynamic pattern-guided feature extractor.

Considering the recent success of Transformer Networks in numerous sequence modeling tasks [16, 17, 18], they also have been used to model human motion sequences [19, 20]. The self-attention mechanism within the transformer has been adapted to compute the pair-wise joint relationships. However, the authors of [12] have shown that GCNs are more robust and efficient compared to transformers in modeling the pairwise relations of joints. As such, we adopt a GCN-based backbone to encode input motion sequences. However, in contrast to existing works in skeleton-based human motion prediction, we leverage a novel feature factorisation strategy to disentangle subject-specific, task-specific, and other auxiliary features from the encoded inputs. Furthermore, a novel auxiliary memory architecture is proposed to query subject-specific historical local patterns, as well as global task-specific patterns, that are embedded within the memory architecture.

Auxiliary Memory-Powered Neural Networks (AMNNs) [21, 22, 23, 24, 25, 26, 27, 28] are a recent and pivotal development within deep learning. They have shown tremendous success in automatically deriving long-term dependencies between input observations, and have been able to attain state-of-the-art results in various machine learning tasks, including, anomaly detection [23, 21, 24], interaction modelling [26], multimodal data fusion [22, 27], visual tracking [25], visual question answering [29], and human action recognition [30].

Specifically, an AMNN utilises explicit storage (memory) to store important facts and automatically retrieve relevant long-term dependencies when making a decision regarding a particular input, which is highly beneficial when extrapolating into the distant future. AMNNs fall under the category of stateful neural networks which maintain and temporarily evolve their states across the entire training/testing phase, in contrast to typical feed-forward and recurrent neural network architectures which only map the relationships within a particular input. This statefulness offers a greater utility for modeling relationships across different data elements in the dataset, enabling elevated levels of knowledge extraction.

There are three primary functions that facilitate this temporal evolution of the knowledge captured in the AMNN. A query function (also called an input controller), which is composed of trainable neural network layers, transforms the input embedding into a vector to query the memory. Using this query vector, the similarity between the content of each memory slot and the query is measured and relevant memory slots for knowledge retrieval are identified. A composer function is used to transform the content retrieved from the identified memory slots into the memory output. The final task is to update the auxiliary memory content and propagate it into its next state. A memory update/write function receives the current memory output and it generates a vector to update the memory. Then the content of the slots which we leveraged in the memory read are updated using the generated memory update vector. It can be seen that the query function plays a pivotal role in identifying salient content in the memory which relates to the current input and can aid the task at hand. However, we observe that the current single-head access schemes used by such methods limit the identification of salient memory slots, as it focuses on the entire content of input and a particular memory slot. But in applications such as human motion synthesis, there are numerous task-specific and subject-specific factors that could be represented to various degrees within the embeddings. As such, a multi-head retrieval scheme is preferred in which varying levels of attention can be paid to these numerous influential factors. To the best of our knowledge, this is the first work to propose such a multi-head knowledge retrieval scheme in AMNNs. Furthermore, we propose two novel objective functions which encourage diversity in the memory content while also encouraging the memory content to be stable (discouraging frequent updates).

Input to our encoder is a sequence of human poses, and our encoder transforms this to a deep representation by hierarchically encoding it at multiple scales. In this subsection, we explain this hierarchical encoding process.

Formally, let $X_{1:T_{obs}}=[x_{1},x_{2},\ldots,x_{T_{obs}}]$ consist of $T_{obs}$ consecutive human pose observations, where $x_{i}\in\mathbb{R}^{K}$ and $K$ is the data dimension that we utilise to describe each pose observation. $x_{i}\in\mathbb{R}^{K}\in\mathbb{R}^{J\times D}$ represents a single human pose which is composed of $J$ joints, and each joint is represented in $D$-dimensional space. In the datasets used in this work pose is observed in 3 dimensions, thus $D=3$. Our objective is to anticipate future poses for the duration $T_{obs}$ + 1 to $T_{obs}+T$. We denote the predicted pose sequence as $\hat{Y}_{T_{obs+1}:T_{obs}+T}=[\hat{y}_{1},\hat{y}_{2},\ldots,\hat{y}_{T}]$, and the ground truth observation as ${Y}_{T_{obs+1}:T_{obs}+T}=[{y}_{1},{y}_{2},\ldots,{y}_{T}]$.

Inspired by the recent success of graph neural networks to model the spatial structure of the poses [4, 3] and the proven ability of the graph convolution operation to retrieve spatial and structural dependencies between human joints, we utilise a Graph Convolution Network (GCN) as our backbone to extract features from the input pose sequence. Following [4, 3] we first replicate the last pose observation, $x_{T_{obs}}$, $T$ times making the input sequence of length $T_{obs}+T$. Similar to [4, 3] we represent pose as a fully connected graph with $K$ nodes and the adjacency matrix, $A\in\mathbb{R}^{K\times K}$, which is learned during the training process represents the strength of the dependencies between pairs of joints.

Our GCN-based pose encoder is composed of $L$ graph stacked convolution layers. At each level, $l\in[1,2,\ldots,L]$, the output, $H^{l+1}$, can be defined as,

where $F_{GCN}$ is the activation function and $W^{l}\in\mathbb{R}^{F^{l}\times F^{l+1}}$ are the trainable parameters of the graph convolution layer. $H^{l+1}\in\mathbb{R}^{K\times F^{l+1}}$ is the output which is passed to the next graph convolution layer. $F^{l}$ denotes the embedding dimension of the layer $l$.

Motivated by the success of [4, 3] in leveraging multi-scale GCNs for capturing the hierarchical spatial and structural relationships of human pose using multi-scale representations, we also employ a series of GCNs to abstract the human pose. Our descending GCN blocks downsample the resolution of the pose sequence by pooling adjacent joints. For instance, if the input pose is represented with 22 joints (i.e. $J=22$) in three dimensions (i.e. $D=3$), then the input feature space to our first downsampling block, $DN_{0}$ is of shape $K_{0}\times F$ with $K_{0}=22\times 3=66$. This input is down-sampled and the input to the second downsampling block, $DN_{1}$, is of shape $K_{1}=12\times 3=36$. Similarly, the $\mathrm{3^{rd}}$ and $\mathrm{4^{th}}$ downsampling blocks, $DN_{2},DN_{3}$, set $K_{2}=7\times 3=21$ and $K_{3}=4\times 3=12$, respectively. Note that for all the downsampling blocks we use the same embedding dimension $F$. We note that this encoding process is identical to the encoder of [4].

This subsection illustrates how we disentangle the embeddings generated by the encoder introduced in Sec. III-A. Specifically, we leverage masking operations to disentangle subject-specific, task-specific, and other auxiliary features from the output of our encoder. Formally, let $z_{t}^{DN_{3}}\in\mathbb{R}^{K_{3}\times F}$ denote the output of the final downsampling block, $DN_{3}$, for the observed pose input at time instance $t$. Then,

can be used to split the respective features across subject, task, and auxiliary segments. Here $\bigotimes$ denotes element-wise multiplication by each mask, $m_{i}\in\mathbb{R}^{K_{3}\times F}$. It should be noted that the masking operation only occurs in the embedding dimension, $F$, and we retain all the elements in dimension $K_{3}$. Details regarding the mask generation process are presented in Sec. III-C.

To ensure that the features are properly disentangled, and the segregated embeddings capture the intended attributes for the specified segments (i.e. subject, task, etc.) and only for that specified segment, we embed additional classification objectives alongside the primary future pose sequence prediction task.

Specifically, each of the supplementary classification heads first concatenates the relevant embeddings for the entire observed sequence such that,

where $\bigoplus$ denotes column-wise concatenation. The resultant feature vectors, $Z^{sub}$ and $Z^{task}$, are passed through a global average pooling operation, $f_{GAP}$, which transforms the input feature vectors to $\dot{z}^{sub}$ and $\dot{z}^{task}$ respectively, where $\dot{z}^{sub}\in\mathbb{R}^{K_{3}\times F}$ and $\dot{z}^{task}\in\mathbb{R}^{K_{3}\times F}$. Then we pass the resultant embeddings, $\dot{z}^{sub}$ and $\dot{z}^{task}$, through the respective classifiers such that,

where $f_{SUB}$ and $f_{TASK}$ denote subject and task classification sub-networks, respectively. Note that the evaluation protocol for skeleton-based motion synthesis in most of the popular benchmarks is leave-one-subject-out. Therefore, direct classification of the subject identity is not appropriate as the model is observing completely unseen subjects during the testing phase. To alleviate this issue we construct the subject classification as a contrastive learning task where two arbitrarily sampled pose sequences, which could be of the same subject or different subjects, are presented to the model and using the two $\dot{z}^{sub}$ embeddings extracted for the two inputs the $f_{SUB}$ network classifies whether they belong to the same subject or not. We use contrastive loss [31] to optimise this task. This contrastive learning helps the model to identify salient characteristics embedded within the input pose sequences that are subject-specific which will, in turn, help the motion prediction. Furthermore, we can use this same pipeline for both the training and testing phases. For task classification, noting that we have the same set of tasks in the training and testing sets, we use categorical cross-entropy loss.

The next task is to generate the masks such that subject, task, and auxiliary features can be disentangled. A naive way to generate the masks is to use a fixed mask such that a certain fixed region (of length $l_{sub}$) is completely dedicated to a specific subset of features, and the rest of the elements in $z_{t}^{DN_{3}}$ are completely masked out. For instance, if we select the first $l_{sub}$ elements for subject-specific features, the next $l_{task}$ for task-specific features, and the rest of the elements to carry auxiliary information, then the 3 masks can be visualised as in Fig. 2.

We see several drawbacks to this approach. First, it assumes that there exist hard/rigid boundaries between subject, task, and auxiliary segments in the feature vector and those elements are either fully active or not; i.e. it does not allow partial activation of the mask. Second, it makes the feature disentanglement process static after the model training process, and doesn’t allow the feature selection to dynamically change based on the embedded information. To resolve these drawbacks we propose to dynamically generate masks based on the encoded information in $z_{t}^{DN_{3}}$.

Specifically, we utilise a set of hard-coded masks (i.e. $\dot{m}^{sub},\dot{m}^{task}$ and $\dot{m}^{aux}$) based on the proportion in the vector $z_{t}^{DN_{3}}$ that we desire the subject, task and auxiliary features to occupy. Then, using a neural network that is parameterised by the function $f_{MASK}$ we generate residuals to augment the fixed masks,$\dot{m}^{sub},\dot{m}^{task}$ and $\dot{m}^{aux}$. Fig. 3 visually illustrates this mask generation process.

Formally, $f_{MASK}$ outputs 3 mask vectors such that,

where $\tilde{m}^{sub}_{t},\tilde{m}^{task}_{t}$ and $\tilde{m}^{aux}_{t}$ each $\in\mathbb{R}^{K_{3}\times F}$ and represent the residual masks for the subject, task and auxiliary regions respectively. Then, the augmented mask can be generated using

This can be seen as injecting plasticity [24, 32] into a fixed set of hard-coded masks. Different initialisation functions can be used to initialise the fixed masks and for simplicity we used a uniform function such that all the elements within the particular segment (i.e. $l_{sub},l_{task},l_{aux}$) are initialised to a value of $0.5$ and $l_{sub}=l_{task}=l_{aux}=F/3$. Furthermore, the generated masks are normalised using the Gumbell softmax operation which is a differentiable relaxed one-hot vector-like operation. The temperature parameter, $\tau$, within the Gumbell softmax controls the sparsity of the resultant mask, with lower temperature values increasing output sparsity. We experimentally choose the value for $\tau$ which allows us to balance the fixed and plastic potions of the generated masks.

We compare this mask generation process with the splitting network proposed in our prior work [33]. In [33] we generated the entire mask while in the proposed work we generate only the residuals for the fixed mask. We believe predicting the entire mask using Gumbell softmax operation is too restrictive as it can lead to most values in the predicted mask being zero. We experimentally compare the proposed architecture and the masking process of [33] where we demonstrate this drawback. We refer readers to the ablation evaluations in Sec. IV-D details. Furthermore, we note that to the best of our knowledge, this is the first work to embed plastic/dynamic mask generation methods within stateful, continual learning neural networks.

The main goal of our feature factorisation process in Sec. III-B is to effectively utilise this to augment the knowledge retrieval process in the auxiliary memory. As illustrated in Sec. II-B, a typical memory architecture is composed of a memory stack with $s$ slots, and $M_{\lambda-1}$ denotes the state of the memory at time $\lambda-1$. Note that this time is measured with respect to the execution of the memory module (i.e. training/testing iteration) and not with respect to the time in input sequences (i.e. $t$).

In a traditional memory retrieval operation we first pass the entire input to the memory, $z_{t}^{DN_{3}}$, through a query function, $f_{QUERY}$, to generate a query vector, $q^{DN_{3}}_{\lambda}$, to query the memory such that

Then we retrieve memory slots that contain information related to our query using

and the memory output, $\mu_{\lambda}$, at time instance $\lambda$ is computed using,

However, we observe several limitations of the direct application of a single-head access scheme for memory knowledge retrieval in motion synthesis. In particular, when there exist numerous task-specific and subject-specific variations embedded in the same feature vector this could negatively impact the softmax score-based identification of relevant memory slots. For instance, if the memory contains information related to the same subject in the query but the tasks are different, there is a possibility that such memory slots will not be identified as relevant to the current query due to the mismatch of the task-specific features. Similarly, information for similar tasks but with different subjects may be ignored as the query operation is considering the entire query vector. This issue can be overcome using the proposed feature factorisation strategy where we can generate multiple queries using the individual factorised attributes. Specifically, we use $z^{sub}_{t}$ and $z^{task}_{t}$ in addition to $Z^{DN_{3}}_{t}$ as inputs to the memory module, and generate two additional query vectors such that,

which are leveraged to retrieve memory slots using

where $e^{s}$ is a matrix of ones, and $\odot$ denotes the outer product which duplicates its left vector $s$ times to form a matrix. Then the knowledge retrieved from memory can be defined by,

Then, we aggregate the information retrieved using the overall, subject-specific, and task-specific queries and concatenate them across the final dimension,

where $\dot{\mu}_{\lambda}\in\mathbb{R}^{K_{3}\times 3F}$. We also concatenate $\dot{\mu}_{\lambda}$ output with $z_{t}^{DN_{3}}$ such that $\tilde{z_{t}}=[z_{t}^{DN_{3}}\bigoplus\dot{\mu}_{\lambda}]$ and $\tilde{z_{t}}\in\mathbb{R}^{K_{3}\times 4F}$. We apply average pooling across the final dimension of $\tilde{z_{t}}$ and the resultant feature vector, $\ddot{z_{t}}\in\mathbb{R}^{K_{3}\times F}$. This augmented feature vector is then used to predict future motion. Fig. 4 visually illustrates the subject-specific feature retrieval process using the proposed multi-head retrieval scheme. Note that an identical process is leveraged to retrieve task-specific features from the auxiliary memory.

The architecture of our decoder module which predicts future motion is presented in Sec. III-F, and in Sec.III-E we outline the process of memory update which allows our model to perform continual learning.

Once informative content is retrieved from the auxiliary memory the next task is to update the memory module to ensure the continual evolution of the knowledge embedded in our memory. For this task the typical process is to leverage the memory output, $\dot{\mu}_{\lambda}$, and pass it through a non-linear function, $f_{WRITE}$ to generate a vector to update the memory such that,

where $\breve{\mu}_{\lambda}\in\mathbb{R}^{K_{3}\times F}$. Then the subsequent state of the memory can be generated using,

where $I$ is a matrix of ones, $e^{s}\in\mathbb{R}^{s}$, $e^{F}\in\mathbb{R}^{F}$ are vectors of ones, $\odot$ denotes the outer product which duplicates its left vector $s$ or $F$ times to form a matrix and,

However, our prior investigations [34, 23, 24] revealed that reliance on the downstream (i.e. classification, regression) objective alone could lead to sub-optimal memory states. For instance, imbalanced or poorly curated datasets could lead the memory to memorize the most frequently observed types of data, ignoring the less frequent classes as these contribute less to the overall loss. Moreover, if the diversity of the dataset is high, or if the embedding size or the number of memory slots is small, this could lead to frequent memory updates making the knowledge stored in the memory highly volatile and unstable making the discovery of long-term dependencies infeasible. To this end, we propose two novel architectural innovations. First, we incorporate an additional neural network-based predictor parameterized by the function $f_{\beta}$ which predicts the memory slots that need updating where,

Second, we propose two loss functions that directly operate on the memory content and encourage diversity and stability. Specifically, our diversity loss, $L_{div}$, will iteratively estimate the similarity between a slot’s content and the rest of the content in memory. Formally, our diversity loss can be defined as,

where $f_{COS}$ is the cosine similarity loss and $M_{\lambda-1}^{i}$ denotes the $i^{th}$ memory slot in the memory.

Next, we introduce $L_{cons}$, which helps consolidate memory content and penalises unstable memory updates. Formally, let $\breve{\beta}^{i}_{\lambda}$ denote the $i^{th}$ element (i.e. $i^{th}$ memory slot) in the output of Eq. 17 at time instance $\lambda$, let $w$ define the window size, then we generate $\frac{\lambda}{w}$ windows to inspect the change in memory updates for the period from time instance 1 to $\lambda$. The change in memory updates between two consecutive time intervals can be evaluated as,

where,

Now we define the memory consolidation loss, $L_{cons}$, as,

This formulation of the loss penalises large memory updates that occur between large temporal intervals. Using the windowing operation in Eq. 19, we obtain the average amount of the change of the slot content within the window and using Eq. 21, and penalise large changes (due to the log scale) that occur once the content has been initially written (due to the exponential scale of $c$). Using the window size we can control the period that we allow the slot content to stabilise and after which we exponentially penalise the content updates. As such, this loss does not negatively impact the continual learning ability of the memory.

We highlight the use of $\breve{\beta}_{\lambda}$ for the loss calculations instead of using $M_{\lambda-1}$ directly. Direct use of the memory content would require the individual memory states at all time instances to be stored, leading to inefficiencies. However, $\breve{\beta}_{\lambda}$ provides a snapshot of how individual memory slots have been updated, and as such, we utilise this vector.

Mirroring the downsampling blocks in our encoder, our decoder is equipped with upsampling GCN modules that gradually increase the resolution of the predicted pose representations. Specifically, the first upsampling block, $AN_{3}$, receives $\ddot{z_{t}}\in\mathbb{R}^{K_{3}\times F}$ and decodes it to a feature vector $z_{t}^{AN_{2}}\in\mathbb{R}^{K_{2}\times F}$. Similar to our encoder, our decoder has four upsampling blocks, $AN_{3},AN_{2},AN_{1}$, and $AN_{0}$. As in [4] four end-GCNs, $E_{0},E_{1},E_{2}$, and $E_{3}$, each with two graph convolution layers, are used to generate future pose sequences at four different scales. However, in contrast to [4] where respective downsampled (i.e. $z_{t}^{DN_{0}},z_{t}^{DN_{1}},z_{t}^{DN_{2}},z_{t}^{DN_{3}}$) and upsampled (i.e. $z_{t}^{AN_{0}},z_{t}^{AN_{1}},z_{t}^{AN_{2}},z_{t}^{AN_{3}}$) feature vectors in individual scales are concatenated and passed to the respective end GCNs, we only pass the upsampled features.

We use $L_{2}$ loss between the ground truth and predicted pose sequences to train the pose prediction head of our framework. We denote this loss as $L_{pose}$.

The overall loss function of our framework then becomes

where $L_{sub}$ is the contrastive loss for the subject identification task and $L_{task}$ is the categorical cross-entropy loss for task identification. $\theta_{pose},\theta_{div},\theta_{cons},\theta_{sub}$ and $\theta_{task}$ are loss weights that control the contributions from individual losses.

Implementation of this framework is completed using PyTorch. The Adam [35] optimiser with an initial learning rate of $2e^{-4}$ is used for optimisation. The learning rate is decreased by $0.98$ every two epochs. The model is trained for 100 epochs on an NVIDIA A100 GPU. The embedding size, $F$, was experimentally chosen and was set to 300. Similarly, hyper-parameters, $w,\theta_{pose},\theta_{div},\theta_{cons},\theta_{sub},\theta_{task}$ were experimentally chosen and were set to $15$, $0.4$, $0.15$, $0.15$, $0.15$, and $0.15$ respectively.

For our evaluations, following the state-of-the-art methods we use two popular motion capture benchmark datasets, namely, the Human3.6M (H3.6M) and CMU Motion Capture (CMU-Mocap) datasets. Details of these datasets are provided in the following subsections.

Mean Per Joint Position Error (MPJPE) has been widely used as the evaluation metric in numerous recent works [4, 12, 15] due to its ability to directly compare different frameworks, its intuitive nature, and the ability to evaluate it directly on skeleton kinematics. Therefore, as our evaluation metric we report MPJPE in millimeters, calculated using

where $\hat{p}_{j,t}\in\mathbb{R}^{3}$ is the predicted position of $j^{th}$ joint in $t^{th}$ frame while $p_{j,t}$ is the corresponding ground truth. Lower error values indicate better agreement between the predictions and ground truths.

As in [3, 4] we generate predictions for different short-term and long-term prediction horizons. Specifically, pose sequence predictions of 80 ms, 160 ms, 320 ms, and 400 ms (i.e. 10 frame sequence) were generated as short-term predictions, while 560ms and 1000ms (i.e. 25 frames) length sequences were generated for long-term predictions.

As baseline methods, we use state-of-the-art methods including Residual Sup [36], Traj-GCN [2], DMGNN [3], MSR-GCN [4], S-DGCN [12], PK-GCN [14], DANet [37], DPnet [15] and GA-MIN [13]. When choosing our baselines we ensured that a variety of different deep learning approaches, including recurrent neural networks, attention-based methods, graph neural networks, hybrid approaches, multi-scale and multi-stage GCN architecture, and geometric-inspired neural network architectures, are compared, enabling a comprehensive comparison.

Quantitative comparisons for short-term and long-term prediction results for the H3.6M dataset are presented in Tabs. I and II, respectively. Following [14, 3, 11, 4] for long-term prediction we report average error metrics and evaluations only for five popular action categories for ease of presentation. GCN-based approaches such as MSR-GCN [4], S-DGCN [12], PK-GCN [14] have been able to achieve comparatively higher performance compared to the RNN-based Residual sup method. However, demonstrating the feature contamination issue when aggregating high-dimensional features using multi-stage graph convolution operation, they struggle to reach the robustness level of GA-MIN [13]. However, when comparing FMS-AM with GA-MIN, which is the previous state-of-the-art method we observe that our method has the lowest MPJPE on average for both short-term and long-term predictions. Specifically, we observe that the GA-MIN method struggles to generate accurate pose predictions over 400 ms on the H3.6M dataset. Large errors in the long-term prediction setting (i.e. Tab. II, 1000ms setting) for classes such as directions (i.e MPJPE $\sim$ 100) and discussion (i.e MPJPE $>$ 106) were observed where there is subject specificity compared to more generalised actions such as walking where we observe comparatively lower MPJPE (i.e MPJPE $\sim$ 43). In contrast, our FMS-AM framework has been able to achieve consistent performance across all the action classes irrespective of subject-specific or class-specific motions. The average results of MPJPEs from Tabs I and II show that FMS-AM on average lowers the short-term prediction error (at 400ms) by 17.40% and the long-term prediction error by 17.54%, both significant margins. We believe the proposed feature factorisation strategy coupled with the innovative multi-head retrieval of the auxiliary memory has allowed superior learning capabilities where subject-specific, class-specific, and auxiliary information are better incorporated into the prediction of future motion sequences.

Similar, consistent performance of the FMS-AM framework is observed on the CMU-Mocap dataset for both short-term and long-term prediction tasks which are presented in Tabs. III and IV, respectively. Specifically, the previous state-of-the-art method, GA-MIN [13], struggles in short-term prediction of action classes such as jumping and soccer where we observe significant errors $>$ 90% and 49%, respectively at the 400 ms setting. Furthermore, for long-term predictions at the 1000 ms setting GA-MIN’s average error is $\sim$85%. In contrast, we observe a significant reduction in prediction error across both short-term and long-term prediction settings in our FMS-AM model where on average it lowers the short-term prediction error by 9.3% at the 320ms setting, and the long-term prediction at the 1000ms setting is reduced by 32 %. These results clearly indicate the utility of the proposed innovations.

In addition to quantitative comparisons with existing state-of-the-art methods in Fig. 5 we present a comparison between GA-MIN [13] and the proposed FMS-AM method for predicting long-term motion patterns in the CMU-Mocap dataset for the running and soccer action classes. When comparing the long-term predictions of the two models, in particular beyond 400ms, we observe that GA-MIN struggles to generate accurate estimations of the future motion while the proposed method achieves significant robustness. Furthermore, we observe that the predictions generated by the proposed FMS-AM for complex action classes such as soccer, where there are unique subject-specific and task-specific motions, are far superior to GA-MIN. Please refer to supplementary material for additional qualitative results.

We conducted a series of ablation studies to systematically analyse the impact of the individual innovations that our FMS-AM framework proposes. Several design choices contribute to the robustness of our model: i) the proposed feature factorisation strategy; ii) the multi-head knowledge retrieval scheme; iii) the proposed auxiliary memory stabilisation losses; and iv) the dynamic mask generation procedure. All of these experiments were conducted on the H3.6M dataset and for testing the ablation models we use the validation set of H3.6M.

Our FMS-AM method contains 30.82M trainable parameters which is not a substantial increase of trainable parameters when compared with the state-of-the-art MGCN model [11] which has 26.51M trainable parameters when considering the significant performance increase that our FMS-AM method achieves. FMS-AM generates predictions (each of which has a prediction length of 1000 ms) for 100 input pose sequences in 5.85675 sec using a single NVIDIA A100 GPU. Note that this includes time taken for both feature extraction and the generation of model predictions.