MotionAGFormer: Enhancing 3D Human Pose Estimation with a Transformer-GCNFormer Network

We begin this section by reviewing the concept of MetaFormer [46], which forms the core of our encoders. A MetaFormer can be described as a generalization of the Transformer architecture [41], wherein the attention module is substituted with any mixer capable of transforming information among tokens. Specifically, for an input $X\in\mathbb{R}^{N\times C}$, with $N$ denoting the token numbers and $C$ representing the embedding dimension, the token mixer can be formally expressed as

$$Y=\mathrm{TokenMixer}(\mathrm{Norm}(X))+X,$$

(1)

where $\mathrm{Norm}(\cdot)$ denotes a normalization method such as batch or layer normalization [13, 1], and $\mathrm{TokenMixer}(\cdot)$ denotes a module that combines information among tokens. Our approach uses two parallel token mixers: Multi-head Self-Attention (MHSA) and Graph Convolutional Networks (GCNs) [17], each contributing uniquely to the information transformation process.

Our objective is to lift a 2D (pixel coordinate) skeleton sequence to accurate 3D pose sequences. To this end, we propose the MotionAGFormer architecture, which uses both attention (Transformer) and graph convolutional (GCNFormer) to lift motion sequences. An overview of this architecture is shown in Figure 2a.

The model takes a 2D input sequence with confidence score $\mathbf{X}\in\mathbb{R}^{T\times J\times 3}$, where $T$ and $J$ refer to the number of frames and joint numbers, respectively. It then proceeds to map each joint in each time frame to a $d$-dimensional feature, $\mathbf{F}^{(0)}\in\mathbb{R}^{T\times J\times d}$, using a linear projection layer. Then a spatial position embedding $\mathbf{P_{pos}^{s}}\in\mathbb{R}^{1\times J\times d}$ is added to the tokens. It is important to highlight here that our model does not disregard the temporal token order as that information is preserved in the GCNFormer stream (further discussion in ablation studies 4.5).

Subsequent to position embedding, we use $N$ blocks of AGFormer (Section 3.3) to compute $\textbf{F}^{(i)}\in\mathbb{R}^{T\times J\times d}$ ($i=1,...,N$) to effectively capture the underlying 3D structure of the skeleton sequence. Finally, we map $\mathbf{F}^{(N)}$ to a higher dimension by applying a linear layer and $tanh$ activation to compute motion semantic $\textbf{M}\in\mathbb{R}^{T\times J\times d^{\prime}}$ and use a regression head to estimate 3D pose $\mathbf{\hat{P}}\in\mathbb{R}^{T\times J\times 3}$. The lifting loss contains position ($L_{3D}$) and velocity ($L_{\Delta\mathbf{P}}$) terms defined as

	$\displaystyle L_{3D}$	$\displaystyle=\Sigma_{t=1}^{T}\Sigma_{j=1}^{J}\|\mathbf{\hat{P}}_{t,j}-\mathbf{P}_{t,j}\|,$		(2)
	$\displaystyle L_{\Delta\mathbf{P}}$	$\displaystyle=\Sigma_{t=2}^{T}\Sigma_{j=1}^{J}\|\Delta\mathbf{\hat{P}}_{t,j}-\Delta\mathbf{P}_{t,j}\|,$

where $\Delta\mathbf{\hat{P}}_{t}=\mathbf{\hat{P}}_{t}-\mathbf{\hat{P}}_{t-1}$, $\Delta\mathbf{P}_{t}=\mathbf{P}_{t}-\mathbf{P}_{t-1}$. The total lifting loss is then defined as

$$L=L_{3D}+\lambda_{\Delta\mathbf{P}}L_{\Delta\mathbf{P}},$$

(3)

where the constant coefficient $\lambda_{\Delta\mathbf{P}}$ is used to balance position accuracy and motion smoothness.

The AGFormer block uses a dual-stream architecture. Each stream consists of two components: a Spatial MetaFormer (Figure 2b) followed by a Temporal MetaFormer (Figure 2c). The Spatial MetaFormer processes individual body joints as distinct tokens, effectively capturing intra-frame relationships within a single frame. The Temporal MetaFormer, on the other hand, treats each frame as a single token, thus capturing inter-frame relationships over time. The key distinction between the two streams lies in their token mixer type. While one stream employs Transformers, the other stream uses GCNFormers.

Transformer stream. This stream employs a Spatial Multi-Head Self-Attention (S-MHSA) to capture spatial relationships, followed by a Temporal Multi-Head Self-Attention (T-MHSA) to capture temporal relationships. The S-MHSA is defined as

$\begin{split}\text{S-MHSA}(\mathbf{Q_{s}},\mathbf{K_{s}},\mathbf{V_{s}})=\mathrm{Concat(head_{i},...,head_{h})}\mathbf{W_{s}}^{(O)},\\ \mathrm{head_{i}}=\mathrm{softmax}(\frac{\mathbf{Q_{s}}^{(i)}(\mathbf{K_{s}}^{(i)})^{T}}{\sqrt{d_{k}}})\mathbf{V_{s}}^{(i)},\end{split}$

(4)

where $\mathbf{W_{s}}^{(O)}$ is a projection parameter matrix, $h$ is the number of parallel attention heads, and $d_{k}$ is the feature dimension of $\mathbf{K_{s}}$. For computing the query matrix $\mathbf{Q_{s}}$, the key matrix $\mathbf{K_{s}}$, and the value matrix $\mathbf{V_{s}}$, we have

$\mathbf{Q_{s}}^{i}=\mathbf{F_{s}W_{s}}^{(Q,i)},\mathbf{K_{s}}^{i}=\mathbf{F_{s}W_{s}}^{(K,i)},\mathbf{V_{s}}^{i}=\mathbf{F_{s}W_{s}}^{(V,i)},$

(5)

where $\mathbf{F_{s}}\in\mathbb{R}^{BT\times J\times d}$ is spatial feature and $\mathbf{W_{s}}^{(Q,i)}$, $\mathbf{W_{s}}^{(K,i)}$, $\mathbf{W_{s}}^{(V,i)}$ are projection matrices and $B$ is the batch size. The S-MHSA result is subsequently fed into a multilayer perceptron (MLP), followed by a residual connection and LayerNorm. This completes the first MetaFormer, i.e. the Spatial Transformer.

Next, we reshape $\mathbf{F_{s}}$ into $\mathbf{F_{T}}\in\mathbb{R}^{BJ\times T\times d}$ to prepare per-joint temporal feature as the input of T-MHSA. Here we have

$\begin{split}\text{T-MHSA}(\mathbf{Q_{T}},\mathbf{K_{T}},\mathbf{V_{T}})=\mathrm{Concat(head_{i},...,head_{h})}\mathbf{W_{T}}^{(O)},\\ \mathrm{head_{i}}=\mathrm{softmax}(\frac{\mathbf{Q_{T}}^{(i)}(\mathbf{K_{T}}^{(i)})^{T}}{\sqrt{d_{k}}})\mathbf{V_{T}}^{(i)},\end{split}$

(6)

where $\mathbf{Q_{T}}$, $\mathbf{K_{T}}$, and $\mathbf{V_{T}}$ are calculated similar to Eqn. (5).

GCNFormer stream. Unlike the Transformer stream, which aggregates global information, the GCNFormer stream focuses on local spatial and temporal relationships present within the skeleton sequence. While the local information is also available to the Transformers, the inclusion of this parallel stream allows the model to more effectively balance the integration of local and global information (see ablation analysis 4.5). The customized GCN module [24] used in our GCNFormer is defined as:

$GCN(\mathbf{F^{(i)}})=\sigma(V^{l}+\mathrm{Norm}(\tilde{D}^{-\frac{1}{2}}\tilde{A}\tilde{D}^{-\frac{1}{2}}\mathbf{F^{(i)}}W_{1}+\mathbf{F^{(i)}}W_{2}).$

(7)

Where $\tilde{A}=A+I_{N}$ represents the adjacency matrix with self-connections added, $I_{N}$ stands for the identity matrix, $\tilde{D}{ii}=\Sigma{j}\tilde{A}{jj}$ is defined as the sum of elements along the diagonal of $\tilde{A}$, and $W_{1}$, $W_{2}$ denote trainable weight matrices specific to each layer. The activation function $\sigma(\cdot)$, such as ReLU, is applied, along with Batch Normalization [13]. The GCN’s output is then passed through an MLP, followed by residual connection and LayerNorm.

The difference between the Spatial GCNFormer and the Temporal GCNFormer lies in their adjacency matrices and input features. The input features resemble that of the Transformer stream. In the Spatial GCNFormer, the adjacency matrix represents the human topology (Figure 3a). For Temporal GCNFormer, on the other hand, we calculate the similarity between a single joint at different time frames using $Sim(\mathbf{F_{T}}^{t_{i}},\mathbf{F_{T}}^{t_{j}})=(\mathbf{F_{T}}^{t_{i}})^{T}\mathbf{F_{T}}^{t_{j}}$ and choose the $K$ nearest neighbors as the connected nodes in the graph (Figure 3b). Hence, the graph topology in Temporal GCNFormer is determined by the learned node features.

Adaptive Fusion. Similar to MotionBERT [53], we use adaptive fusion to aggregate extracted features of the Transformer and GCNFormer streams. This is defined as:

$$\leavevmode\resizebox{381.5877pt}{}{$\mathbf{F}^{(i)}={\alpha_{TF}}^{(i)}\circ\mathbf{F_{TF}}^{(i-1)}+{\alpha_{GF}}^{(i)}\circ\mathbf{F_{GF}}^{(i-1)}$},$$

(8)

where $\mathbf{F}^{(i)}$ represents the feature embedding extracted at depth $i$, the element-wise multiplication denoted by $\circ$, and $\mathbf{F_{TF}}^{(i-1)}$, $\mathbf{F_{GF}}^{(i-1)}$ refer to the extracted Transformer stream and GCNFormer stream features at depth $i-1$, respectively. The adaptive fusion weights $\alpha_{TF}$ and $\alpha_{GF}$ are defined as

$$\leavevmode\resizebox{381.5877pt}{}{${\alpha_{TF}}^{(i)},{\alpha_{GF}}^{(i)}=\mathrm{softmax}(W\cdot\mathrm{Concat}(\mathbf{F_{TF}}^{(i-1)},\mathbf{F_{GF}}^{(i-1)}))$},$$

(9)

where $W$ is a learnable linear transformation.

Human3.6M is a widely used indoor dataset for 3D human pose estimation. It contains 3.6 million video frames of 11 subjects performing 15 different daily activities. To ensure fair evaluation, we follow the standard approach and train the model using data from subjects 1, 5, 6, 7, and 8, and then test it on data from subjects 9 and 11. Following previous works [50, 40], we use two protocols for evaluation. The first protocol (referred to as P1) uses Mean Per Joint Position Error (MPJPE) in millimeters between the estimated pose and the actual pose, after aligning their root joints (sacrum). The second protocol (referred to as P2) measures Procrustes-MPJPE, where the actual pose and the estimated pose are aligned through a rigid transformation.

MPI-INF-3DHP is another large-scale dataset gathered in three different settings: green screen, non-green screen, and outdoor environments. Following previous works [50, 40], MPJPE, Percentage of Correct Keypoint (PC) within 150 mm range, and Area Under the Curve (AUC) are reported as evaluation metrics.

Model Variants. We build four different configurations of our model, as summarized in Table 1. Our base model, known as MotionAGFormer-B, strikes a balance between accurate estimation and computational cost. The remaining variants are named according to their parameter size and computational demands and selection of each variant can be based on an application’s requirements, such as choosing between real-time processing or more precise estimations. The motion semantic dimension is $d^{\prime}=512$, the expansion layer of each MLP is $\alpha=4$, the number of attention heads is $h=8$, and the number of temporal neighbours in GCNformer stream is $k=2$, for all experiments.

Experimental settings. Our model is implemented using PyTorch [30] and executed on a setup with two NVIDIA A40 GPUs. We apply horizontal flipping augmentation for both training and testing following [53, 50]. For model training, we set each mini-batch as 16 sequences. The network parameters are optimized using AdamW [23] optimizer over 90 epochs with a weight decay of 0.01. The initial learning rate is set to 5e-4 with an exponential learning rate decay schedule and the decay factor is 0.99. We use the Stacked Hourglass [29] 2D pose detection results and 2D ground truths on Human3.6M, following [53]. For MPI-INF-3DHP, ground truth 2D detection is used following a similar approach used in comparison baselines [50, 40].

We compare our MotionAGFormer with other models on Human3.6M. For a fair comparison, only results of models without extra pre-training on additional data is included. The results, outlined in Table 2, demonstrate that MotionAGFormer-L attains a P1 error of 38.4 mm for estimated 2D pose and 17.3 mm for ground truth 2D pose. Remarkably, this is achieved with approximately half the computational requirements and parameters compared to the previous SOTA (MotionBERT [53]), while being 0.8 mm and 0.5 mm more accurate, respectively. When comparing the Base and Large variants, MotionAGFormer-L shares the same P1 error as MotionAGFormer-B in the presence of noisy data, while exhibiting a 2.1 mm reduction in error when using ground truth 2D data. Furthermore, MotionAGFormer-S takes in only a third of the frames compared to the baselines, yet it manages to attain a superior P1 error compared to a majority of them. Similarly, MotionAGFormer-XS delivers comparable performance to PoseTransformerV2, even though it receives only a ninth of the input information (27 vs. 243 frames) and operates with approximately seven times fewer parameters.

In evaluating our method on the MPI-INF-3DHP dataset, we modified our base and large variants to use 81 frames due to shorter video sequences. Across all variants, our method consistently outperforms others in terms MPJPE. Notably, our large variant achieves remarkable results with an 85.3% AUC and a 16.2 mm P1 error. This outperforms the best models by a significant margin of 1.4% in AUC and 6.9 mm in P1 error. However, it achieves 98.2% PCK, which is 0.5% lower than the PCK performance of the compared models (Table 3).

A series of ablation studies were conducted on the Human3.6M dataset to explore different design choices for the AGFormer block.

The initial ablation study investigates the influence of favoring the number of AGFormer blocks and the width of our model on the P1 error. As shown in Table 4, the trend generally leans towards favoring a model that is deeper but narrower. Interestingly, a model with 16 AGFormer blocks and a width of 128 exhibits similar performance to a model featuring 12 AGFormer blocks with a width of 256, while the first configuration uses approximately three times less memory and computational resources.

The second part of the ablation study explores alternative modules for the graph stream within the AGFormer block. The results are presented in Table 5. Shifting from GCNFormer to GCN blocks degrades the P1 error by 0.7 mm. Similarly, substituting the Temporal GCNFormer with TCN degrades the P1 error by 0.2 mm when using GCN and 0.9 mm when using GCNFormer. Lastly, replacing the GCN with a CTR-GCN [5] increases the P1 error by 2.5 mm. We hypothesize that is because of the continuous refinement of topology for each channel in CTR-GCN during training. As a result, every joint not only obtains information from its immediate neighbors but also from all other joints. This prevents the generation of complementary information for the transformer stream.

To explore the impact of positional embedding on the the final performance, we conducted a series of experiments outlined in Table 6. Surprisingly, including temporal positional embedding in addition to spatial positional embedding leads to a 0.6 mm increase in P1 error. As mentioned in Section 3.2, this result stems from the non-permutation equivariant nature of the GCNFormer stream. Unlike transformers, our network inherently maintains the temporal sequence of frames. However, as shown in Figure 5, adding temporal positional embedding leads to a better acceleration error (0.88 mm) compared to using spatial positional embedding (0.94 mm).

Finally, to verify the efficiency of the proposed AGFormer block, Table 7 shows alternative blocks. When using GCNFormer, a P1 error of 57.5 mm is observed, indicating its limited capability to accurately capture the underlying 3D sequence structure. Nevertheless, a hybrid approach involving both GCNFormer and Transformer yields a noteworthy improvement, reducing the P1 error by 5.2 mm compared to using Transformer alone. Furthermore, the sequential fusion of these two modules is not as effective as their parallel integration.

Validation of MotionAGFormer-B is conducted using the adjacency matrix of temporal GCNFormer and visualization of 3D human pose estimation. The instances for validation are randomly chosen from the evaluation set of Human3.6M.

Qualitative Comparisons. Figure 4 compares MotionAGFormer-B with recent approaches including STCFormer [40], PoseFormerV2 [50], and MotionBERT [53]. By design, confidence scores of the 2D detector are only included into MotionBERT and MotionAGFormer-B. Overall, MotionAGFormer-B shows better reconstruction results across all three samples than PoseFormerV2 and STCFormer, while maintaining competitive performance with MotionBERT. Specifically, in Figure 4a, MotionAGFormer exhibits improved alignment with the ground truth in comparison to alternative approaches. In Figure 4b, it displays a slightly superior alignment, whereas in Figure 4c, its alignment is marginally less optimal than that of MotionBERT.

Temporal adjacency visualization. The adjacency matrix of temporal GCNFormer is visualized in Figure 6. In the initial layers, individual joints exhibit distinct adjacency matrices, which should vary across different sequences due to the changing joint positions over time. Nevertheless, as we progress through the model’s depth, it appears to learn a representation where each joint is most akin to its own state in neighboring frames. Consequently, this leads to connections being formed with adjacent frames.