Towards Precise 3D Human Pose Estimation with Multi-Perspective Spatial-Temporal Relational Transformers

Before the widespread application of transformers in various image tasks, most 2D human keypoint detection methods relied on Convolutional Neural Networks (CNNs) as the main backbone. These 2D human keypoint detection approaches are categorized into multi-person and single-person detection based on image content. Multi-person human keypoint detection is further divided into top-down [19, 20, 21, 22] and bottom-up [3, 4] approaches. In the bottom-up approach, all keypoints of all individuals are detected first, followed by using clustering algorithms to classify keypoints belonging to the same person, resulting in the final output. On the other hand, the top-down approach involves using object detection algorithms [23, 24] to obtain bounding boxes for each person and then performing single-person keypoint detection based on these bounding boxes. Among these methods, the top-down approach consistently achieves state-of-the-art results on benchmark datasets, leading to the choice of employing the top-down approach in this work.

While CNNs and their variants remain the primary backbone architecture for computer vision applications, the modeling capabilities of transformers [25, 26] for various information relationships are unparalleled by CNNs. In this context, we aim to leverage the transformer architecture to extract rich feature information from 3D human pose datasets. Despite the continued prominence of CNN models, the unique ability of transformers to harvest complex relationships motivates our exploration of their potential in capturing intricate features within 3D human pose data.

3D human pose recognition can be broadly categorized into direct methods [5] and two-stage methods [6, 7, 27]. Direct methods aim to extract raw feature information directly from images. For instance, C2F-Vol [5] draws inspiration from the Hourglass network structure employed in 2D Human Pose Estimation (HPE) and represents 3D pose in the form of 3D heatmaps. Two-stage methods, on the other hand, simplify the prediction process. The baseline approach utilizes a relatively simple feedforward neural network for estimating 3D pose, taking 2D keypoint coordinates as input and directly mapping 2D pose to 3D space through fully connected layers with residual connections. Hossain and Little [14] introduce a Recurrent Neural Network (RNN) with Long Short-Term Memory (LSTM) units to leverage temporal information within input sequences. Pavllo et al. [15] incorporate a temporal convolutional network to estimate 3D pose generated from continuous 2D sequences of keypoints. In [6, 28], efficient human pose estimation has been explored by using event point cloud data. However, these networks [14, 15] often struggle to model information relationships between frames in multi-view video sequences effectively. To address this limitation, we employ a transformer structure to extract information from multi-view video datasets. Leveraging rich feature information, this approach yields accurate 2D human pose estimations. Finally, a two-stage method is employed to achieve 3D human pose recognition.

After the initial introduction of Transformers for image tasks, numerous variant architectures suitable for various image-related tasks have been proposed. DETR [23], applied in object detection, achieved an end-to-end framework for object detection. ViT [26] successfully employed a pure Transformer structure for image classification, demonstrating excellent performance. Given the substantial computational resources required for self-attention calculations on images, several models focusing on reducing computational complexity have been introduced. The Swin transformer [29] divides images into small windows, performing self-attention computations within these windows. In [30], the presented TCFormer achieves sparse input by clustering and pruning the input tokens. DynamicViT [31] evaluates tokens generated for an image, discarding low-scoring background tokens to facilitate sparse input. Moreover, the transformer architecture has been introduced to various tasks like semantic segmentation [32, 33], content outpainting [34, 35] and pose estimation [2, 27, 36, 37]. In this work, our proposed network architecture leverages windowed self-attention to efficiently extract spatial information regarding human key points in the image. Furthermore, we introduce an image relations module to harvest temporal relationship features among frames in video sequences and 3D spatial position features for enhancing the estimation performance.

In this section, we introduce a multi-stage framework for 3D sequence-to-sequence (seq2seq) human pose detection, as illustrated in Fig. 2. Our architecture primarily comprises a spatial module and a frame-image relation module, both adeptly leveraging the self-attention mechanism [25]. The spatial module is utilized to extract human body pose features inherent in the images themselves, while the frame-image relation module extracts temporal relationships and 3D spatial positional relationship features between the images. Connecting these two components is a frame-image information aggregation module, responsible for aggregating all tokenized information containing image details into a sequence of tokens that represent the frames within a video. Finally, keypoint coordinates for all input frames are directly estimated through a regression head, in accordance with the standard seq2seq architecture.

The fundamental premise of our research centers on the self-attention mechanism [25], an overarching architecture adept at modeling dependencies within input sequences. Initially, the input images $I$ undergo processing through embedding [26] operations,

where $Z\in\mathbb{R}^{N\times C}$ represents the patch sequence, with $N$ being the length of the patch sequence and $C$ being the dimensionality of each patch. Subsequently, the self-attention mechanism transforms $Z\in\mathbb{R}^{N\times C}$ into three vectors, $Q$, $K$, and $V$, through three distinct linear transformations:

Among these, $Q$ can be understood as the information to be queried, $K$ represents the patches to be queried, and $V$ signifies the features of each patch. Following the derivation of these three vectors, the computation of self-attention can be expressed as:

where$\sqrt{d}$ serves as a normalization factor for the values of $Q{\cdot}K$, preventing numerical instability that may lead to gradient vanishing. The $Softmax(\cdot)$ function normalizes the computed results, generating a weight matrix that, when multiplied by the feature vectors $V$, yields the attention values. To capture intricate feature information, the self-attention mechanism employs a multi-head self-attention layer ($MSA$) for parallel processing of feature information. Each head simultaneously computes self-attention, and the outputs from $k$ heads are concatenated. This can be expressed as:

In 3D pose estimation datasets, rich feature information is present, and we aim to model the specific dependencies among these features. Therefore, we employ the self-attention mechanism as the core mechanism in our network.

As a network tailored for processing video sequences, we need to design a spatial feature extraction module capable of handling long input video frame sequences. Given the substantial volume of video frame data, we employ a moving window self-attention mechanism to extract spatial features, which helps alleviate the computational burden and reduce processing time. Specifically, we partition a single image into multiple small blocks using a small window, limiting self-attention computations to the interior of these blocks. Simultaneously, we incorporate window movement to capture global features, facilitating interaction with image information outside the current window. Subsequently, within the generated image blocks after movement, another round of self-attention computation is performed. Consequently, as illustrated in Fig. 3(a), a complete spatial transformer block involves two rounds of self-attention computation, which can be expressed as:

In this context, $\hat{z}^{l}$ and $z^{l}$ respectively denote the conventional window module of self-attention blocks and the output features of the $MLP$ module. Here, $W-MSA$ and $SW-MSA$ refer to utilizing conventional and sliding window partitions in window-based self-attention.

To enable the network to learn hierarchical features, a patch merging module performs downsampling on the images. This aids in reducing the length of the patch sequence, increasing the patch dimension, and enlarging the receptive field for extracting contextual cues. The downsampling operation generates images at different scales, facilitating the network in learning features from multiple scales.

Following the extraction of image spatial features using the spatial module, we consolidate the information of each image into a vector that encapsulates the spatial characteristics of the image. Subsequently, we concatenate the feature information of each frame’s image as tokens and input them into the standard transformer model.

When modeling the relationships between images, as illustrated in Fig. 3(b), we employ a standard transformer model [25], as a significant amount of spatial image feature information has already been consolidated into a single vector during the spatial module, facilitating the flexible computation of global self-attention. In a standard transformer model (assuming input of $32$ frames), if the input frame image size is $(32,224,224,3)$, after mapping, the size of tokens input to the transformer module is $(32,196,768)$. However, in our structure, after the spatial feature extraction, the size of tokens input to the transformer model is $(1,32,768)$. This significantly reduces computational complexity, enabling effortless calculation of global self-attention. The Image Relations Module is divided into two parts. The first part models the temporal relationships between video frames, utilizing self-attention mechanisms to learn global dependencies between video frames. The second part models the 3D spatial positional relationships between corresponding images. Since data collection involves the use of four cameras, a 3D pose corresponds to four 2D poses, and there exists a spatial positional relationship between these four 2D poses. We utilize self-attention mechanisms to learn this relationship. Additionally, sometimes, to fully leverage temporal information in video sequences, long sequences of video frames may be inputted, a scenario our model can readily accommodate.

Our Image Relations Module can be represented as follows:

where $MSA$ represents the multi-head self-attention module, $LN(\cdot)$ [38] represents the normalization layer, $Y$ is the output, and $Z$ represents the output of each layer of the transformer module, with a total of $L$ layers.

We employ positional embeddings [26] to represent the temporal dependencies and spatial relationships in the 3D space of the images:

where $z$ represents the feature information of each image, $E_{time}$ and $E_{3D}$ denote two types of relationships.

Human3.6M [10] is a large public dataset for 3D human pose estimation research, featuring $3.6$ million images with corresponding 3D human poses. The dataset includes $11$ professional actors ($6$ male, $5$ female) and spans $7$ scenes (such as discussions, smoking, photography, and phone calls). Comprising videos captured by $4$ calibrated high-resolution cameras at $50Hz$, the dataset’s labels are derived from precise 3D joint positions and angles obtained from a high-speed motion capture system. We utilize five subjects from the dataset for training and reserve two subjects for testing purposes.

We evaluate our approach on the Human3.6M [10] dataset. To comprehensively assess our method, the experiments are primarily divided into three categories. The first category involves evaluating the 2D detection results as our method aims to improve 3D results through enhanced 2D human pose detection. The second category assesses the application of 2D detection results to 3D detection for evaluating the 3D pose results. The third category evaluates the impact of different numbers of input frames on the detection results since our method models temporal relationships based on video frame sequences.

Our method is implemented using PyTorch [46]. All experiments were conducted on a single NVIDIA RTX 3090 GPU. We selected five subjects from the Human3.6M dataset for training and evaluated the model on two subjects, utilizing an input resolution of $224{\times}224$. The training process spanned $100$ epochs, employing the AdamW [47] optimizer, cosine decay learning rate scheduler, and linear warm-up for the initial $20$ epochs. The initial learning rate was set to $0.001$, with a weight decay of $0.05$. During training, three different lengths of sequence frames were chosen, i.e., $f{=}8,f{=}32,f{=}128$. Our spatial module was constructed using a fine-tuned pre-trained model based on the Swin transformer architecture [29], while the temporal module underwent fine-tuning using a pre-trained vision transformer [26].

We conducted a quantitative comparison using ViT-S [26], ViT-B [26], HRNet-32 [4], and HRNet-48 [4]. The model excluding the Image Relations Module is designated as the baseline model, and experiments were performed on this baseline to investigate the impact of the Image Relations Module on performance. Training was carried out on these networks using the Human3.6M dataset [10], and evaluation metrics such as AP (Average Precision), AR (Average Recall), PCK (Percentage of Correct Keypoints), and MSE (Mean Squared Error) were employed for the assessment of 2D human body pose. We computed the time taken for each model to train one epoch on the S1 subject as an indicator of the model’s inference speed.

From Table LABEL:table1, it is evident that our approach achieves the highest accuracy with minimal errors. In comparison to the baseline model, our overall model exhibits a higher accuracy, indicating the effectiveness of the Image Relations Module and its significant enhancement of overall performance. Despite not being the fastest in terms of model inference speed, our method demonstrates exceptional cost-effectiveness. It notably improves prediction accuracy without substantially increasing the inference time. Moreover, our model shows a marginal increase in inference time compared to the Baseline model, suggesting that the Image Relations Module does not introduce a significant computational overhead in terms of the inference duration.

The 2D human body pose estimation network proposed in this paper is designed to facilitate 3D human body pose estimation. To demonstrate the effectiveness of our network, we utilize the 2D human body pose estimation results as input for 3D human body pose estimation. PoseFormer [27] is employed as the 3D human body pose detection network, and the network is trained following the experimental settings of PoseFormer [27].

We use MPJPE (Mean Per Joint Position Error) [48] as the evaluation metric, which represents the Euclidean distance between predicted and ground truth joint positions. Evaluation is conducted on the test set (S9, S11) for $15$ actions. As shown in Table LABEL:table2, inferences are made for the $15$ actions of two test subjects (S9, S11), and the last column represents the average error for each action.

As shown in Table LABEL:table2, we utilized the 2D poses generated by our method as input, employing PoseFormer [27] as the 3D pose network. We conducted a quantitative evaluation of the average error for each joint position using MPJPE and P-MPJPE as evaluation criteria. We drew insights from relevant experimental results in PoseFormer [27]. In the table, red indicates the best performance, and blue indicates the second-best. Inference was performed for $15$ actions on two test subjects (S9, S11), with the last column representing the average error for each action. From Table LABEL:table2, our method exhibits leadership in both evaluation metrics. Compared to the initial PoseFormer [27], our method reduces the average error by $9\%$, decreasing from $44.3$ to $40.3$. In terms of the MPJPE metric, our method achieves a leading position in almost all $15$ actions, except for the $Sit$ action. However, in the P-MPJPE [48] metric, our method achieves the optimal result for all actions, demonstrating an enhancement in 3D human body pose estimation effectiveness.

As shown in Fig 4, our method detects all images in the Human3.6M dataset and outputs 2D human body poses for all images. Many complex actions are accurately detected, and from Fig 4, we can see that many occluded human keypoints are accurately predicted, thanks to the picture relation module in our network, which can predict these occluded human keypoints based on temporal relations and 3D spatial location relations. We input all the predicted 2D human body poses into the 3D human body detection network Poseformer to reconstruct the 3D human body poses.

Most 3D pose detection is based on video datasets [9, 10, 11], so we want the network to be able to process information from an entire video and learn the relationships between all video frames. We set up three different lengths of input frame sequences, $f{=}8,f{=}32,f{=}128$ to evaluate the effect of frame sequence length on the network. In Table LABEL:table3, we show the effect of the number of frames on the experiment. As the number of frames increases, the accuracy improves, indicating that the longer the input video sequence, the easier it is to capture the relationship between the frame images, resulting in better overall performance. This result proves that our network is able to model the relationships between long sequences of inputs, extract the temporal relationships in them, and achieve better output.