HSTFormer: Hierarchical Spatial-Temporal Transformers
for 3D Human Pose Estimation

3D human pose estimation (HPE) aims to predict the coordinates of human body joints in 3D space from images and videos. Hence, it plays important roles in numerous applications, such as human activity recognition [17], healthcare [11], augmented reality (AR) [8], virtual reality (VR) [21] and human-robot interaction [22]. Direct inferring 3D pose from a single image is a challenging ill-posed problem because part of the 3D information is lost in 3D to 2D projection. An alternative approach for 3D HPE is from video, given that human pose changes smoothly and structurally through time. With the advances in deep learning, estimating 2D human poses has become a standard technique. By first converting the human image sequence to 2D pose sequences, 3D HPE from the video can be reduced to the problem of inferring a 3D pose sequence from its 2D counterpart.

To solve this problem, earlier approaches[1, 6, 25, 30] use dilated temporal convolution or graph convolution networks to exploit spatial and temporal representations of 2D pose sequences. However, they generally rely on temporal dilation techniques or a predefined adjacency matrix to model temporal relationships. Consequently, the temporal connectivity inside these models is still limited.

Recently, transformer [28] with self-attention mechanism is shown capable of capturing long-range relationships of the input sequences, and has been successfully applied to many challenging computer vision tasks [2, 7, 34, 40]. The 2D-to-3D pose sequence estimation fits naturally to the transformer. As a matter of fact, most models that achieve current SOTA results are based on transformers [40, 14, 26, 38]. Among these models, PoseFormer [40] is a seminal work followed by MHFormer [14] and P-STMO [26]. This group considers all joints of each frame as a whole to learn the global pose-level temporal correlation across all frames. Their basic architecture is shown in Figure 1(b). As a separate group itself, MixSTE [38]’s basic architecture is shown in Figure 1(c). It decomposes a pose sequence into multiple joint sequences and uses stacks of spatial-temporal transformers to infer 3D poses by learning the local joint-level temporal correlation of each joint separately. These demonstrate that the temporal correlations of joints can be further dug to improve 3D HPE performance.

However, as Figure 1 illustrates, aforementioned transformer-based SOTA approaches focus on spatial nuances and hence are lack of a systematic information propagation from local joints to global poses temporarily. On the other hand, we are inspired from the residual neural network and vision transformers developments, such as the Feature Pyramid Network (FPN) [16] improves the ResNet[9], as well as Pyramid Vision Transformer (PVT) [32] advances the Vision Transformer (ViT) [7]. We found that both the FPN and the PVT in an hierarchical and systematic pyramid structure, improve their predecessors with better information propagation and consequently better task results.

Motivated by the literature, we propose a novel framework based on spatial-temporal transformers that leverages the hierarchical bottom-up structure of a pose shown in Figure 1(a). A pose is a composition of a set of articulated body parts, and each body part is a collection of corresponding joints. Analyzing a pose sequence is performed bottom-up in this hierarchy. First, joint sequences are independently passed through a temporal transformer. Then, the outputs of joint sequences are aggregated to become body part sequences and pass through temporal transformers for different body parts. These outputs are aggregated again to become the pose sequence as input to a temporal transformer for the pose. Finally all the outputs from the joints, parts, and pose transformer are fused to adaptively integrate the multi-level complementary information for 3D pose estimation. This hierarchically bottom-up approach enables us to exploit the structural nature of human poses and extract valuable information locally and globally. Figure 1(d) illustrates the systematic hierarchical spatial-temporal transformers. Specifically, bottom-up structural transformer encoders are organized to learn different correlations of joints from local to global as follows: (i) the Spatial Correlation (SC) of an individual single frame; (ii) the Joint-level Temporal Correlation (JTC) cross frames; (iii) the Body-part-level Temporal Correlation (BTC), which groups joints correlation across frames; and (iv) the Pose-level Temporal Correlation (PTC) of all joints across frames. Compared to the previous approaches illustrated in Figure 1(b) [40, 14, 26], and Figure 1(c) [38], which are less structurally and sporadically use transformer encoders, HSTFormer is able to model the joints’ spatial-temporal correlations more comprehensively and systematically. Our main contribution can be summarized as follows:

• •

  A novel transformer-based framework, hierarchical spatial-temporal transformers (HSTFormer), is proposed to model multiple levels of joints’ spatial-temporal correlations structurally in a bottom-up fashion. It is able to propagate the joints’ movement information smoothly and effectively. Such a framework is capable of accurately estimating 3D human poses in both simple and complex scenes following the 2D-to-3D lifting pipeline.

• •

  A body-part temporal transformer encoder is proposed to tackle the grouped joints correlation across frames. To the best of our knowledge, this is the first study in 3D pose estimation that uses transformer to focus on grouped joints correlation cross-temporal realm. The proposed design solely reduces MPJPE error with a significant amount of $4.7$ mm ($11.8\%$).

• •

  Extensive experiments on Human3.6M, MPI-INF-3DHP, and HumanEva datasets are conducted to demonstrate the superior performance and high generalization ability of the proposed HSTFormer for 3D human pose estimation. Specifically, it outperforms the existing SOTA approaches by a remarkable margin on the challenging MPI-INF-3DHP dataset, decreasing the MPJPE by 24.6% (from 54.9 mm to 41.4 mm).

Existing solutions to this problem can be divided into two categories: direct estimation approaches and 2D-to-3D lifting approaches. Direct estimation approaches [23, 13] designed end-to-end frameworks to estimate the joints’ 3D coordinates directly from images or videos without intermediate 2D pose representations, which are straightforward ways but remain challenges due to the lack of sufficient 3D in-the-wild data. 2D-to-3D lifting approaches [35, 25, 18, 40, 14, 38] have two stages which first detect 2D pose joints and then lift them to 3D. Many existing works are based on the 2D-to-3D lifting pipeline and currently outperform the direct estimation approaches. Meanwhile, their generalizability is improved because of ignoring the persons’ appearances when lifting. However, the 2D-to-3D lifting approaches also remain challenges due to occlusion and depth ambiguity in 2D poses. Especially, it is difficult to predict a 3D pose accurately only based on the spatial information of a 2D pose from a single frame [19].

To alleviate such an issue and to improve accuracy and robustness, many approaches [1, 6, 25, 30, 40, 14, 38] have integrated temporal information from 2D pose sequences to explore the joints’ spatial and temporal correlations. For example, Hossain and Little [10] applied the Long Short-Term Memory (LSTM) to explore the temporal information. Pavllo et al. [25] proposed a dilated temporal convolution network to capture global contextual information and estimate 3D pose from consecutive 2D sequences. Cai et al. [1] utilize graph convolution networks to exploit spatial and temporal graph representations of human skeletons for 3D pose estimation. However, convolutional neural network-based approaches typically rely on temporal dilation techniques and graph convolution network-based approaches depend on a predefined adjacency matrix to model spatial-temporal relationships. Eventually, the spatial-temporal connectivity explored by them is still limited.

Based on the mechanism of the transformer, the correlations through long sequence data can be extracted, which makes it possible to explore the temporal representations across the video frames for 3D HPE. Spatial and temporal transformers are gradually applied for 3D HPE [40, 14, 38, 26]. PoseFormer [40] explored the spatial joint correlations from each frame and the temporal joint across frames by using a spatial-temporal transformer structure, which is the first purely transformer-based approach for 3D HPE. MHFormer [14] was further proposed to learn the spatial-temporal information by presenting a multi-hypothesis transformer. MixSTE [38] developed a mixed spatial-temporal transformer to extract the all joints correlation across frames by applying the spatial transformer encoder and temporal transformer encoder alternatively. P-STMO [26] presented a spatial temporal many-to-one model which is pre-trained by using self-supervised learning for performance improvement. Our designs are inspired by the aforementioned works while making the model have the capability to learn the different levels of spatial-temporal information. Unlike previous works, we design two more modules to extract the spatial-temporal correlation of a group of joints across frames and adaptively integrate the learned multi-level spatial-temporal information of joints for accurate 3D HPE.

Figure 2 shows the overview of the proposed 3D human pose estimation method which follows the popular 2D-to-3D lifting pipeline. An embedding layer first converts the input 2D pose sequence to high dimensional representations. Let $X$ be the input 2D pose sequence $X\in\mathbb{R}^{T\times J\times 2}$ with $T$ frames and $J$ joints per frame. $X$ obtained by an off-the-shelf 2D pose detector from a video. The embedding layer maps $X$ to high-dimensional representations $X^{\prime}\in\mathbb{R}^{T\times J\times D}$ by:

where $W_{L}\in\mathbb{R}^{2\times D}$ is a linear projection and $LN(*)$ means layer normalization. In this work, we set $D=32$ to trade off the computational efficiency and performance as processed in [40].

Next, a cascade of different types of transformer encoders process these features in spatial and temporal domain. All these transformer encoders are based on the typical multi-head attention transformer encoder block (TE). Spatial transformer encoder (STE) processes the joint features in each frame independently. Joint temporal transformer encoder (JTTE) works on the temporal feature sequence of each joint separately. Body-part temporal transformer encoder (BTTE) takes grouped joint temporal feature sequences that corresponding to a particular body part as input. Pose temporal transformer encoder (PTTE) groups all joint temporal feature sequences into a temporal sequence as its input. These encoders have multiple layers, and we use $N_{en}$ to denote the number of layers. In summary, the propsoed STE, JTTE, BTTE and PTTE in Figure 2 correspond to the SC, JTC, BTC and PTC in Figure 1(d), respectively.

The following sections explain each of the aforementioned encoders. If we represent input features as a tensor with shape $T\times J\times D$, each encoder slices this tensor along different axes and with different block sizes as a sequence of input. Before diving into the details, Figure 3 provides an overall illustration on how the slicing is performed at each transformer encoder.

In this paper, we present HSTFormer, a novel framework based on hierachical spatial-temporal transformers for 3D human pose estimation from 2D pose sequence. HSTFormer processes pose sequence in a hierarchical paradigm, from joints to body-parts, and eventually to the entire pose. Four transformer encoders are concatenated following the kinesiological orders: spatial transformer encoder, joint temporal transformer encoder, body-part temporal transformer encoder, and pose temporal transformer encoder. In addition, a fusion model gathers all layers is used in predict the 3D poses. Extensive experiments and detailed ablation studies are conducted. A superiror and consistent performance is demonstrated with the effectiveness of the hierarchical design. Our method outperforms state-of-the-art approaches by a large margin on the challenging and MPI-INF-3DHP dataset with complex outdoor scenes.