Video-Based Human Pose Regression via Decoupled Space-Time Aggregation

Given a video frame $\mathcal{I}(t)$ at time $t$ containing multiple persons, we are interested in estimating locations of pose joints for each person. To enhance pose estimation for the frame $\mathcal{I}(t)$, we leverage the temporal dynamics from a consecutive frame sequence $\mathbf{\mathcal{S}}=\langle\mathcal{I}({t-T}),\dots,\mathcal{I}(t),\dots,\mathcal{I}({t+T})\rangle$, where $T$ is a predefined temporal span. Our method follows the top-down paradigm. Initially, we use an human detector to identify individual persons in the frame $\mathcal{I}(t)$. Subsequently, each detected bounding box is expanded by 25% to extract the same individual across the frame sequence $\mathbf{\mathcal{S}}$, resulting in a cropped video clip $\mathbf{\mathcal{S}}_{i}=\langle\mathcal{I}_{i}({t-T}),\dots,\mathcal{I}_{i}(t),\dots,\mathcal{I}_{i}({t+T})\rangle$ for every individual $i$. The goal of estimating human pose for individual $i$ within the specified video frame $\mathcal{I}(t)$ can then be denoted as

$$\{\textbf{x}_{i}^{j}(t)\}_{j=1}^{n}=\text{HPE}({\mathcal{S}}_{i}),$$

where $\text{HPE}(\cdot)$ denotes the human pose estimation module, $\textbf{x}_{i}^{j}(t)$ is the $j$-th pose joint for the individual $i$ in video frame $\mathcal{I}(t)$, and $n$ represents the number of joints for each person, e.g., $n=15$ for PoseTrack datasets [16, 1, 7]. For simplicity in the following description of our algorithm, unless otherwise specified, we will refer to a specific individual $i$.

We adopt the regression-based method to implement the human pose estimation module $\text{HPE}(\cdot)$. Compared to the heatmap-based method, the regression-based method offers several advantages: i) It eliminates the need for high-resolution heatmaps, resulting in reduced computation and storage demands. This makes it more apt for real-time video applications and facilitates deployment, especially on edge devices. ii) It provides continuous outputs, avoiding the quantization issues inherent in heatmap methods. In the regression-based pose estimation module $\text{HPE}(\cdot)$, the global feature maps extracted by a CNN backbone are fed into a regression module, which then directly produces the coordinates of the joints, i.e.,

$$\{\textbf{x}_{i}^{j}(t)\}_{j=1}^{n}=\text{REG}(\hat{\mathcal{S}}_{i}),$$

(1)

where $\text{REG}(\cdot)$ denotes the regression module and $\hat{\mathcal{S}}_{i}=\langle\mathcal{F}_{i}({t-T}),\dots,\mathcal{F}_{i}(t),\dots,\mathcal{F}_{i}({t+T})\rangle$. Here, $\mathcal{F}_{i}(t^{\prime})$ with $t^{\prime}\in[t-T,t+T]$ is the global feature maps extracted by the CNN backbone from the cropped image $\mathcal{I}_{i}(t^{\prime})$. Note, when regressing the pose joints at the current frame time $t$, we aim to utilize the temporal feature information across the video clip $\mathbf{\mathcal{S}}_{i}$, rather than solely relying on the feature information from the current frame $\mathcal{I}_{i}(t)$.

Prior work, such as Poseur [25], has demonstrated that the spatial dependencies among pose joints can be naturally captured by applying the self-attention mechanism [33] over them. It follows that we can also employ the self-attention mechanism on the global pose features in the temporal sequence $\hat{\mathcal{S}}_{i}$ to discern the temporal dependency of individual’s pose over the time interval $[t-T,t+T]$. As depicted in Fig. 1(b), each pose joint exhibits a relatively independent temporal trajectory. Hence, it’s more appropriate to model the temporal dependency at the joint level rather than for the entire pose. To this end, extra efforts are required to convert each global feature $\mathcal{F}_{i}(t^{\prime})$ into a set of joint-aware feature embeddings. This procedure is finished in the Joint-centric Feature Decoder (JFD) , which can be denoted as

$$\{\mathcal{F}_{i}^{j}(t^{\prime})\}_{j=1}^{n}=\text{JFD}(\mathcal{F}_{i}(t^{\prime})),\quad t^{\prime}\in[t-T,t+T],$$

(2)

where $\mathcal{F}_{i}^{j}(t^{\prime})$, termed a feature token, represents the feature embedding of the $j$-th joint of the pose at the video frame of time $t^{\prime}$. Let’s represent the feature tokens for each joint of the pose over the time span $[t-T,t+T]$ as $\tilde{\mathcal{S}}_{i}$. That is, $\tilde{\mathcal{S}}_{i}=\langle\{\mathcal{F}_{i}^{j}({t-T})\}_{j=1}^{n},\dots,\{\mathcal{F}_{i}^{j}({t})\}_{j=1}^{n},\dots,\{\mathcal{F}_{i}^{j}({t+T})\}_{j=1}^{n}\rangle.$ We subsequently utilize the feature tokens in $\tilde{\mathcal{S}}_{i}$ to model the spatiotemporal dependencies of pose joints via Space-Time Decoupling (STD),

$$\{\textbf{f}_{i}^{j}(t)\}_{j=1}^{n}=\text{STD}(\tilde{\mathcal{S}}_{i}),$$

(3)

where $\textbf{f}_{i}^{j}(t)$ is the aggregated space-time feature for the $j$-th joint of the pose at the current video frame $t$, which is then fed to a joint-wise fully connected feed-forward network to produce the coordinates of the joint.

Our proposed Decoupled Space-Time Aggregation Network (DSTA) is composed of three primary modules: the backbone, $\text{JFD}(\cdot)$, and $\text{STD}(\cdot)$. We learn DSTA by training the backbone, JFD, STD modules in an end-to-end manner. The architecture and workflow of DSTA are depicted in Fig. 2. Subsequent sections delve into the specifics of the JFD, STD, and the computation of the loss function.

As shown in Fig. 2, the purpose of JFD is to extract the feature embedding for each joint from the given global feature maps $\mathcal{F}_{i}(t^{\prime})$ with $t^{\prime}\in[t-T,t+T]$. As suggested by [25], one potential approach to construct the joint embedding is as follows: initially, a traditional regression method such as [20] is utilized to regress the joint coordinates from $\mathcal{F}_{i}(t^{\prime})$. For each joint, its $x$-$y$ coordinates are converted into position embedding using sine-cosine position encoding [33]. Concurrently, a learnable class embedding is designated for every joint type. The final feature embedding for each joint is derived by summing its position embedding with the respective class embedding. However, this approach loses crucial contextual information of the joints within the pose that is learned in the global feature maps $\mathcal{F}_{i}(t^{\prime})$. Though we can augment each joint with relevant contextual feature from the global feature maps, such as the approach in [25] which uses the joint’s embedding as a query and applies a multi-scale deformable attention module to sample features for each joint from the feature maps, this method incurs significant computational costs.

We employ a straightforward yet efficient approach to construct the joint embeddings from the provided global feature maps $\mathcal{F}_{i}(t^{\prime})$. Given the global feature maps produced by the backbone, previous heatmap-based methods convolve these maps via convolution layers to generate a heatmap feature for each joint [2, 22]. We follow this strategy, deriving the feature embedding for each joint from $\mathcal{F}_{i}(t^{\prime})$ through a convolution layer or a fully connected layer (FC). In our setup, the ResNet backbones (like ResNet50 or ResNet152) are followed by a global average pooling layer and a FC layer. The FC layer comprises $2048\times K$ neurons. Here, 2048 represents the dimensionality of $\mathcal{F}_{i}(t^{\prime})$ after undergoing global average pooling and flattening. Meanwhile, $K$ is calculated as $n\times 32$, where $n$ indicates the number of pose joints, and 32 signifies the dimension of the joint embedding. The output of the FC layer is the feature embedding for each joint, where the output is evenly divided into $n$ parts, denoted as $\{\mathcal{F}_{i}^{j}(t^{\prime})\}_{j=1}^{n}$, with each part representing a feature embedding for a joint. Further implementation details regarding more backbones (e.g., HRNet backbone) can be found in the supplementary material.

STD is designated to model the spatial and temporal dependencies between joints based on their embeddings over the time span $[t-T,t+T]$, i.e., all feature tokens in $\tilde{\mathcal{S}}_{i}=\langle\{\mathcal{F}_{i}^{j}({t-T})\}_{j=1}^{n},\dots,\{\mathcal{F}_{i}^{j}({t})\}_{j=1}^{n},\dots,\{\mathcal{F}_{i}^{j}({t+T})\}_{j=1}^{n}\rangle$. In numerous applications [6, 25], the self-attention mechanism’s proficiency in capturing long-distance dependencies within sequences has been thoroughly demonstrated [33]. Thus, a direct approach to capturing the spatio-temporal dependencies between joints is to apply the self-attention module to the sequence of feature tokens in $\tilde{\mathcal{S}}_{i}$,

$$\{\mathcal{\hat{F}}_{i}^{j}({t})\}_{j=1}^{n}=\text{S-ATT}(\tilde{\mathcal{S}}_{i}),$$

(4)

where $\text{S-ATT}(\cdot)$ denotes the self-attention module [33], and $\mathcal{\hat{F}}_{i}^{j}({t})$, which encodes the spatial and temporal information learned by the S-ATT module, represents the updated feature token for the $j$-th joint at the current video frame $t$. In our implementation, the S-ATT module adheres to the conventional Transformer architecture [33]. In our setup, $4$ identical layers are stacked sequentially. Each layer comprises two sub-layers: the first one employs a multi-head self-attention mechanism, and the second one utilizes a simple, token-wise, fully connected feed-forward network. The input feature tokens pass through these modules in sequence, each producing an updated version that serves as the input for the subsequent layer. Additionally, each of the initial input feature tokens is equipped with a learnable position embedding, and their sum forms the final input.

During training, the entire model undergoes end-to-end optimization, aiming to minimize the discrepancy between the coordinates of the predicted joints and the ground truth joints in the current frame $t$. To boost the regression performance, we employ the residual log-likelihood estimation loss (RLE) as proposed in [20], in lieu of the conventional regression loss ($l_{1}$ or $l_{2}$). We extend the RLE loss, originally designed for image-based pose regression, to the context of video-based pose regression. Given an input cropped video clip, $\mathcal{S}_{i}$, for individual $i$, we calculate a distribution, $P_{\theta,\phi}(\{\textbf{x}_{i}^{j}(t)\}_{j=1}^{n}|\mathcal{S}_{i})$, which reflects the likelihood that the ground truth at the current frame $t$ appears at the predicted locations $\{\textbf{x}_{i}^{j}(t)\}_{j=1}^{n}$. Here, $\theta$ represents the parameters of our model, and $\phi$ represents the parameters of a flow model. The flow model is not required to operate during inference, thereby introducing no additional overhead at test time. The learning process involves the simultaneous optimizations of the model parameters $\theta$ and $\phi$, aiming to maximize the probability of observing the ground truth $\mu_{g}$. This is achieved by defining the RLE loss as follows:

$$\mathcal{L}_{rle}=-P_{\theta,\phi}(\{\textbf{x}_{i}^{j}(t)\}_{j=1}^{n}|\mathcal{S}_{i})\Big{|}_{\{\textbf{x}_{i}^{j}(t)\}_{j=1}^{n}=\mu_{g}}.$$

(10)

For a more detailed discussion and further information about the RLE loss, we refer readers to [20].

We have conducted evaluations on three widely-utilized video-based benchmarks for human pose estimation: PoseTrack2017 [16], PoseTrack2018 [1], and PoseTrack21 [7]. These datasets contain video sequences of complex scenarios involving rapid movements of highly occluded individuals in crowded environments. To assess the performance of our models, we utilize the Average Precision (AP) metric [28, 2, 22, 20]. The AP is calculated for each joint, and the mean AP across all joints is denoted as mAP. Our method is implemented using PyTorch. Unless otherwise specified, the input image size is $384\times 288$ when using the HRNet-w48 backbone, while for other backbones, the input image size is $256\times 192$. We pretrained the backbones on the COCO dataset. For further implementation details, please refer to the supplementary materials provided.

We conduct ablation experiments to analyze the influence of each component using the PoseTrack2017 validation set. The temporal span $T$ is set to 1, consisting of one preceding and one subsequent frames, totalling 2 auxiliary frames, and the ResNet-50 backbone is employed.

Impact of different modules. Table 6 lists the performance impact of each module of our approach. When we adopt global pose features instead of modeling the temporal dependency at the joint level, i.e., without the JFD and STD modules, the algorithm achieves an mAP of 73.8, decreasing 4.8 points. When capturing spatiotemporal relations based on joints using Eq. 4, i.e., Space-Time coupling, the accuracy reaches 74.8 mAP. It aligns with our assumption that modeling temporal dependency at the joint level, as opposed to the entire pose, is more appropriate. Furthermore, when using the SD and TD modules to separately model the temporal dynamic dependencies and spatial structure dependencies, the algorithm achieves the highest accuracy of 78.6 mAP. It proves that the temporal dependencies of each joint should be individually captured, as every joint exhibits an independent temporal trajectory. Meanwhile, we can see that the TD module capturing temporal dependencies has a much greater impact (78.1) on overall performance than the SD module capturing spatial dependencies (71.4). We believe this is mainly because the feature token extracted by the JFD module for each joint already contains its spatial context information within the pose. This means the spatial structure information complemented by the SD module is limited.

Choice of JFD. As discussed in Sec. 3.2,  [25] proposes an alternative approach to constructing feature embeddings for each joint. We compare our JFD module with this method in Table 6, where  [25] (b) augments the feature embedding of each joint with relevant contextual features while [25] (a) does not. As can be seen, our approach improves accuracy by 4.0 points with a relatively small computational overhead, achieving the highest accuracy.

Auxiliary Frames. In addition, we investigate the impact of using different numbers of auxiliary frames. The results presented in Table 6 consistently demonstrate that increasing the number of auxiliary frames leads to improved performance across various backbone networks. It aligns with our intuition that more auxiliary frames can provide more complementary information, thereby facilitating the enhancement of pose estimation for the key frame.