Spatial-Temporal Decoupling Contrastive Learning for Skeleton-based Human Action Recognition

Skeleton-based action recognition aims to classify actions from sequences of human keypoints. In early studies, RNN was a natural choice to handle the skeleton sequences. HBRNN Du et al. (2015) applied a hierarchical RNN architecture to model the long-range temporal dependency of the skeleton sequences. STA-LSTM Song et al. (2017) proposed an end-to-end spatial and temporal attention model to focus on discriminative joints of the skeleton within each frame of the inputs. Inspired by the success of CNN in image recognition, CNN-based methods have also been studied. HCN Li et al. (2018) treated the joint dimension as channels to aggregate different levels of spatial-temporal context information. TA-CNN Xu et al. (2022) adopted a pure CNN architecture to model the irregular skeleton topology.

The human body can be abstracted as a graph structure, which treats the joint as a node and the bone between two joints as an edge. Thus, GCN-based methods were introduced to this task. ST-GCN Yan et al. (2018) adopted GCN on the predefined spatial-temporal graphs to model the relations between joints. 2S-AGCN Shi et al. (2019) modeled the correlation between two joints given corresponding features with the self-attention mechanism. CTR-GCN Chen et al. (2021a) proposed a channel-wise topology refinement graph convolution to model fine-grained relation. InfoGCN Chi et al. (2022) combined a learning objective and an encoding method to break the information bottleneck.

With the popularity of transformers in computer vision, transformer-based methods have also been investigated for this task. ST-TR Plizzari et al. (2021) proposed a two-stream transformer architecture to model the spatial and temporal dimensions respectively. Hyperformer Zhou et al. (2022) applied transformer to model spatial-temporal features from the hyper-graph of the sequences.

Recently, contrastive learning has been widely used in self-supervised representation learning. The core idea is to learn the discriminative features by pulling the positive pairs and pushing away the negative pairs in the embedded space. The positive pairs are features from transformed sample versions with different augmentations, and the negative pairs are from different samples. InstDisc Wu et al. (2018) maintained a memory bank for storing representations. SimCLR Chen et al. (2020) proposed to adopt a huge batch size to learn more useful representations. MoCo He et al. (2020) built a dynamic dictionary with a momentum update based on a memory bank to keep the stored representations consistent. Inspired by this, our work tries to introduce memory bank mechanisms for contrastive learning to obtain discriminative features.

Contrastive learning was usually adopted in prior works Xu et al. (2023) to learn the invariance from the skeleton sequences for unsupervised learning in this field. Moreover, several works are applying contrastive learning to supervised action recognition. GAP Xiang et al. (2023) proposed a framework for contrasting the features from visual and text modalities to enhance the representation by using knowledge about actions and human body parts. SkeletonGCL Huang et al. (2023) explicitly explored the rich cross-sequence relations by using graph contrastive learning. However, SkeletonGCL can only be incorporated into the GCN-based methods, which limits its generalization. FR-Head Zhou et al. (2023) employed multi-level contrastive learning in GCN-based methods to distinguish ambiguous actions.

However, the methods mentioned above are more like surgery for GCN-based skeleton encoders, leading to limitations on generalizability to other types of methods.

Skeleton-based action recognition is to predict the categories from the input sequences. We define the human skeleton sequence with $J$ keypoints and $T_{0}$ frames in 3D space as $\boldsymbol{S}\in\mathbb{R}^{J\times T_{0}\times 3}$. The sequence is fed into a skeleton encoder $\mathcal{F}$ to extract the features $\boldsymbol{X}\in\mathbb{R}^{J\times T\times C}$. The skeleton encoder aggregates spatial and temporal information to obtain discriminative spatial-temporal features. After fully spatial-temporal aggregations, a global pooling (GP) layer is adapted to summarize the global features. Finally, a softmax-based fully-connected (FC) layer maps the global feature to a probability prediction $\hat{\boldsymbol{y}}\in\mathbb{R}^{K}$ of $K$ candidate categories. The process can be defined as:

In the process of optimization, a standard cross-entropy loss is applied to supervise the prediction distribution and the ground truth $\boldsymbol{y}$ as follows:

Human actions have the property of spatio-temporal coupling, the feature $\boldsymbol{X}$ after aggregation contains complex spatial and temporal relations, which may mix the spatial and temporal information. Thus, we decouple the feature into the spatial-aware and temporal-aware features to reduce the spatial-temporal information coupling. To further obtain more discriminative features, we apply the decoupled features to contrastive learning to explore the cross-sequence semantic relations. The core is to explore the latent data distributions by pulling together the features from the positive pairs and pushing away the negative pairs.

NTU RGB+D. NTU60 Shahroudy et al. (2016) is the most widely used dataset for skeleton-based action recognition. It contains 56,880 samples in 60 action classes. These samples are performed by 40 distinct subjects. It has been configured into two benchmarking structures: cross-subject (X-sub) and cross-view (X-view). In the X-sub, sequences from 20 individuals are allocated for training purposes, with the remaining 20 sets of sequences earmarked for validation. Meanwhile, under the X-view, skeleton sequences are divided based on camera views; two camera views are used for training while the remaining ones are utilized for evaluation.

NTU RGB+D 120. NTU120 dataset Liu et al. (2019) adds 57367 new skeleton sequences and 60 new action classes to the original NTU60 dataset. There are 32 various configurations in it, each of which depicts a different location and background. The authors offered the cross-subject (X-sub) and cross-setup (X-set) as two benchmark evaluations. In the X-sub, sequences from 53 subjects are for training, and the other 53 subjects are for testing. In the X-set, skeleton sequences are split by setup ID. Samples from even set-up IDs are used for training, and the odds are used for evaluation.

Northwestern-UCLA. NW-UCLA dataset Wang et al. (2014) contains 1494 video clips of 10 different actions captured from three Kinect cameras. We follow the same evaluation protocol in Wang et al. (2014): the first two cameras for training and the other for testing.

To fully validate the effectiveness and generalizability of STD-CL, we take four different model-based approaches as baseline models. (1) CNN-based method. We select HCN Li et al. (2018) as the CNN-based method to validate STD-CL. We reproduce the experimental results using the released code and conduct our STD-CL on it. And we follow the training recipes as the paper describes. (2) GCN-based method. 2S-AGCN Shi et al. (2019) and CTR-GCN Chen et al. (2021a), the widely used GCN-based model for skeleton-based action recognition. For CTR-GCN, we follow their training strategies. Particularly, for 2S-AGCN, we update the training strategies from CTR-GCN, which can improve the performance significantly. (3) Transformer-based method. For transformer-based methods, we choose Hyperformer as the baseline. More implementation details are presented in the supplementary material.

We compare our method with previous SOTA methods in Tables 1 and 2. To fully validate the proposed STD-CL, we report the experimental results in a single modality and the ensemble results of multi-modality. $J$ means the joint stream, $B$ denotes the bone stream, $2S$ denotes the ensemble results of the above two streams in default, and $4S$ incidents the ensemble results of additional joint motion and bone motion streams. Particularly, due to the design of their model architecture, the experimental results of VA-LSTM Zhang et al. (2017), ACG-LSTM Si et al. (2019) and HCN Li et al. (2018) are the ensemble results of two streams. Therefore, we classify the results into $2S$.

As shown in Table 1, on NTU60 and NTU120, combined with STD-CL, all four various baselines achieve solid gains on these four benchmarks of the two datasets over different settings. Moreover, the improvement in a single stream can be generalized to the results of the multi-stream ensemble. Take CTR-AGCN as an example, it improves by 0.6% (89.8% to 90.4%) in the joint stream, 0.6% (90.2% to 90.8%) in the bone stream, and 0.2% (92.0% to 92.2%) in their fusion result.

As shown in Table 2, on NW-UCLA, STD-CL outperforms 2S-AGCN by 1.5%. It also outperforms CTR-CCN by 0.7%. And it outperforms the recent work InfoGCN and Hyperformer by 0.2% and 0.4%, respectively. Considering that the performance on this dataset is already very high, such improvement brought by STD-CL is significant.

We compare our method with the recent contrastive learning methods in supervised skeleton-based action recognition. The comparison is conducted on the X-sub benchmark of the NTU120 dataset. As shown in Table 3, our STD-CL achieves competitive improvements. Furthermore, most previous methods are only compatible with GCN-based methods. To take advantage of our method, we combine the STD-CL with Hyperformer Zhou et al. (2022), which is a transformer-based method. Our method outperforms previous contrastive learning methods by 1.2%, which is a promising result. For more comparisons, please refer to the supplementary materials.

In this section, we evaluate the different experimental settings on the X-sub benchmark of the NTU120 dataset to verify the design of the proposed STD-CL. For more ablation studies, please refer to the supplementary materials.

Contrastive Strategies. In Table 3, we conduct experiments to validate the contrastive strategy. We employ the decoupled spatial features or decoupled temporal features to contrastive learning, and these lead to performance improvements, which are owed to the decrease in the degree of mixture in the global features. Furthermore, we find that global feature contrast can improve the performance on the baseline with 0.5% (84.9 % to 85.6 %), which owes to the explicit modeling of latent data distributions. Furthermore, we combine the global feature contrast and our STD-CL, and the performance drops slightly (85.6% to 85.5%), which may be due to that the global features contrast limits the feature decoupling to some extent.

Performance on Different Level Actions. To evaluate the effectiveness of the proposed STD-CL, we conduct recognition accuracy for different action categories with different levels of difficulty. Specifically, we gather actions whose accuracy is over 90% as Easy Level, between 80% to 90% as Medium Level, and lower than 80% as Hard Level for their respective classification results. The experimental results are displayed in Table 5. From the results, we can observe that the proposed STD-CL achieves relatively significant improvements in the hard and medium-level actions, which include more fine-grained action categories. Moreover, the proposed STD-CL has made some gains in the easy-level actions, thanks to the explicit modeling for the latent data distributions of contrastive learning.

Furthermore, we compare our results with a SOTA model CTR-GCN Chen et al. (2021a), and the results of class with accuracy differences higher than 3% between CTR-GCN and our method are displayed in Figure 4. We can obverse that actions such as “hit with object”, “point to something”, “wipe face”, “reading” and “yawn” benefit from the proposed STD-CL due to the great capacity to obtain discriminative and semantically distinct features. Besides, the reason for the poor performance of actions “take object out of bag”, “put on a shoe”, and “tear up paper” is that such actions are strongly object-related, making it challenging to recognize and contrast.

Training Consumption. In Table 6, we report the training consumption on NTU120. With our method, the training memory usage only slightly increases with different baseline models, and the increase in training time remains within an acceptable range, which proves the efficiency of the design.

In this section, we validate our STD-CL through t-SNE Van der Maaten and Hinton (2008) distribution visualization of feature representations in the test set of the X-sub benchmark of the NTU120 dataset. As shown in Figure 5, our STD-CL can exploit the data distributions explicitly with more compact intra-class distances and clear classification boundaries. Thus, the features extracted from the proposed STD-CL are more discriminative, which shows that the decoupled feature contrast improves the feature extraction capacity of the skeleton encoders.