Balanced Representation Learning for Long-tailed
Skeleton-based Action Recognition

Mixup [53] was proposed as a regularization technique for improving the generalization of train data. The method constructs virtual training examples and extends the training distribution by linear interpolations of feature vectors. It works by mixing two images and their labels linearly as Eq.1

where $(x_{i},y_{i})$ and $(x_{j},y_{j})$ are two examples drawn at random from the training data, and $\lambda\in[0,1]$.

For skeleton-based action recognition, spatial structure information is necessary to understand the action. Moreover, the same skeleton parts have distinctive significance in different actions. Hence, for generating meaningful samples to attenuate the skew in sample space, the proposed rebalanced partial mixup considers mixing the skeleton part of different bodies in a rebalanced manner.

The rebalanced manner means using two respective mixing factors to mix up in the sample space and label space. This is because the previous single mixing factors simply extend the training sample space, but do not further address the imbalance of the action data. The proposed strategy thereby turns to disentangle the mixing process of sample space and label space. An adaptive label factor is designed to provide an effective regularization focusing on tail classes. Through rebalancing the label space, the strategy can assign a higher weight to the minority class and alleviate the effects of imbalanced action data.

The formulation of the rebalanced partial mixup is defined as follows:

where $x_{i},x_{j}$ are the joint indices of the body. $\lambda_{x}$ means the ratio of selected joints for the whole body. The exact form of $\lambda_{y}$ is defined according to the following formula:

In Eq.5, $n_{i}$ and $n_{j}$ denote the number of samples in the corresponding action classes that sample $i$ and sample $j$ belong to. $k$ is a hyper-parameter, which indicates the degree of imbalance between the two categories. Specifically, if $n_{i}$ is more than $k$ times larger than $n_{j}$, $\lambda_{y}$ is set to $0$, less than $1/k$, $\lambda_{y}$ is set to $1$, otherwise, $\lambda_{y}$ is equal to $\lambda_{x}$. Different parts of the body are chosen to formulate new samples, and the corresponding rate is adjusted accordingly. The radio for selecting random samples is set to $1/16$, which means choosing one sample per $16$ samples. We adopt combining the upper body and the lower body directly as shown in Fig. 3 and apply the rebalanced label design.

Actually, the key to the proposed method is generating meaningful samples in a rebalanced manner. The rebalanced partial mixup can effectively aggregate the spatial structure information of long-tailed action samples, expanding the space of skeleton samples. And the rebalanced label design rebalances the label space and alleviates the effects of imbalanced data. The performance can be further improved by the effective regularization technique. In this way, the strategy generates more meaningful samples.

Temporal distinctive information is also important for long-tailed action data. For the oversampling of imbalanced skeleton data, the sub-string sampling with bilinear interpolation does not make full use of the temporal information of the skeleton sequences. Hence, temporal reverse perception is proposed to help the model better understand temporal information in skeletal data.

First, the original input sequence is divided uniformly into some splits with equal lengths and then one frame is randomly selected from each split. Through the approach, the sampler can generate more distinctive skeleton sequences with similar temporal attributes. And the temporal integrity of the skeleton sequences is well-preserved after multiple random samplings. In addition, the set of tested skeleton sequence lengths is $\{300,100,64,50,32,16\}$. It is experimentally verified that $64$ is the best for long-tailed skeleton data. The uniform sampler effectively maintains sample variability, especially when combined with the rebalanced partial mixup method.

Furthermore, due to the limited number of samples in the tail classes, oversampling can lead to information redundancy. This redundancy of temporal information reduces the benefit of generating more data, which influences representation learning. The proposed temporal reverse perception can drive the sampler to take special enhancements for the tail classes. Specifically, the reverse sampler adaptively selects the sampling frequency according to the actual action data distribution, which can further reduce the invalid overlap and maintain meaningful temporal information.

Briefly, the temporal reverse perception is composed of the uniform sampler and the reverse sampler. The approach is to sample frames uniformly from the original sequences and take more samples at random for the tail classes. Combined with rebalanced partial mixup and temporal reverse perception, the proposed spatial-temporal action exploration strategy can provide more valuable data for the network to learn better representation.

This indoor captured dataset [1] is a large-scale human action recognition dataset containing 56,880 skeleton action sequences of 60 action classes. The dataset recommends two benchmarks: (1) cross-subject (X-sub): training data comes from 20 subjects, and testing data comes from the other 20 subjects. (2) cross-view (X-view): training data comes from camera views 2 and 3, and testing data comes from camera view 1.

This is currently the largest indoor action recognition dataset [2], which is an extended version of NTU RGB+D with 60 more action classes. The dataset contains 114,480 videos and consists of 120 classes. Similarly, the recommended two settings are suggested: (1) cross-subject (X-sub): training data comes from 53 subjects, and testing data comes from the other 53 subjects. (2) cross-setup (X-set): training data comes from 16 even setup IDs, and testing data comes from 16 odd setup IDs.

Northwestern-UCLA [5] is captured by using three Kinect cameras. It contains 1494 samples covering 10 action categories, and each action category is performed by 10 subjects. We adopt the same evaluation protocol in [5]: training data comes from the first two cameras, and testing data comes from the other camera.

The dataset [8] is adapted from the Kinetics 400 video dataset using the OpenPose toolbox in 2D keypoint modality. It contains 240,436 training and 19,796 evaluation skeleton clips over 400 classes, where each skeleton graph contains 18 body joints. At each time step, two people are selected for multi-person clips based on the average joint confidence. Regarding the experiments using 2D estimated skeleton, we use publicly available HRNet skeletons provided by PoseConv3D [34] for fair comparison. Typically, Top-1 classification accuracy is used in the evaluation protocol.

To validate the effectiveness of our method for long-tailed skeleton data, we construct the long-tailed datasets based on NTU 60, NTU 120, and Northwestern-UCLA. As stated before, Kinetics 400 is composed of many unique short videos. The statistics of samples versus classes indicate that the dataset itself actually exhibits a long-tailed data distribution. Therefore, we also choose Kinetics Skeleton 400 for the experiments.

For constructing the long-tailed version of the above datasets, we follow the general setting of long-tailed learning, i.e., truncating a subset with the Pareto distribution from the balanced version [16]. Specifically, to create the imbalanced version, the number of training examples per class is reduced, and the validation set is kept unchanged [45]. Imbalance ratio $\beta$ is utilized to denote the ratio between sample sizes of the most frequent and least frequent class, i.e., $\beta={\rm max}_{i}\{{n}_{i}\}/{\rm min}_{i}\{{n}_{i}\}$. The whole long-tailed imbalance follows an exponential decay in sample sizes across different classes. For NTU 60, the maximum number of samples per class under cross-subject and cross-view settings is set as 600. The imbalance ratio is fixed at 100. As for NTU 120, we adjust the maximum number of samples per class to 600 for the cross-subject setting and 400 for the cross-view setting, taking into account the change in the number of categories and samples. Additionally, we maintain the imbalance ratio at 100. Regarding Northwestern-UCLA, the maximum number of samples per class is set as 100. Due to the limited number of training samples per class, the tail-most class would only have one sample if the imbalance ratio is set to 100. Hence, we set the imbalance ratio to 10. Fig. 4 shows the number of training samples per class on different datasets.

The spatial-temporal action exploration strategy and detached action-aware learning schedule of the proposed method are proposed for long-tailed skeleton-based action recognition, and can be combined with any action recognition backbones. Meanwhile, as mentioned before, various graph models do not have a large influence on the quality of representation learning. Hence, ST-GCN++ [12] is chosen as our backbone for the experiments. It is a recent representative approach using the GCN technique.

In the experiments, the initial learning rate is set to 0.1 and decays with a cosine LR scheduler. Moreover, stochastic gradient descent (SGD) optimization with the Nesterov momentum of 0.9 and the weight decay of $5\times 10^{-4}$ is employed to tune the parameters. Considering the large differences in the size of the datasets, we set the batch size for NTU 60, NTU 120, Northwestern-UCLA, and Kinetics Skeleton 400 datasets to 128, 128, 16, and 32, respectively. The model is implemented with PyTorch deep learning framework. And the maximum number of training epochs is set to 120. We set $k=3$ using the grid search. When adopting the detached action-aware learning schedule, the last 20 epochs are set as the detached stage. We set $\beta_{y}\in[0.99,0.9999]$, i.e. $\upsilon=0.99$ and $\lambda=0.0099$. As for the data pre-processing method and hyper-parameter settings, we follow the strategy in previous work [12]. The comparison methods use the same training and test sets as the proposed method in this paper, and these methods are implemented using official open-source code. The same data augmentation is used for fair performance comparison.

The proposed method is compared with quite a few SOTA methods on both X-sub and X-view benchmarks of the NTU 60 dataset. The comparisons under standard and long-tailed settings are displayed in Table I. There are several important observations as follows.

In the long-tailed setting, there is a general degradation in the performance of previous methods. It verifies that current action recognition methods trained on balanced action datasets can not adapt well to long-tailed data. Rather, the proposed method significantly improves the performance in the long-tailed setting, i.e., 81.8% for X-sub benchmark and 85.4% for X-view benchmark. The proposed method outperforms FR Head [38] by 4.5% and 4.9%, and the baseline method ST-GCN++ by 6.8% and 5.4% on the two benchmarks respectively. In the standard setting, the proposed method also achieves the performance on par with other SOTA methods. The superiority is achieved by balanced representation learning, which generates more valuable skeletal samples and mitigates representation bias.

Note that many action recognition methods employ the multi-stream fusion framework to improve performance. For a fair comparison, we follow the rule of these methods by fusing the results of four modalities. The ensemble of 4-stream which includes joint, bone, joint motion, and bone motion is denoted as 4 ensemble. Here, motion means joint movement between two subsequent time frames. In addition, we add the skip-modal and skip-motion-modal representations, so the proposed method can be denoted as 6 ensemble. In Table I, the results of 4 ensemble and 6 ensemble are both reported in detail. In other tables, only the results of 6 ensemble are reported by default.

It is known that ST-GCN [9] is currently the most popular backbone model for skeleton-based action recognition. Our backbone network, i.e. ST-GCN++ [12], is actually an upgradation over ST-GCN, achieving better performance under the standard setting. However, under the long-tailed setting, the performances of the two models are relatively close. The imbalanced data leads to skewed sample space and results in representation bias, ultimately degrading the representations learned by previous methods. Integrated with the proposed method, the performance of ST-GCN++ under the long-tailed settings catches up with the accuracy of the original ST-GCN under the standard settings. The proposed method generates more valuable samples to reshape the sample space, and mitigates existential bias in the representation space, effectively addressing the dilemmas of skewed sample space and representation bias. The results imply that the proposed method can learn unbiased balanced representations from imbalanced data and handle imbalanced skeleton data well.

On the NTU 120 dataset, as shown in Table II, the proposed method also boosts the performance significantly when compared to other methods. The proposed method achieves the accuracy of 69.7% for the X-sub benchmark and 71.3% for the X-set benchmark under the long-tailed setting. In addition, the proposed method outperforms FR Head by 4.1% on X-sub benchmark and 3.1% on X-set benchmark, which is also a remarkable improvement. Similar to the results on NTU 60 dataset, the performances demonstrate the effectiveness of the proposed method.

For Northwestern-UCLA and Kinetics, Table IV and Table V tabulate the experimental results. From Table IV, it can be easily found that an obvious performance improvement has been attained by the proposed method. The proposed method achieves an accuracy of 89.2% for Northwestern-UCLA under the long-tailed setting, which outperforms InfoGCN [37] by 17.2%. This improvement could be attributed to the relatively smaller sample size of Northwestern-UCLA. The proposed method can generate more distinctive samples when the data is small, which can further enhance the diversity of the data and improve the model performance more effectively. As shown in Table V, it can be seen that the proposed method outperforms the performance of other SOTA methods by 0.9% in Top-1 accuracy on Kinetics Skeleton 400 dataset, which proves that our approach works well in the real-world long-tailed datasets.

To demonstrate the advantages of the proposed method over current algorithms for long-tailed data distribution, we have conducted extensive experiments to compare with the following competing algorithms: (1) Cross-entropy (CE) loss; (2) Focal loss [56], which is a common re-weighting loss function for imbalanced class samples; (3) Weighted softmax loss [41], which directly uses label frequencies of training samples for loss re-weighting; (4) Random oversampling (ROS) [59], which oversamples minority samples to balance the class distribution in the training data; (5) Mixup [53], which generates new training data by mixing two samples and their labels; (6) Remix [43], which oversamples minority classes by assigning higher weights to the minority labels when using Mixup; (7) CB loss [17], which introduces the effective number to approximate the expected sample number of different classes; (8) Label-Distribution-Aware Margin Loss with deferred re-weighting (LDAM-DRW) [45], which regularizes the minority classes by minimizing a margin-based generalization bound; (9) Equalization loss [20], which ignores the gradients for rare categories; (10) Balanced Softmax [46], which is an elegant unbiased extension of Softmax. (11) GumbelCE [47], which develops a Gumbel optimized strategy. (12) KPS loss [42], which presents a key point sensitive loss to improve the generalization performance of the classification model.

For fair comparisons, the same data augmentation is applied to all the compared algorithms in the experiments. Training with cross-entropy (CE) loss is regarded as the baseline method. If any algorithm utilizes a different loss function, the CE loss is substituted for the corresponding loss. Experiments are conducted on the long-tailed setting of NTU 60 and NTU 120. The performance is measured by classification accuracy using the joint data. Following [44], we also report the accuracy of three disjoint subsets: Many-shot classes (classes with more than 100 training samples), Medium-shot classes (classes with 20 to 100 samples), and Few-shot classes (classes under 20 samples).

The performance comparison is shown in Table III. It can be observed that, compared to vanilla training with CE loss, the proposed method maintains performance for the many-shot classes and significantly improves recognition accuracy for the medium-shot and few-shot classes. Meanwhile, the proposed method achieves significant performance advantages compared to other long-tailed methods. For the many-shot classes, the performance improvements of different long-tailed algorithms are relatively close, but for the medium-shot and few-shot classes, the proposed method is superior. The superiority is achieved through balanced learning of consensus knowledge and specific patterns of tail classes, resulting in more effective constraints being imposed on different classes. Balanced representation learning helps the model to understand the action data effectively regardless of the head or tail category, thus improving the overall recognition performance.

To further analyze the effectiveness of the proposed method, we conducted additional experiments to validate the impact of the spatial-temporal action exploration strategy and the detached action-aware learning schedule. The performance is measured by classification accuracy on NTU 60 using the joint data.

Firstly, we analyze the spatial-temporal action exploration (STAE) and compare the accuracy gains of rebalanced partial mixup (RPM) and temporal reverse perception (TRP) in Table VI. The STAE strategy leads to significant improvements, with gains of 3.4% on X-sub benchmark and 2.9% on X-view benchmark. It is proven by the results that both RPM and TRP contribute to the action recognition model. The STAE strategy strives to aggregate spatial structure information and make full use of temporal distinctive information. The experimental results demonstrate that the proposed strategy generates more meaningful skeletal samples, and these valuable data further help the model to understand the long-tailed action.

Secondly, to validate the efficacy of the proposed detached action-aware learning schedule, we construct different ablation experiments and report their performance in Table VI. It can be seen that directly using re-weighting losses throughout the training process leads to a certain degree of performance degradation. Despite the performance of focal loss being relatively stable, the gap between the weighted loss and CE loss is obvious. In addition, for the X-sub benchmark, the accuracy of the action-aware loss also drops by 0.1%. And then the detached learning schedule is adopted. It can be observed that the performances of different losses become generally better. Meanwhile, the detached action-aware loss is more stable, bringing a performance boost compared with the original CE loss.

Finally, we scrutinize the contribution of each component. The results are listed in Table VII. $\rm C_{STAE}$ and $\rm C_{DAA}$ refer to the spatial-temporal action exploration strategy and detached action-aware learning schedule respectively, and $\rm C_{MS}$ denotes the ensemble of multiple streams. Accordingly, the absolute accuracy and $\Delta$ are reported. $\Delta$ represents the relative accuracy improvement over the baseline ST-GCN++ [12] by using combinations of different components. As shown in Table VII, we can observe that the performance of the model is improved by different components. By comparing the performance of variants denoted by $\mathcal{M}_{1}$, $\mathcal{M}_{2}$, $\mathcal{M}_{3}$, and $\mathcal{M}_{4}$, $\rm C_{STAE}$, $\rm C_{DAA}$, and $\rm C_{MS}$ bring 3.4%, 2.6%, and 0.8% improvement respectively. By mining multiple streams, the model can obtain more valuable information for action recognition. After combining all components, the method results in $\Delta=6.8\%$ improvement on the NTU-60 X-sub setting. The results indicate that the components of the proposed method contribute to understanding the long-tailed action data.

The proposed balanced representation learning method (BRL) also has a superior interoperability capacity and is easy to incorporate into other methods. In this subsection, the scalability of the proposed method is assessed on NTU 60 with a combination of various skeleton-based action recognition backbones. We apply the proposed BRL to popular recognition methods, i.e. ST-GCN++ [12], MS-G3D [11] and CTR-GCN [36].

The results with different recognition backbones are tabulated in Table VIII. For ST-GCN++, the method results in a 6.8% gain on X-sub benchmark and a 5.4% gain on X-view benchmark. Combined with the proposed method, the performances of other algorithms are also improved to some extent. For example, for MS-G3D, the method brings as much as 7.6% gain on X-sub benchmark and 4.8% gain on X-view benchmark. And for CTR-GCN, the method brings a 7.7% gain on X-sub benchmark and a 4.6% gain on X-view benchmark. The results show that our method consistently improves the performance of different action recognition backbones. It is inspiring that the proposed method can be easily integrated into various action recognition algorithms.

To further verify the effectiveness of the proposed method, the learned representations of five head categories and five tail categories are visualized in Fig. 5. It exhibits the t-SNE embeddings of the latent representations across different methods. The sample points of tail classes are overlaid by grey area to show the improvements brought by different components. It can be observed that the proposed method can obtain more accurate classification boundaries for tail classes. With more valuable data and more effective weighting constraints, the proposed method can indeed learn balanced representations for long-tailed skeleton action datasets.

To analyze the performance of the proposed method on different action classes, we construct the following experiments.

On the one hand, based on Table VII, we further analyze the effects of different components on the performance of different classes. As shown in Fig. 6, three schemes are tested on the X-sub benchmark of NTU 60-LT and NTU 120-LT, including BRL, BRL without DAA, and BRL without STAE and DAA. In order to facilitate the visualization, we aggregate the classes into 10 groups according to the label frequency. To be specific, for NTU 60, the dataset has 60 classes, and each group includes 6 classes. The most frequent 3 groups are named Head1 to Head3. The 4 groups with medium frequency are named Mid1 to Mid4, and the rest 3 least frequent groups are named Tail1 to Tail3 in order. Per-class error rates on the NTU 60-LT/NTU 120-LT datasets are plotted. The results verify that the proposed components effectively improve the accuracy of tail classes and that the proposed method can simultaneously maintain the performance of head classes well.

On the other hand, we compare the proposed method with other related methods, such as Focal loss [56], CB loss [17], LDAM-DRW [45], and Equalization loss [20]. The experiments are conducted on the X-sub benchmark of NTU 60-LT and NTU 120-LT. We also aggregate the classes into ten groups based on the label frequency and further integrate the results into the head, middle, and tail parts for ease of comparison. The experimental results presented in Fig. 7 reveal a common trend across different methods, where performance is better for head classes and worse for middle and tail classes. This can be attributed to the fact that the head class is typically more diverse in terms of available samples, resulting in more consistent recognition performance. Additionally, compared to the baseline CE loss, other methods such as Focal loss, CB loss, and LDAM-DRW show improvements in reducing the error rates for middle and tail classes, while EQL is relatively less effective. Finally, in comparison with other long-tailed methods, the proposed method can learn balanced representations of the tail classes more effectively, which leads to a significant reduction in the error rates of tail classes. The proposed method can maintain the performance of head classes as well.