Fine-Grained Side Information Guided Dual-Prompts for
Zero-Shot Skeleton Action Recognition

(1) Topology-based Multi-part Skeleton Generation. According to the knowledge of human topology structure, we decompose the skeleton into several joint groups with different fine-grained levels, keeping the same with (Xiang et al., 2023). Specifically, a two-part strategy means dividing the skeleton into the upper body and lower body, and four-part and six-part strategies gradually decompose the upper body and lower body into finer partitions, respectively. Afterward, we obtain the fine-grained skeleton representation $f_{v}=\{f_{v}^{e}\in\mathbb{R}^{\widehat{T}\times\widehat{V}\times C}\}_{e=1}^{K}$ with the spatial topology and temporal continuity by the visual feature extractor $\phi(\cdot)$, as opposed to the previous studies pooling the spatial-temporal dimensions $\widehat{T}\times\widehat{V}$ for global feature $f_{v}\in\mathbb{R}^{C}$. The $\widehat{T}$ is the down-sampled temporal dimension, $\widehat{V}$ is the joint group of the $e$-th part, and $K$ is the number of the skeleton parts ($K=\{2,4,6\}$).

(2) One-to-One Side Information Generation. Compared to the previous studies that use the action category names or their enriched descriptions as the global semantics, we generate the one-to-one spatial-temporal descriptions of the skeleton elements as the extra side information at the fine-grained level. Specifically, we utilize the GPT-3.5 (Brown et al., 2020) as the expert knowledge base to generate side information by providing appropriate questions. For example, we obtain the side information of head description in action ”drink water” with the question ”please describe the [head] actions simply when people [drink water]”: head tilts back slightly. This way, we can depend on the abovementioned skeleton partition strategies and action category names to generate all actions’ fine-grained multi-part skeleton side information. Afterward, we utilize the pre-trained language model as the semantic feature extractor $\psi(\cdot)$ to extract the embeddings of the category name and side information, which can be described as $f_{cn}$ and $f_{si}=\{f_{si}^{e}\}_{e=1}^{K}$.

(1) Skeleton Representation Network Stream. The skeleton sequence comprises the human pose coordinates, lacking detailed descriptions of the surrounding environment and human appearance compared to RGB-D videos. Thus, several action categories are highly similar if we only observe the skeleton sequence. With these realities, it is challenging to distinguish highly similar actions such as ”reading” and ”writing” because only the hand movements are slightly different. Therefore, we employ a cross-attention mechanism with the proposed visual-attribute prompt to improve the intra-class compactness in skeleton space by fully exploring the spatial representations and capturing the long-term dependencies of the skeleton actions.

(2) Semantic Embedding Network Stream. Human actions may have similar movements in several body parts, resulting in the same specific side information. For example, ”brush teeth” and ”cheer up” have the same side information of head movements: head tilts forward slightly. These identical side information belonging to different action categories can make aligning nonhomogeneous skeleton sequences in the right direction challenging. In other words, these identical side information belonging to various action categories are inseparable in semantic space. To solve this problem, we introduce learnable semantic-part prompts to supplement the category-specific knowledge for side information.

(1) Multi-Part Cross-Entropy Loss. To align the skeleton space with the semantic space at the fine-grained level, the initial step is to bring the pairs of part skeleton-semantic closer together. Therefore, we achieve this by calculating the dot product between the skeleton part representation $F_{v}^{e}$ and the fine-grained side information embeddings $F_{si}^{e}$. The multi-part cross-entropy loss can be defined as follows:

where $B$ is the batch size.

(2) Semantic Cross-Entropy Loss. To ensure the knowledge that the semantic-part prompt learned is the corresponding category, we suggest promoting the embedding of the augmented side information to have the highest compatibility with its corresponding category name embedding. By doing this, the augmented side information is near its category semantic center, prompting the inter-class separability with each other. The $L_{SCE}$ is defined as:

(3) Global Cross-Entropy Loss. Besides the alignment at the fine-grained level, we propose the $L_{GCE}$ to align the skeleton and semantic space at the global level. The optimization operation can be described as follows:

(1) NTU RGB+D 60 Dataset (Shahroudy et al., 2016). It contains 56880 action samples with multi-modalities, including RGB, depth map, skeleton, and infrared (IR) video, performed by 40 subjects and classified into 60 action categories. For skeleton modality, each sample consists of a maximum of two people, each comprising 3D coordinates of 25 joints. Two official benchmarks are applied: (1) Cross-subject (Xsub): the training set contains 20 subjects, and the remaining subjects are used for testing; (2) Cross-view (Xview): the training set contains view2 and view3, while the view1 make up the test set.

(2) NTU RGB+D 120 Dataset (Liu et al., 2019a). It is the extension of the NTU RGB+D 60, which contains 114,480 action samples of 120 action categories and has the same multi-modalities. Similarly, it also split the dataset into two official benchmarks: (1) Cross-subject (Xsub): the training set contains 53 subjects and the rest of the subjects’ data for testing; (2) Cross-setup (Xset): even camera IDs belong to the training and the testing data consists odd IDs.

(3) PKU-MMD Dataset (Chunhui et al., 2017). It contains two phases for action recognition with increasing difficulty, which covers the same multi-modalities as the NTU dataset. The actions are collected into 51 action categories, and almost 20000 instances are included. Two official experimental settings are proposed: (1) Cross-subject (Xsub): 57 subjects belong to the training set and 9 subjects for the testing set; (2) Cross-view (Xview): the middle and right views are chosen for the training set, and the left is for the testing set.

(1) Zero-Shot Learning. In this study, we evaluate the effectiveness of the proposed STAR method compared with other state-of-the-art methods on three datasets in the ZSL setting. As shown in Table 2 and Table 3, our STAR method achieves the best accuracy on NTU series datasets with different unknown-known split strategies. For the cross-subject task, STAR outperforms SMIE by 3.5% and 3.6% in the 55/5 split and 48/12 split of the NTU 60, respectively. Meanwhile, STAR appears to have a similar trend of improvement in NTU 120, indicating that the STAR method can capture the spatial-temporal characteristics of actions at the fine-grained level between cross-subjects. Our STAR method achieves high accuracy for the cross-view task in NTU 60, with 81.6% and 42.5% in different split strategies. Additionally, STAR also achieves state-of-the-art results in the cross-setup task of the NTU 120. These results demonstrate the robustness of our method in dealing with view and setup changes, showing the remarkable capacity to generalize models into unseen views while maintaining accuracy for unknown categories. Lastly, we also explore the different scenarios (PKU-MMD II), as shown in Table 4. We can find that our method dramatically improves accuracy across various tasks and split strategies. Notably, our proposed method can effectively enhance the recognition of unknown categories by learning the fine-grained relationship between skeleton and semantic spaces rather than global alignment.

(2) Generalized Zero-Shot Learning. Here, we compare our proposed STAR with other state-of-the-art methods on three datasets in the GZSL setting. To make a fair comparison with our alignment method, we employ the calibrated stacking method in Section 3.5 as the gating module of all methods to classify known and unknown categories. As shown in Table 2, Table 3, and Table 4, our proposed STAR achieves the highest accuracy across various tasks and split strategies. Additionally, our method achieves a balanced accuracy regarding known and unknown categories. For instance, our STAR can obtain better performance on known categories and unknown categories as 59.3% and 59.5% for the cross-setup task in NTU 120 with 110/10 split dataset, and thus result in the harmonic mean as 59.4%. In contrast, JPoSE (Wray et al., 2019) has a considerable margin (19.2%) between their accuracy of known and unknown categories in the same dataset, resulting in poor performance on the harmonic mean. Therefore, this demonstrates that our STAR can learn the discriminant and transferable representations at the fine-grained level, thereby alleviating the problems of domain bias.

(1) Influence of Known-Unknown Categories Settings. Different known-unknown category settings may affect the method’s performance. We optimize the experiment procedure and randomly resplit categories into three known-unknown settings to evaluate their robustness. These three split settings are non-overlap, and we compute their average results in the ZSL and GZSL to eliminate the variance, as shown in Table 5. Our proposed STAR performs better than all previous methods in the ZSL, improving by 13.3% and 6.4%, respectively. It demonstrates that STAR is robust in the various known-unknown category settings, and learning prior action knowledge from body parts at the fine-grained level can increase generalization performance to recognize unknown actions. In the GZSL, the STAR outperforms the second by 12.9% and 14.4% (relatively 25.6% and 27.3% ), showing that the proposed STAR alleviates the issues of domain bias effectively.

(2) Influence of Fine-grained Levels. Under the baseline settings, we explore how different topology-based partition strategies at various fine-grained levels affect the proposed STAR. The strategies we tested include two-part (upper body, lower body), four-part (head, hand-arm, hip, leg-foot), and six-part (head, hand, arm, hip, leg, foot). According to the results presented in Table 6, the six-part partition strategy performs the best on two datasets. This suggests aligning the skeleton and semantic space at the fine-grained level is necessary. Meanwhile, the higher the fine-grained level, the better the recognition performance.

(3) Influence of Components. In Table 7, we assess the effectiveness of modules, prompts, and loss functions in the STAR. We can observe that the STAR performs significantly worse without the attention-based mechanism (AM), resulting in a decrease of nearly 2.7%/4.9% and 7.2%/4.5% on two datasets. This indicates that the AM can capture skeleton actions’ spatial characteristics and long-term temporal dependency. Additionally, dual-prompts (VPP and SPP) are crucial in improving intra-class compactness in skeleton space and prompting inter-class separability in semantic space. If we remove them, approximately 3.0%/8.5% and 1.9%/5.8% drop on the NTU RGB+D 60 dataset, respectively. Moreover, the multi-part cross-entropy mechanism can effectively align the skeleton space and semantic space at a fine-grained level to transfer knowledge, resulting in improvements of 3.2%/8.9% and 17.7%/7.9% on two datasets, respectively. The semantic cross-entropy and global cross-entropy constraints ensure the optimization direction is towards the action category name as the center, effectively prompting the harmonic mean in the GZSL and mitigating the bias problem.

(4) Influence of Hyper-parameters. We investigate the influence of the $\gamma$ and $m$ in STAR by selecting a range of values. Our findings, illustrated in Fig. 3(a) - Fig. 3(b), showed that increasing the calibration factor value leads to a decrease in the accuracy of known categories and an increase in the accuracy of unknown categories, enabling us to determine the optimal calibration factor value for GZSL. This calibration factor effectively balances the accuracy of known and unknown categories, achieving the trade-off between them to alleviate the bias problem. Additionally, our experiments with motion attribute $m$ in Fig. 3(c) - Fig. 3(d) show that the recognition accuracy remains stable with the number of attributes increases, indicating the robustness of proposed prompts.

(1) t-SNE Visualizations of Spaces. As shown in Fig. 4, we plot the t-SNE visualization of the skeleton and semantic spaces for known and unknown categories on the NTU RGB+D 60 dataset. Fig. 4(a) - Fig. 4(d) represent the distribution of skeleton features with the visual-attribute prompt or not, showing the capacity to improve the intra-class compactness of skeleton space over the known and unknown categories. Meanwhile, we find that the area of the triangle in Fig. 4(e) - Fig. 4(h) increases with the addition of the semantic-part prompt, demonstrating that the semantic-part prompt can improve the inter-class separability of semantic space. Besides, the optimized part descriptions are pulled closer to their semantic center (category name) for better alignment with the decomposed human skeleton.

(2) Visualization of the Confusion Matrices. Here, we compare the capacity of STAR with SMIE (Zhou et al., 2023) for distinguishing high-similar skeleton actions that never appeared before. We draw the confusion matrices of two methods for unknown categories on the cross-subject task of the NTU RGB+D 60 dataset under the 55/5 split setting, as shown in Fig. 5. It shows that SMIE is deficient in recognizing skeleton actions with subtle differences, such as writing and reading. These skeleton actions throughout the body only have differences in the hand because the appearance and environment information are dropped. In contrast, our method STAR can distinguish these highly similar skeleton actions based on the prior knowledge learned from the known categories, showing the necessity of aligning the skeleton and semantic spaces at the fine-grained level.