Part-aware Prototypical Graph Network for One-shot Skeleton-based Action Recognition

During the past decade, deep learning methods have achieved significant progresses in skeleton-based action recognition. Early approaches exploited RNNs [6, 17, 11, 9, 29] for sequential modelling and CNNs [8, 7, 12] for pseudo-image analysis. A milestone of the GCN-based method is ST-GCN [27], which defines a sparse connected spatial-temporal graph that both considers natural human body structure and temporal motion dependencies in space-time domain. Since then, a large variants of most recent works adopt the GCNs for skeleton-based action recognition [20, 19, 31, 22, 5, 13, 28, 4, 2, 3]. However, most of existing skeleton-based action recognition methods are developed for strong settings with many shots during training. Therefore, they cannot be easily adopted in our one-shot setting which focuses on meta-level knowledge learning.
Partial body analysis. There have been some works focusing on modeling the body parts to enhance local contexts. For example, MV-IGNet [26] develops a graph learning framework to enhance the spatial context modelling capability, where the multi-level spatial topologies are constructed for feature extraction. In PA-ResGCN [22], a part-wise attention module is adopted for feature enhancement. However, we address the problem under one-shot setting, which focuses on learning meta-knowledge of composing action prototypes from shared parts. To this end, we generate part-aware prototypes using a hypothesis-and-select strategy. We propose to construct a rich set of part graphs using a set of rules and then utilise a class-agnostic attention fusion module to select informative parts for part-aware prototype generation. By contrast, prior works mainly focus on learning a representation where both body-level and part-level representations are learned simultaneously for recognition, which require abundant data in training and typically perform poorly in the data-scarce setting.

One-shot skeleton based action recognition is a relatively new task in the community and only few works tackle this problem. Action-Part Semantic Relevance-aware (APSR) framework [10] adopts the semantic relevance between each body part and each action class at the distributed word embedding space. Single Level Deep Metric Learning [15] and Skeleton-DML [14] convert the original skeletons into images, extract features using CNNs and apply multi-similarity miner losses. The most recent work JEANIE [25] encodes the 3D body joints into temporal blocks with GNNs and then simultaneously perform temporal and view-point alignment of query-support in the meta-learning regime. However, none of above works adopt the part-aware prototypes for the few-shot skeleton-based action recognition.

We consider the problem of one-shot skeleton-based action recognition, which aims to classify skeleton sequences from only one labelled sample per class. To this end, we adopt a meta-learning strategy [21] that builds a meta learner $\mathcal{M}$ to resolve a family of classification tasks(also called episodes) $\mathcal{T}=\{T\}$ sampled from an underlying task distribution $P_{\mathcal{T}}$.

Formally, each meta classification task $T$, consists of a support set $S$ with labeled skeleton samples and a set of query skeleton sequences $Q$. In the $C$-way one-shot setup, the support set $S=\{(\mathbf{x}^{s}_{c},c)|c\in\mathcal{C}_{T}\}$, where $\mathbf{x}^{s}_{c}$ indicates the skeleton sequence, $c$ indicates the action label, $C_{T}$ is the subset of class sets for the task $T$ and $|C_{T}|=C$. The query set $Q=\{(\mathbf{x}_{j}^{q},c_{j}^{q})\}$, where $\mathbf{x}_{j}^{q}$ is the query skeleton sequence, and $c_{j}^{q}\in\mathcal{C}_{T}$ is the corresponding label which is known during training but unknown during testing.

We introduce a meta training set $\mathcal{T}_{train}={\{(\mathcal{S}_{n},\mathcal{Q}_{n})\}}^{|\mathcal{T}_{train}|}_{n=1}$ over the training class(also called base class) set $C^{train}$. The meta learner $\mathcal{M}$ is therefore trained episodically on the tasks $\mathcal{T}_{train}$ and is able to encode the knowledge on how to perform action recognition on different action categories across tasks. Finally, to evaluate our meta learner, we construct a test set of tasks $\mathcal{T}_{test}={\{(\mathcal{S}_{m},\mathcal{Q}_{m})\}}^{|\mathcal{T}_{test}|}_{m=1}$, where the test class(also called novel class) set $\mathcal{C}^{test}$ is non-overlapped with $\mathcal{C}^{train}$.

The meta classification task $T$ is tackled by learning the distance between two skeleton sequences. Concretely, given a query sample $\mathbf{x}^{q}$ and a support sample $\mathbf{x}^{s}$, the goal is to learn a model $D$ that can measure the distance between $\mathbf{x}^{q}$ and $\mathbf{x}^{s}$. Formally,

To achieve this, we decompose the goal into three steps. Firstly, a cascaded graph embedding network $\mathcal{F}_{embed}$ is employed to transform the raw inputs into multiple part-based representations.

where $\Gamma_{k}\in\mathbb{R}^{d}$, $d$ is the dimension of the feature. Then we adopt a part fusion module $\mathcal{F}_{fuse}$ to fuse the part-based representations and generate part-aware prototypical embeddings:

We separately generate embeddings for support sample and query sample. Finally, we exploit a distance function $d(.,.)$ on the query embedding and support embedding to calculate their distances. Below we will introduce our model architecture design for the embedding and fusion module.

An overview of our framework is shown in Figure 2. We implement $\mathcal{F}_{embed}$, $\mathcal{F}_{fuse}$ and distance calculation process with three modules: cascaded embedding module, attentional part fusion module and matching module. Concretely, given a skeleton sequence $\mathbf{x}\in\mathbb{R}^{D\times T\times V}$, where $D\in\{2,3\}$ denotes the 2D or 3D coordinates of joints, $T$ indicates the sequence length, and $V$ represents the number of semantic joints. The goal of the cascaded embedding module $\mathcal{F}_{embed}$ is to transform a raw sequence $\mathbf{x}$ into multiple part-based representations. The attentional part fusion module highlights important parts and generate part-aware prototypical representations. And the matching module aims to exploit the prototypical representations to perform classification. Below, we introduce the details of our network design.

For a c-way, 1-shot setting, we randomly sample c classes with each class containing 1 example as the support set. We construct the query set to have c examples, where each unlabelled sample in the query set belongs to one of the c classes. Thus each task has a total of 2c examples for training. We report the accuracy by adopting the evaluation protocols described above in the following experiments. For NTU-RGB+D 120, we adopt the data-preprocessing procedure as introduced in [30]. For NW-UCLA and , we adopt the same data-preprocessing in [5]. During training and testing, the maximum frame number is set to $T=64$.

In our experiments, we adopt $L_{body}=5$ layers of ST-GCN[27] as the backbone network for body-level representation learning. Then, we generate K part graphs, each with $V^{\prime}$ node representation and feed into $L_{part}=5$ layers of non-local blocks for part-level representation modelling. Finally, an attention-based mechanism is adopted to fuse part representations.

We optimised our model with Stochastic gradient descent(SGD), with a starting learning rate of 0.1 and decaying at 100 and 200 epochs by 0.1. We report the performance at 300 epochs. All experiments are conducted using PyTorch deep learning framework with 4 Tesla V100 GPUs.

We evaluate our proposed method on three public benchmarks under one-shot setting.

On NTU RGB+D 120 dataset, we perform experiments on five different experimental class reduction ratios, from 20 to 100. For a fair comparison, an identical evaluation protocol is adopted for all listed methods. The results are shown in Table I. Specifically, the proposed model obtains an accuracy of 65.6% with 100 training classes, significantly outperforming state-of-the-art methods [10, 15, 14, 25] based on the one-shot learning by 8.6% and exceeding the APSR method [10] by a large margin of 10.3%. As the number of training classes decreases, the gaps in the accuracy of the listed methods compared to our method gradually narrow but there is still a gap of at least about 5% (e.g., the JEANIE [25] method achieves 38.5% accuracy using 20 training classes, while our method reaches 43.0% accuracy). Furthermore, it presents four variants based on the ProtoNet [21] as the baseline methods, including ST-GCN [27], MV-IGNet[26], MS-G3D [13] and CTR-GCN [3]. As shown in Table I, the variant of ProtoNet [21]+ST-GCN [27] gives better results than the other three variants in most cases. It suggests that directly adopting advanced graph networks in many-shot setting can not bring better generalisation in one-shot setting, where the key is to learn novel action classes given knowledge from known base classes. In contrast, our part-aware prototype modelling exploits part representations for metric learning and is more effective.

We also evaluate the proposed part-aware architecture on NW-UCLA [24] to verify its effectiveness and generalisation for one-shot skeleton-based action recognition. Since there is no previous work on one-shot learning based on this dataset, we thus re-implement the open-sourced models, SL-DML [15] and Skeleton-DML [14] and the comparisons are mainly made between the proposed model and four baseline models. Table II shows the results on NW-UCLA dataset. Our method performs the best among all listed methods, followed by the ProtoNet [21]+MS-G3D [13] with 2.1% lower accuracy.

In this subsection, we perform ablation study to evaluate the effectiveness of our proposed modules and attentional fusion strategy. Except for the experiments in Incremental ablation study, all the following experiments are performed by modifying the target component based on the full model. All the experiments are conducted on the NTU-RGB+D 120 dataset, with 100 classes for training, 20 classes for testing.

In Figure 5, we visualise the attention prediction of attentional part fusion module. For each skeleton, we show the top-3 important part graphs. We can observe that, for different actions, our attention block can select different information parts, while for the same action, attention is similar. This demonstrates that our attention mechanism is class-agnostic.

In Figure 6, we compare our model with baseline global modelling method on 8 novel fine-grained actions which highly rely on hands. We can observe that our method outperforms the baseline ST-GCN on each class, demonstrating the effectiveness of our local part-level modelling.