Learning from Temporal Spatial Cubism for Cross-Dataset Skeleton-based Action Recognition

Skeleton-based action recognition has attracted growing attention in the realm of computer vision and a variety of methods have been proposed over the past decades. For a detailed survey we refer the reader to (Han et al., 2017; Liu et al., 2020a; Wang et al., 2018), while here we provide a brief literature review. The early works on skeleton-based action recognition are based on hand-crafted features (Vemulapalli et al., 2014; Weng et al., 2017; Koniusz et al., 2016; Wang et al., 2016b), while recent approaches are devised by designing deep neural networks (DNNs) like convolutional neural networks (CNNs) (Liu et al., 2017; Tang et al., 2018; Li et al., 2018) and recurrent neural networks (RNNs) (Song et al., 2017; Zhang et al., 2017, 2018). In order to better capture the relationship of different joints in the spatial domain or dependency of different frames in the temporal domain, a number of works utilized the attention mechanisms (Song et al., 2017; Zhang et al., 2018; Si et al., 2019) and graph neural networks (GNNs) (Yan et al., 2018; Shi et al., 2019a, b; Li et al., 2019; Si et al., 2019) more recently. Besides, there are various works using both skeleton joints and RGB videos as inputs for action recognition (Choutas et al., 2018; Zolfaghari et al., 2017; Verma et al., 2020). For example, Verma et al. (Verma et al., 2020) design two deep neural networks (DNNs) models for the multi-modal inputs respectively, and use a weight product model (WPM) to fuse the softmax scores obtained from the two DNNs. Different from these works which deal with the input videos from the same dataset during training and testing phases, we study a more practical and challenging UDA setting to deal with the samples across different datasets.

Reducing the domain shift between the source and target datasets is the core of UDA. In the past few years, a series of models have been built upon deep neural networks for learning domain-independent representations, which show more promising results than early methods based on hand-crafted features (Huang et al., 2006; Pan et al., 2011; Gong et al., 2013). Among these, one representative strategy is to design adaptation layers for aligning different distributions (Long et al., 2017), and another popular scheme is to include a domain discriminator sub-network for adversarial learning (Ganin et al., 2016; Long et al., 2017; Saito et al., 2018; Long et al., 2018). More recently, there are several attempts on leveraging self-supervised learning for UDA (Carlucci et al., 2019; Sun et al., 2019). Under a multi-task learning paradigm, they optimized the model with the supervision from the source domain, and the auxiliary self-supervision from both source and target domains. Motivated by the success of these methods in the image domain, we move a further step in the field of skeleton-based action recognition. Note that our exploration is non-trivial since the intrinsic structure of the skeleton-based video is quite different from image, and further generalization and adaptation are required.

Compared with image-based domain adaptation, video-based domain adaptation is a seldom-explored field. In the literature, a few works have been proposed for RGB videos, by foreground-weighted histogram decomposition (Sultani and Saleemi, 2014), or performing adversarial learning on the video features (Jamal et al., 2018). More recently, Chen et al. (Chen et al., 2019a) devised TA³N by introducing a temporal relation module and domain attention mechanism. For skeleton-based video, Tas et al. (Tas and Koniusz, 2018) and Lin et al. (Lin et al., 2020) study the supervised domain adaptation and transfer learning settings, where the action labels of the target dataset are required at the training or fine-tuning stages respectively. The most relevant work to ours is GINs (Tang et al., 2020c), which also studied the problem of cross-dataset skeleton-based action recognition under the UDA setting. In comparison, we proposed a setting with three datasets with larger-scale, and devised a self-supervised learning framework rather the adversarial-based method used in (Tang et al., 2020c). Experimental results also show the advantage of our method.

The paradigm of self-supervised learning is to design auxiliary task(s) with the supervision of the data itself, for example, predicting spatial context (Doersch et al., 2015) or image rotation (Gidaris et al., 2018), solving jigsaw puzzles (Noroozi and Favaro, 2016) and many others (Zhang et al., 2016; Larsson et al., 2017; Pathak et al., 2016; Doersch and Zisserman, 2017; He et al., 2020; Chen et al., 2020). There have been a number of self-supervised learning methods for RGB videos, according to the information of ordering (Misra et al., 2016; Fernando et al., 2017; Lee et al., 2017), geometry (Gan et al., 2018), correspondence (Wang et al., 2019a; Dwibedi et al., 2019; Lai et al., 2020), motion and appearance statistics (Wang et al., 2019b) or spatio-temporal cubic puzzles (Kim et al., 2019). Compared with these works, besides temporal ordering, we further explore the relationship of different human body parts for skeleton-based videos by learning from spatial Cubism, and leverage the advantage of self-supervised learning to seek a better alignment between the source and target domains.

We use $\mathcal{D}_{s}=\{(\boldsymbol{x}_{i}^{s},y_{i}^{s})\}_{i=1}^{n_{s}}$ to denote a source domain, which contains skeleton-based action videos $\{\boldsymbol{x}_{i}^{s}\}_{i=1}^{n_{s}}$ and their action labels $\{y_{i}^{s}\}_{i=1}^{n_{s}}$. Here $i$ denotes the index of the $i$-th video, $n_{s}$ means the number of videos, and the subscript of $n_{s}$ denotes the source domain. Similarly, the target domain is defined as $\mathcal{D}_{t}=\{(\boldsymbol{x}_{j}^{t}\}_{j=1}^{n_{t}}$, where the action labels are unavailable during the network optimization but can used for the performance evaluation. Since the videos in the source and target domains are from different datasets, they correspond to two different joint distributions as $P(\boldsymbol{X}_{s},\boldsymbol{Y}_{s})$ and $Q(\boldsymbol{X}_{t},\boldsymbol{Y}_{t})$. The training should be performed on the source domain with the action labels, and a split of target domain data where the action labels are unavailable. The testing process is based on the other split of the target domain data which is invisible during the training phase. See section 4 for more details. There is a more challenging cross-dataset setting which assumes that data from target domain are totally unavailable. The experimental results under this setting are introduced in the section 4.6.

The motivation of this work is to leverage self-supervised learning and Cubism transformation to reduce the domain shift between the skeleton-based action videos from the source and target datasets. The concept “Cubism” here is originally an art genre from the early 20th century, which breaks and reassembles the objects to convey a greater context. Inspired by this idea and the progress in self-supervised learning (Fernando et al., 2017; Lee et al., 2017; Misra et al., 2016), we design two auxiliary pretext tasks, named as temporal Cubism (section 3.3) and spatial Cubism (section 3.4), for skeleton-based action recognition under a new cross-dataset scenario. Accordingly, we devise two networks as the Tem-Cub Network and Spa-Cub Network, where “Tem-Cub” and “Spa-Cub” are abbreviations of “temporal Cubism” and “spatial Cubism”. During the training phase, each network is optimized based on one of the self-supervised tasks and the main prediction task jointly. At the inference period, the final result is obtained by fusing the prediction scores of two networks. We elaborate on each stage of our pipeline in detail as follows.

Fig. 3 illustrates our proposed strategy of learning from temporal Cubism. The term temporal Cubism here means we shuffle a video in temporal dimension and reorganize them in a new frame order. Mathematically, we organize each skeleton-based action video as a representation with the size of $F\times K\times D$, where $F$ denotes the number of the frames, $K$ is the number of the joints, and D represents the dimension of joint coordinates. Given a video sample $\boldsymbol{x}$, we first divide it into $N$ segments uniformly in the temporal domain as $\boldsymbol{x}=\begin{bmatrix}(\boldsymbol{x}^{(1)})^{T}&(\boldsymbol{x}^{(2)})^{T}&\dots&(\boldsymbol{x}^{(N)})^{T}\end{bmatrix}^{T}$, where we choose $N=3$ in this paper empirically. Then a new video $\boldsymbol{x}_{tem}$ with corresponding permutation label $l_{tem}$ in temporal domain is acquired by permuting the segments. This transformation $\varphi_{tem}$ could be presented by a partitioned permutation matrix $\boldsymbol{P}_{tem}$ as:

There is only one identity matrix on each row and on each column, and the remaining elements are zero matrices. For example, if the permutation is to exchange the order of the first and the third segments, the transformation can be written as below:

In this way, we build the permuted videos and form the augmented source and target datasets which are presented as follows:

Here $n_{s}^{\prime}$ and $n_{t}^{\prime}$ denote the overall number of videos in the augmented source datasets and the augmented target datasets. For the $i$-th video, $y_{i}$ and $l_{tem,i}$ represent the action label and permutation label of temporal Cubism respectively. Based on the augmented datasets, we design an auxiliary classification task in the temporal domain, which guides the network to learn to recognize the temporal permutation. During training, the ordered and permuted video samples are packed into batches with a certain ratio which is dependent on a hyper-parameter $p_{t}$ indicating the percentage of the ordered samples in a batch. This hyper-parameter will be studied in section 4.4. Moreover, the permutation way is chosen with equal probability so that transformed videos with different ordering labels are of an identical proportion. The mixed batches are fed into a CNN-based backbone (we detail it in section 4.3) followed by two parallel full-connected classifiers. $f_{cls}$ takes the features of the ordered and disordered samples from the source domain and predicts the action classes while $f_{tem}$ targets to recognize the ordering label for the samples from both source and target domains. Two kinds of losses are computed and combined after the classifiers to optimize the network. Comprising two parts of losses, the total loss $\textbf{{J}}_{tem\_total}$ could be formalized as:

Here we adopt the cross-entropy loss for $J_{c}$ and $J_{tem}$. $f_{cls}$ and $f_{tem}$ are the softmax scores of action classes and temporal permutation classes. $\theta_{b},\theta_{cls}$ and $\theta_{tem}$ denote the trainable parameters of the backbone, action recognition fc layer and temporal permutation recognition fc layer respectively. Here $\lambda_{t}$ is the hyper-parameters to balance the effects of the losses of the main task and self-supervision learning, which will be studied in section 4.4 as well.

As shown in Fig. 4, we design a new self-supervised classification task based on the spatial Cubism among the different body parts. Specifically, for a skeleton-based action video $\boldsymbol{x}$ defined in the last subsection, we organize the body parts according to the following ordered list: $\boldsymbol{x}=\begin{bmatrix}\boldsymbol{x}^{(t)},\boldsymbol{x}^{(la)},\boldsymbol{x}^{(ra)},\boldsymbol{x}^{(ll)},\boldsymbol{x}^{(rl)}\end{bmatrix}$. The five blocks are corresponding to the data of trunk, left arm, right arm, left leg and right leg respectively. Similar to the temporal Cubism, we can obtain a new sample by performing spatial transformation $\varphi_{spa}$ with another permutation matrix $\boldsymbol{P}_{spa}$ as:

Here we design two concrete instances for $\boldsymbol{P}_{spa}$ as $\boldsymbol{P}_{a}$ and $\boldsymbol{P}_{\ell}$ to swap the coordinates of the joints of the arms and legs respectively:

Through these transformations, the skeleton-based video would convey the screwy actions which refer to the spatial Cubism. By learning to discover these, the network would have a better generalization ability on the spatial domain. Similar to that of the temporal Cubism, we construct the augmented source dataset $\mathcal{D}^{\prime\prime}_{s}$ and target dataset $\mathcal{D}^{\prime\prime}_{t}$ as follows:

We introduce a hyper-parameter $p_{s}$ to indicate the percentage of the ordered samples in a batch during the training phase. The total loss $\textbf{{J}}_{spa\_total}$, in this case, could be formalized as:

The variables in Equation (7) and (8) have the similar definitions with those in Equation (3) and (3.3). We present a mathematical algorithm of our method in Algorithm 1.

In order to further boost performance, we explore several approaches to couple the temporal and spatial Cubism transforms. One is to apply the two kinds of transforms simultaneously and therefore divide the videos into finer-grained atoms (see section 4.4 for details). However, this results in a more complex task, which might bring more difficulty for optimizing the network and more cost in data pre-processing.

Though feature-level fusion is a common two-stream fusion strategy, we do not apply it in this paper. This is because spatial and temporal streams implement different auxiliary tasks, and feature-level fusion will make it much more difficult to recognize the ordering label. Actually, as explored by several previous works (Simonyan and Zisserman, 2014; Wang et al., 2016a), it is more effective and efficient to separately deal with the temporal and spatial information and combine them after the network. Hence, we explore several approaches to fuse softmax scores from the temporal and spatial streams during the inference stage, e.g., Weighted Arithmetic Mean (WAM), Weighted Root Squared Mean (WRSM), Weighted Geometric Mean (WGM) and Max Pooling (MP). The experimental results and more details are shown in Table 11 in later section.

To our best knowledge, there are very few benchmarks for cross-dataset skeleton-based action recognition. Although the recent work (Tang et al., 2020c) has proposed two benchmarks for this problem, they mainly have two drawbacks. The first is the scales of these datasets are relatively small, and the second is that it adopts a “test-time training” strategy (Sun et al., 2020), which utilizes the test data (without using their action labels) during training. This might be not practical in some real-world scenarios.

To address these issues, we define two groups of experiment settings crossing three large-scale datasets in this paper: NTU RGB+D (Shahroudy et al., 2016), PKU-MMD (Liu et al., 2020b), and Kinetics (Carreira and Zisserman, 2017). We present a comparison of the proposed settings with the previous work (Tang et al., 2020c) in Table 1. We detail the corresponding datasets and the experimental settings as follows.

In order to make a better evaluation of our method, we also conduct experiments on the SBU Kinect Interaction dataset (SBU) (Yun et al., 2012), Online RGBD Action dataset (ORGBD) (Yu et al., 2014) and MSRDaily Activity3D dataset (MSRDA3D) (Wang et al., 2012), following the same settings proposed in the previous work (Tang et al., 2020c). The experimental results and analysis are described in detail as below.

In the following section, we first conduct experiments and acquire the results on Source Only and Target Only. Source Only indicates a baseline method which trains a model in the source domain, and directly evaluates the testing data on the target domain without supervision. Target Only denotes to utilize the ground-truth action labels in the target domain for training, which provides an upper bound for this problem. Besides, because there are very few models designed for skeleton-based action recognition under the UDA setting, we compare our model with some image-based UDA models (i.e., MMD, DANN, JAN, CDAN, BSP, MCC) and a RGB video-based model (i.e., TA³N). We replace those models’ backbone with HCN and apply the same experimental setting with our model for fair comparison. Specifically, for TA³N, besides replacing the feature extractor with HCN, we add a fully-connected layer between HCN and the spatial-temporal adversarial module in TA³N to make them compatible. Moreover, we also compare our method with GINs, which is the most related work on cross-dataset skeleton-based action recognition.

In our paper, there are mainly three kinds of information to be utilized for cross-dataset transfer. They are temporal information (T), spatial information (S) and their combination (TS), which are used for our Tem-Cub Network, Spa-Cub Network and TS-Cub Network respectively. For the compared methods, most of them perform transfer based on the temporal-spatial feature extracted by HCN backbone (TS), except the GINs (Tang et al., 2020c) focused on the relation of different joints at the spatial domain (S).

We conduct experiments on a system with the Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.00Ghz. We implement our method with the PyTorch toolbox and train the model on Nvidia GTX 1080 Ti GPU. The training epoch $\Gamma$ is set to 400 for both Tem-Cub and Spa-Cub. We adopt the HCN (Li et al., 2018) as the backbone because of its effectiveness and efficiency. In order to ameliorate the performance, we made two modifications. The PReLU (He et al., 2015) is used as the non-linear activation function instead of the commonly used ReLU and a rotation pre-processing is applied before the networks to eliminate the difficulty for bridging the domain gap induced by the videos captured from different views. specifically, we rotate the skeleton around the z axis to make the connection of the joints right shoulder and left shoulder parallel with the x axis and the connection of spine base and spine parallel with y axis.

We display the compared confusion matrices in the bottom row of Fig. 6. Our method could recognize the action of reading book well, but would be sometimes confused by the action of eating and using laptop.

We make further exploration about testing our proposed TS-Cub under a more challenging cross-dataset setting, which assumes that data from target domain are totally unavailable during the training period. We detail this unsupervised domain adaptation setting in Table 12. During the training phase, the permuted video samples from source domain are delivered into the network along with the ordering data. The final losses are composed of the main classification task and the auxiliary Cubism task.

We present the experimental results in Table 13, other compared methods in the previous section are absent because they all required the target data during training, which is not available in this setting. As shown in Table 13, our TS-Cub consistently outperforms the baseline Source Only on the six tasks, which indicates its robustness for cross-dataset skeleton-based action recognition.

Recently, Choi et al. (Choi et al., 2020) study the cross-dataset action recognition problem by combining the domain adversarial task with the clip order prediction task. Motivated by this work, we further conduct experiments to see whether our self-supervised pretext tasks are complementary with the conventional domain adversarial task. Since (Choi et al., 2020) is designed for RGB video while our work focus on skeleton-based video, we use HCN (Li et al., 2018) as the backbone similar with TA³N (Chen et al., 2019a). Then we perform temporal Cubism at raw data level as well as apply adversarial learning in feature level based on CDAN (Long et al., 2018) (denoted as “Tem-Cub (Ours) + CDAN” in Table 14). We also conduct experiment on combining our Spa-Cub with CDAN (i.e., “Tem-Cub (Ours) + CDAN”), and ensembling the results of “Tem-Cub (Ours) + CDAN” and “Spa-Cub (Ours) + CDAN” (i.e., “TS-Cub (Ours) + CDAN”).

We present the compared results in Table  14. On P$\leftrightarrow$N setting, we find the performance could be further improved by combining our approach with CDAN (Long et al., 2018), which shows the complementary characteristics of our method and adversarial approach. However, on the N$\to$K setting, we found that the performance drops slightly when combining with CDAN. This might attribute to the videos from the Kinetics are collected in the wild, and the skeleton-based inputs are obtained from 2D pose estimation algorithm rather than 3D sensor for NTU and PKU datasets. In this case, the adversarial approach (e.g., CDAN) might have more difficulty in dealing with this kind of data with more noise. In comparison, our method is more robust to generalize to this more challenging scenario.