A Central Difference Graph Convolutional Operator for Skeleton-Based Action Recognition

A graph is usually represented as $G=(V,E)$, where $V=\left\{v_{1},\ldots,v_{N}\right\}$ is a set of $N$ nodes representing joints of a skeleton and $E$ is a set of edges representing bones between joints. Given a skeleton graph, multiple layers of vanilla graph convolutions are applied to extract multi-level features. Suppose $v_{i}$ is a vertex in the graph, a vanilla graph convolution operation centred at it is expressed as:

where x and y denote an input feature map and output feature map, respectively. w is the weight function that provides a weight vector based on the given input. $R_{i}$ is the 1-distance neighboring vertexes of $v_{i}$. $l_{i}$ is a partitioning function to partition vertexes in $R_{i}$ into a fixed number of subsets, with each subset having its own convolution weight vectors. In the classic partition strategy [15], $R_{i}$ is divided into three subsets: $S_{1}$, $S_{2}$, and $S_{3}$. In specific, $S_{1}$ is $v_{i}$ itself; $S_{2}$ is the centripetal subset which contains vertexes closer to skeleton gravity center in $R_{i}$; and $S_{3}$ is the centrifugal subset which contains vertexes away from skeleton gravity center. $Z_{ij}$ denotes a normalizing term which balances the contribution of each subset to the output.

In general, there are two main steps in the vanilla graph convolution: 1) sampling neighboring vertexes of $v_{i}$ over the input feature map x; and 2) aggregating the values of neighboring vertexes and $v_{i}$ itself via weighted summation.

Central difference convolution (CDC) introduces the central-oriented local gradient features to enhance model’s discrimination capacity, which has been applied in many fields, such as face anti-spoofing[21, 22, 23, 24], remote heart rate measurement[25] and gesture recognition[26], and has achieved great results. Inspired by the CDC networks, we propose to integrate spatial gradient information into the graph convolution operator to enhance its representation and generalization capability.

The proposed central difference graph convolution also consists of two steps, i.e., sampling and aggregation. The sampling step is similar to vanilla graph convolution while the aggregation step is different. As shown in Fig. 1(a), central difference graph convolution prefers to aggregate the center-oriented gradient of sampled vertexes, which is expressed as:

When $v_{j}=v_{i}$, the gradient value is always equal to zero relative to the central location $v_{i}$ itself.

For action recognition, existing works focus on establishing dynamic or action specific connections between nodes, so as to aggregate information from either explicitly or implicitly associated nodes. However, the gradient between connected nodes is also potentially useful according to the work of CDC [21]. Hence, the combination of vanilla graph convolution and central difference graph convolution is expected to provide robust and differentiated modeling capability. Thus, the above-mentioned central difference graph convolution is generalized to:

where hyperparameter $\alpha\in[0,1]$ tradeoffs the contribution between nodes and gradient information. Eq.(3) is referred to as central difference graph convolution (CDGC).

Notice that the gradient information in CDGC is different from that carried by the commonly used bone modality. First, each bone points from its source joint to the target joint. The source joint is defined as the closer one to the center of gravity of skeleton, and the target joint is defined as the further one. Thus, the direction of each bone is fixed, as shown in Fig. 2(a). However, in CDGC, when one of two adjacent nodes is taken as the center node, the direction of the gradient vector is opposite to that obtained when the other node is taken as the center node. According to Eq.(2) and as shown in Fig. 2(b), when $v_{1}$ is the center, the gradient direction of $v_{2}$ relative to $v_{1}$ points from $v_{1}$ to $v_{2}$. When $v_{2}$ is the center, the gradient direction of $v_{1}$ relative to $v_{2}$ points from $v_{2}$ to $v_{1}$, which is opposite to the former. Thus, the information they captured is different. Second, when a connection is established between two nodes that are not connected in the nature skeleton graph, the difference between them is more prominent. If there is a coordination between hand $v_{3}$ and feet $v_{4}$ in an action, the gradient information of $v_{3}$ will be captured by the proposed CDGC when calculating the feature of $v_{4}$. However, there is no bone between the two nodes.

A vanilla graph convolution is implemented [15] as:

Here the weight vectors of multiple output channels are stacked to form the weight matrix W. A is a normalized adjacency matrix. In order to simplify the expression of the subsequent formula, we omit the non-linear activation function here. In practice, the input feature map is represented as a tensor of $(C,T,V)$ dimensions, where $C,T$ and $V$denote the number of channels, frames, and vertexes, respectively. The graph convolution is implemented by multiplying the input feature map with the normalized adjacency matrix A on the second dimension and then performing a standard 2D convolution.

Eq.(2) can be rewritten to:

In implementation, the summation of trainable parameters in front of $\textbf{x}(v_{i})$ can be obtained by summing the second dimension of the adjacency matrix. Therefore, the implementation of the central difference graph convolution can be expressed as:

where $\hat{\textbf{A}}$ is obtained by first summing the second dimension of the adjacency matrix A, i.e. $\sum_{j}\textbf{A}_{ij}$, and then broadcasting the resulting $N\times 1$ vector $\bar{\textbf{A}}$ to $N\times C$ matrix $\hat{\textbf{A}}$ with $\hat{\textbf{A}}=\bar{\textbf{A}}\cdot\textbf{1}$, with 1 being the vector of $1\times C$ dimensionality. $\odot$ represents the element-wise Hadamard product. Therefore, the implementation of CDGC is formulated as:

According to Eq.(7), CDGC can be easily implemented by a few lines of code in PyTorch [27] or TensorFlow [28].

In general, implementation of graph convolution depends on an adjacency matrix. The adjacency matrix represents explicit and implicit connections between nodes, then features of the connected nodes are aggregated. However, the GCNs based on an adjacency matrix is usually computationally expensive, and the calculation process is complex and time-consuming. Recently, an advanced graph convolution model called shift-GCN[16] is proposed. It utilizes a simple graph shift operation and point-wise convolutions to aggregate neighboring node features instead of regular graph convolutions. This greatly speeds up the training process of GCN. Our accelerated version of CDGC is also based on shift operations, and therefore is also computationally efficient. Moreover, the spatial shift operation makes the receptive field of each node cover the full skeleton graph, and the features of a node after shifting are composed of the features of all other nodes in the graph. Derived from this, we propose a simple solution to realize our CDGC: making difference between the original features and the shifted features.

In Fig.3, we take a single node $v_{i}$ (right hand) as an example and draw its original feature vector and shifted feature vector. Assuming that $x_{0ij}$ represents the feature of $j_{th}$ channel of node $v_{i}$. $x_{ij}$ represents the feature of $j_{th}$ channel of node $v_{i}$ after shift, which is from one of the channels of node $v_{k}$. Performing $x_{ij}-x_{0ij}$ obtains the gradient feature of $j_{th}$ channel for node $v_{k}$ relative to $v_{i}$. By subtracting the original features from the shifted features, the gradient features including all body nodes relative to right hand are obtained. In general, given a spatial skeleton feature map $\textbf{X}_{\textbf{0}}\in\mathbb{R}^{N\times C}$, where $N$ stands for the number of joints in skeleton graph, and $C$ is the number of channels of each node. $\textbf{X}\in\mathbb{R}^{N\times C}$ represents the feature map of the shifted spatial skeleton. Performing $\textbf{X}-\textbf{X}_{\textbf{0}}$ can obtain the gradient features of all the remaining nodes in the skeleton graph with respect to their central nodes.

In addition, in the accelerated CDGC, the connection strength between different nodes is the same. But the importance of human skeletons is different. Therefore, we introduce a learnable mask M and multiply it with the obtained feature map to adjust the importance of different connections. The accelerated CDGC can be expressed as:

where W is the weight matrix composed of weight vectors of multiple output channels. As described in Sec. II-B, both node information captured by vanilla graph convolution and gradient information extracted by CDGC are useful for recognition. Hence, we combine the ordinary shift graph convolution with the accelerated CDGC to construct an effective graph convolution operator. This operator can be expressed as:

where $\alpha\in[0,1]$ controls the accelerated CDGC term. The higher value of $\alpha$ is, the more important of central difference gradient information is. The accelerated CDGC is implemented according to Eq.(9).

NTU RGB+D 60[29] is currently the most widely used indoor-captured action recognition dataset, which contains a total of 56,880 3D skeleton video clips. The clips are captured from three cameras with different settings. All video clips contain a total of 60 human action classes including both single-actor and two-actor actions. Original work[29] suggests two evaluation protocols: Cross-Subject (CS) and the Cross-View (CV). In the CS evaluation, the dataset is divided into training and testing sets according to the subjects. The training set contains 40,320 videos from 20 subjects, and the rest 20 subjects with 16,560 video clips are used for testing. In the CV evaluation, dataset is divided by the camera ID number. The 37,920 videos captured from camera two and three are used in the training and the 18,960 videos from camera one are used for testing. We report the Top-1 accuracy on both protocols. For inputs with more than one stream, e.g., bones, a score-level fusion result is reported.

NTU RGB+D 120[30] is more challenging since it involves more subjects and action categories. The dataset contains 114,480 action samples in 120 action classes. Samples are captured by 106 volunteers with three cameras views. This dataset contains 32 setups, and each setup denotes a specific location and background. According to[30], this dataset suggests two evaluation metrics: Cross-Subject and Cross-Setup. The former one splits subjects in half to training and testing parts while the latter one divides the samples based on the camera setup IDs.

Our model is based on shift-GCN. Specifically, the model consists of one input block and nine basic blocks. In each basic block, we use the proposed accelerated CDGC to learn spatial features, and the temporal correlation is still modeled by temporal shift graph convolution. Both of them followed by a batch normalization (BN) layer and a ReLU layer. A residual connection is added to each basic block to avoid the degradation problem with the increase of network depth, and make the network easy to optimize and fast to converge. A BN layer is added at the beginning to normalize the input graph, and a global average pooling layer is added at the end to pool the feature maps of different samples into the same size. Finally, the output feature is sent to a SoftMax classifier to generate the prediction for action recognition.

All experiments are conducted on the PyTorch deep learning framework [27]. SGD with Nesterov momentum (0.9) is used to train the model for 140 epochs. Learning rate is set to 0.1 and divided by 10 at epoch 60, 80 and 100. Cross-entropy is selected as the loss function to back-propagate gradients. For NTU RGB+D 60 and NTU RGB+D 120, the batch size is 48, and we adopt the same data preprocessing as in [17]. The experiments are performed on two NVIDIA TitanXP GPU.

In this section, we use 2s-AGCN [17] as the backbone to evaluate the effectiveness of our method. All ablation studies are conducted on the NTU RGB+D 60 dataset.

In this experiment, we combine the accelerated CDGC with shift-GCN and compare it with the state-of-the-art methods on NTU RGB+D 60 and NTU RGB+D 120. In addition, as the key hyperparameter of CDGC, $\alpha$ in each layer of the network can also be learned together with the network training. Thus, we also conduct experimental exploration with the learnable $\alpha$ using the same experimental settings as the chosen optimal value 0.3. Like [16], we build four stream networks: joint stream, bone stream, joint motion stream and bone motion stream. For comparison, we report the results of single-stream(joint stream), double-stream(joint stream and bone stream) and four-stream(all). In the Tables IV and V, we abbreviate them as 1s, 2s and 4s, respectively.

The bold numbers indicate that our method has improved the accuracy of skeleton-based action recognition compared with the corresponding baseline. It can be seen from Table IV that our model achieves the best performance for the cross-subject protocol, i.e. 91.0$\%$, 0.3 percentage points higher than the baseline shift-GCN with optimal value $\alpha$ = 0.3. NTU RGB+D 120 dataset is more challenging than NTU RGB+D 60. We can see from Table V that the performance of our model is 0.4 and 0.2 percentage points higher than the baseline for both cross-subject and cross-setup protocols, respectively, and has achieved the best results among the reported to date. We can also see from the two tables that the accuracy of learnable $\alpha$ is almost the same as that of $\alpha$ = 0.3, which proves that $\alpha$ = 0.3 is an optimal choice. Note that the number of parameters in our model is equal to the baseline. The results have demonstrated the effectiveness of CDGC.