Dynamic Spatial-temporal Hypergraph Convolutional Network for Skeleton-based Action Recognition

CNNs have achieved remarkable results in processing euclidean data like images and RNNs have achieved significant advantages in processing time series. So the research began to emerge with CNN and RNN methods [8, 9, 10]. The advent of graph convolutional neural networks (GCNs) have significantly improved the processing of non-Euclidean data. Yan et al. [1] proposed the ST-GCN model, which can better handle the topological relationship of skeletons to achieve more effectiveness. Chen et al. [2] proposed the CTR-GCN method, which can dynamically learn joint information on space to achieve channel topology modeling. Song et al. [11] proposed a family of efficient GCN baselines with high accuracies and small amounts of trainable parameters. This methods use a strategy of spatial-temporal separate learning in model configuration.

The graph structure used in traditional graph convolutional neural networks focuses on establishing a correspondence between two objects to form a set. Hyperedges can connect arbitrary objects to form hypergraphs to obtain high-order dependencies. Feng et al. [12] proposed hypergraph neural network (HGNN) to learn the hidden layer representation considering the high-order data structure. Jo et al. [13] proposed a pairwise hypergraph transformation method that makes the network focus on the feature information of the edges. Jiang et al. [14] proposed a dynamic hypergraph network (DHNN) to update the hypergraph structure dynamically, etc. In this paper, we use the generic hypergraph convolution described with HGNN [12].

$\displaystyle X_{i+1}=\sum_{A{i}\in A_{k}}X_{i}\cdot\widetilde{A_{i}}\cdot M_{i},$

(1)

where $X_{i}\in R^{C_{in}\times T\times V}$ denotes the input features of the i-th layer of the network, $C_{in}$ is the number of channels, $\widetilde{A_{i}}$ denotes the adjacency matrix of the normalised spatial segmentation method, $A_{k}$ denotes spatial skeletal segmentation method [1], and $M_{i}$ denotes the learnable weight features.

In the dynamic time-point hypergraph construction method, we use the k-NN method to construct the features after linear transformation and dimensionality reduction to reduce the computational cost. As shown in Fig. 3, the hypergraph $H_{T}\in R^{T\times T}$ is formulated as:

$\displaystyle H_{T}=TH(\delta(\varepsilon(X),k),$

(2)

where $k$ denotes the number of neighbors of the each hyperedge, $X\in R^{C_{in}\times T\times V}$ denotes the input features, $\varepsilon(\cdot)$ is the linear transformation dimensionality reduction function, $\delta(\cdot)$ is the dimensionality transformation function, and $TH(\cdot)$ denotes the k-NN function.

We use k-NN, k-means, and the combination of centrifugal and centripetal joints [1] to construct the spatial hypergraph matrix, as shown in Fig. 3. The spatial hypergraph method is formulated as:

$\displaystyle H_{N}=SH(A),$

(3)

where $SH(\cdot)$ are the three methods mentioned above, and $A$ is the topological adjacency matrix.

Further, we construct two spatial-temporal dependencies $H_{ST}\in R^{V\times T},H_{TS}\in R^{T\times V}$ based on the above hypergraph. As shown in the following equation:

$\displaystyle H_{ST}/H_{TS}=\Phi((H_{N},H_{T})/(H_{T},H_{N})),$

(4)

where $\Phi(a,b)=tanh(\mu(a)-\varphi(b))$, and $\mu(\cdot)$, $\varphi(\cdot)$ denotes two dimensional transformation operation functions.

In addition, $HC(H_{i})$ denotes the hyperedge feature is not considered:

$\displaystyle HC(H_{i})=X_{i}\cdot\widetilde{H_{i}}\cdot\Theta_{i}$

(6)

Finally, we obtain the following features: $HC^{T}(H_{N})$, $HC^{T}(H_{T})$, $HC(A)$, $HC(H_{ST})$, $HC(H_{TS})$, denoted by $A^{{}^{\prime}}-E^{{}^{\prime}}$, respectively. The implementation process is shown in Fig. 3.

With the multiple high-order information obtained from convolution operations, we constructed the HIF module for parallel modeling. The implementation is shown in Fig. 3. First, we use temporal features to refine spatial features to obtain high-order joint information. Then through spatial-temporal aggregation in the channel dimension and channel concatenating operations to capture the properties between features. The final output is $F_{out}\in R^{C_{out}\times T\times V}$.

Specifically, to avoid the loss of the original topology information, we use the original natural joint topology as an auxiliary learning component. It is combined with spatial hypergraph features to learn more connected joint features of the action. As indicated below:

$\displaystyle Y_{2}=(HC^{T}(H_{N})+HC(A))/\beta,$

(7)

where $\beta$ is the learning weights.

We use a multi-scale convolution, TF, with an attention mechanism [15, 2] to extract information at different scales. As expressed in the following equation:

$\displaystyle Z_{out}=TF\left(F_{out1}+F_{out2}+F_{out3}\right).$

(8)

$Z_{out}\in R^{C_{out}\times T\times V}$ is the output of a block of dynamic spatial-temporal hypergraph convolutional network. The network consists of ten blocks. The difference in $F_{out1}$, $F_{out2}$, $F_{out3}$ is composed of three different spatial hypergraph topologies, respectively.

Our experiments use cross entropy as the loss function and use three Tesla M40 GPUs. In addition, we use dual correlation channel data to aid fuse learning, which is obtained by subtracting two channels. The experiments use the stochastic gradient descent (SGD). The weight decay is set to 0.0004 and the Nesterov momentum is 0.9. The train contains 90 epochs, the first five epochs use a warm-up strategy, and we set cosine annealing for decay [18]. For the NTU-RGB+D [8] and NTU-RGB+D 120 datasets [17], the batch size is set to 64. For the Northwestern-UCLA dataset [16], we set the batch size to 16.

Unless otherwise stated, we analyze our method on the X-sub benchmark of the NTU RGB+D dataset.

In Fig. 4 (a), (b), We observe that different combinations of hypergraphs is able to learn different spatial-temporal information. In addition, (a) focuses more on information from the elbow joint and (b) focuses on information features of the hand, suggesting that these joints are more discriminatory for overall. In Fig. 4 (c), (d), the further changes in joint features occurred and the joints are differentiated in temporal aspects. It is shown that the channel learning method can effectively perform information fusion.

This section compares our dynamic spatial-temporal hypergraph neural network with state-of-the-art methods. For a fair comparison, we use four streams of fusion results, considering joint, bone, and motion data stream forms [2, 20, 19]. We use different fusion parameters [15] for different data to accommodate the diversity of varying stream forms.

The results are shown in Table 3. It can be seen that our model exhibits comparable or better performance than the state-of-the-art approach for the three datasets. Compared to the hypergraph methods, our proposed method has reached state-of-the-art accuracy in hypergraphs and by a margin of 1.5$\%$ and 2.8$\%$ on X-Sub, which fully illustrates the significant advantage of our method.