Expansion-Squeeze-Excitation Fusion Network for Elderly Activity Recognition

Most existing RGB-based Action Recognition methods [39, 40, 41, 42, 43, 44, 1, 45, 46, 47, 48, 49] can be categorized into two classes. The first class is based on a two-stream network that usually uses RGB and optical flow to model spatial and temporal information, respectively. Karen et al. [50] proposed a two-stream network to model spatial and temporal information in RGB and optical flow frames for the first time. Subsequently, Wang et al. [51] proposed a Temporal Segment Network (TSN) to model long-range temporal action. Crasto et al. [52] proposed a distillation network that uses optical flow to distill RGB data, which can make the RGB-based model learn the temporal information. It is well known that optical flow is computed by using the RGB frames, which is time-consuming and would bring in a bottleneck. The second class is based on a series of 3D convolutional networks, such as, C3D [16], I3D [7], T3D [53], Res3D [54], and so on, which are extended from 2D networks in spatiotemporal dimension. Due to the computation consumption of general 3D convolutional networks, Qiu et al. [55] proposed a Pseudo-3D residual network (P3D) that decomposes the convolutions into separate 2D spatial and 1D temporal filters. Moreover, Feichtenhofer et al. [3] proposed to use two different frame rates to accelerate train 3D networks.

The above-mentioned methods are proposed to address the problem of normal action recognition. Compared with the normal action recognition task, the elderly activity recognition task requires more discriminative information to identify some subtle individual actions and human-object interactions in elderly activities. Due to the lack of temporal information on RGB data, the network implementing on RGB data cannot accurately describe the actions with obvious temporal information. Although some temporal information can be captured from optical flow data, the calculation of optical flow is too time-consuming.

So far, there are many action recognition works based on skeleton [13, 9, 10, 11, 10, 56, 57], called skeleton-based action recognition methods. The main target of skeleton-based action recognition is to learn temporal and spatial information from skeleton sequences. In the early stages, researchers utilized various deep neural networks. e.g., Recurrent Neural Network (RNN), Convolution Neural Network (CNN), and Long Short-Term Memory (LSTM), to model skeleton motions for capturing temporal and spatial information. For example, Zhang et al. [9] proposed an adaptive model designed by CNN and RNN to solve the problem of view difference during the skeleton data collection and action shooting. Banerjee et al. [10] used CNN to extract different complementary motion information from skeleton sequences, e.g., distance and angle between joint points, to better model the temporal information of skeleton data. Recently, researchers used Graph Convolution Network (GCN) to model temporal and spatial information from skeleton sequences by treating skeletons, joints, and bonds as graphs, nodes, and edges, respectively [11, 58, 12]. For example, Yan et al. [11] proposed a Spatial-Temporal Graph Convolutional Network (ST-GAN) that employs graph convolution to aggregate the joint features in the spatial dimension. Liu et al. [58] proposed a multi-stream graph convolutional network to avoid the missing of structural information in the training phase. Cheng et al. [12] proposed a Shift Graph Convolutional Network (Shift-GCN) for solving the problem of inflexible acceptance domain of graph convolution network on both spatial and temporal dimensions.

It can be found that skeleton-based action recognition methods mainly design an effective model to capture temporal and spatial information from skeleton sequences, which is difficult to recognize human-object actions, e.g., blowing hair, combing hair, etc.

Many works [59, 60, 19, 20, 61, 26, 23, 24, 30, 29, 62, 47, 63, 46, 64, 65, 66] consider aggregating information from multi-modal data for various image and video content analysis tasks, such as video action recognition, video anomaly detection and localization, and so on. For example, Wang et al. [23] and Tang et al. [24] proposed the score-level and feature-level multi-modal fusions, respectively. Those straightforward fusion methods did not consider the interaction among different multi-modal information. Zhang et al. [30] proposed to employ several convolutions to fuse the RGB and depth features, which only considered the interaction among multi-modal information from the local view. Zhou et al. [29] proposed to fuse the RGB and thermal image features by channel and spatial-wise attentions, respectively. Overall, on the one hand, different modality data have different distributions. On the other hand, multi-modal data often contain irrelevant (modal-specific) information. Thus, how to aggregate information from multi-modal features is the main problem. Multi-modal data or features fusion across different modalities is always a hot topic, some classical methods [31, 32, 33] were proposed to utilize the linear embedding or attention mechanism to fuse multi-modal features. For example, Hori et al. [32] propose a multi-modal attention model that selectively fuses multi-modal features based on learned attention. With the advancement of Convolutional Neural Networks (CNN), some nonlinear fusion approaches of multi-modal features are proposed by designing various convolutional networks as the attention mechanism [35, 36, 37, 38, 67]. For example, Kuang et al. [36] proposed a multi-modal fusion network based on CNN for face anti-spoofing detection. Fooladgar et al. [37] used an efficient attention method based on CNN architecture to fuse RGB data and depth maps in channel-wise way. Su et al. [68] used the soft attention method to interact multi-modal data in channel-wise way. To fully fuse the multi-modal features, we consider interacting information of multi-modal features not only in channel-wise way but also in modal-wise way. In this work, we design a flexible Expansion-Squeeze-Excitation (ESE) in Modal-fusion Net (M-Net) and Channel-fusion Net (C-Net) to learn modal and channel-wise nonlinear attention for capturing the modal and channel-wise dependencies among features, respectively.

Squeeze-and-Excitation Networks (SENet) [34] has shown remarkable performance in the ImageNet database by explicitly capturing the channel-wise dependencies between feature maps. Here, the channel-wise dependencies are quantified via the nonlinear attention corresponding to each channel. The nonlinear attention is obtained by Squeeze-and-Excitation (SE). SENet has been proven that learning channel-wise nonlinear attention can improve the discriminative ability of features.

In SENet, Squeeze-and-Excitation (SE) can be divided into two steps: squeeze and excitation in turn, which are used for obtaining channel-wise representation and channel-wise nonlinear attention. Specifically, assuming $C$ feature maps, denoted by $X=\{X_{c}\}_{c=1}^{C}$, for one feature map $X_{c}$ in the $c$-th channel, the squeeze operation is performed as follows,

where $H$ and $W$ denote the height and width of the feature map, respectively, $Y_{c}$ is the result of global average pooling for the $c$-th channel. In the squeeze step, all feature maps are transformed to a channel-wise representation $Y=[Y_{1},Y_{2},\cdots,Y_{c},\cdots,Y_{C}]^{T}$, which is a one-dimensional vector. In the excitation step, the channel-wise representation $Y$ is fed into multi fully-connected layers to obtain the channel-wise attention $W$, as follows,

where ${F_{*}}$ denotes a Fully Connected (FC) layer in this work, and ${\sigma}$ is a sigmoid function. Finally, the original feature maps $X$ can be updated to $\hat{X}$ by the following equation

where $\otimes$ denotes channel-wise multiplication between $X$ and $W$. Compared with the original feature maps $X$, the updated feature map $\hat{X}$ has enhanced the discriminative information, while suppressing the useless or irrelevant modal-specific information to some extent.

Through the above analysis, we can find that SENet uses the global average pooling to interact spatial information. Since the global average pooling is implemented from the global view, the learned nonlinear attention is significantly affected by noise (e.g., a large area of background). In this work, to well capture the local and global spatial information of features, we consider interacting spatial information from the local and global views.

For the task of elderly activity recognition, we attempt to aggregate the discriminative information from both RGB videos and skeleton sequences by attentively fusing multi-modal features. Thus, multi-modal features fusion across multiple modalities is key to modeling the motion among RGB videos and skeleton sequences. Based on Squeeze-and-Excitation in SENet, we design a new Modal-fusion Net (M-Net) and a new Channel-fusion Net (C-Net) to learn the Modal-wise Expansion-Squeeze-Excitation Attention (M-ESEA) and Channel-wise Expansion-Squeeze-Excitation Attention (C-ESEA) for capturing the modal and channel-wise dependencies among features, respectively. By integrating M-Net and C-Net as the main components, a novel Expansion-Squeeze-Excitation Fusion Network (ESE-FN) is proposed to attentively fuse the multi-modal features with M-ESEA and C-ESEA. The framework of ESE-FN is shown in Figure 3, which mainly consists of four parts, i.e., feature extractor module, modal-fusion module, channel-fusion module, and multi-modal loss.

Denoted ${\left\{v_{i}|i=1,2,3,...N\right\}}$ as an video set with size $N$, we first split each video into ${T}$ clips and random sample a frame from each clip. Then, we get the frame set ${\left\{r_{i}|i=1,2,3,...T\right\}}$ as one input video ${r}$, which are fed into the RGB backbone (e.g., ResNeXt101 [69]) to extract the RGB feature ${f_{r}\in{R}^{d_{1}\times 1}}$ ( $d_{*}$ denotes the feature dimension in this paper), as follows,

Likewise, for video $r$, the corresponding skeleton sequence $s$ is obtained and fed into the skeleton backbone (e.g., Shif-GCN [12]) to extract the skeleton feature ${f_{s}\in{R}^{d_{2}\times 1}}$,

Generally, the multi-modal features, i.e., $f_{r}$ and $f_{s}$, have different sizes and cannot be directly concatenated. Thus, we use two MLPS to unify the size of $f_{r}$ and $f_{s}$, and then concatenate them, as follows,

where ${f\in{R}^{d\times n}}$, $n$ is the number of modality, ${H}_{r}$ and $H_{s}$ are MLP, and ${{\text{Concat}}(\cdot)}$ is a modal-wise concatenation operator.

Subsequently, we transpose the feature $f$ and feed it into M-Net for modal-wise fusion, as follows,

where ${F_{\text{Trans}}{(\cdot)}}$ is a transpose operator. Likewise, $h_{m}$ is transposed and fed into C-Net for channel-wise fusion, as follows,

We sum up feature $h_{mc}$ in modal-wise way and get the fused multi-modal feature $f_{rs}$.

Finally, RGB feature $f_{r}$, skeleton feature $f_{s}$ and fused multi-modal feature $f_{rs}$ are fed into a new Multi-modal Loss (ML) for optimizing all parameters of ESE-FN, as follows,

where $\Theta$ denotes the parameter set of the proposed ESE-FN. The following sections introduce M-Net, C-Net, Multi-modal Loss in details.

The detailed configuration of Modal-fusion Net (M-Net) is shown in Figure 4. M-Net can attentively aggregate the local and global spatial discriminative information of multi-modal features via M-ESEA in modal-wise way. Specifically, we firstly extend the transposed feature ${f\in{R}^{n\times d}}$ to ${f\in{R}^{n\times d\times 1}}$ as the input of M-Net. Similar to Squeeze-and-Excitation in SENet, we divide the implementation of M-Net into three steps, i.e., modal-wise expansion, modal-wise squeeze, and modal-wise excitation in turn, which are detailed in the following.

Modal-wise expansion step. It uses several stacked convolutional layers with different kernels to interact modal-wise information by expanding the modal information from the local view. Then the expanded feature ${f_{ml}}$ can be described as follows,

where ${f_{ml}\in{R}^{m\times d_{m}\times 1}}$, $d_{m}$ is the spatial dimension after convolution transformation (${m\times d_{m}>n\times d}$). Here, ${f_{ml}}$ can be also regarded as the single-modal feature set $\{f_{ml}^{i}\in R^{(m/n)\times d_{m}}\}_{i=1}^{n}$.

Modal-wise squeeze step. It utilizes ${f_{ml}}$ to calculate the modal-wise representation ${{f_{mlg}}\in{R}^{m\times 1}}$ (${m<n\times d}$) from global view via the average pooling, as follows,

Modal-wise excitation step. It utilizes the modal-wise representation ${f_{mlg}}$ to learn the Modal-wise Expansion-Squeeze-Excitation Attention (M-ESEA) ${W_{m}}$ for capturing the modal-wise dependence of features. Finally, the original input feature $f$ can be updated to the modal-wise fused feature ${h_{m}\in{R}^{n\times d\times 1}}$ based on the following equations,

Compared with SENet, it is noted that we additionally bring in the expanding of features as the partner of the squeezing. Expanding and squeezing provide the interactions of features in up-size and down-size ways, which can capture both of local and global dependencies of features. Different from SENet dealing with the single-modal feature map, we deal with the multi-modal features which can been seen as a feature map by concatenating two modal feature vectors. In the modal-wise expansion step, the convolutions with different kernels can be used to capture the local spatial correlation of multi-modal features in different regions, which can capture the local dependencies of features. In the modal-wise squeeze step, the global average pooling can be used to capture the global spatial correlation of multi-modal features, which can capture the global dependencies of features.

In this section, we focus on our Channel-fusion Net (C-Net), as shown in figure 5. C-Net aims to learn C-ESEA for fusing multi-modal features in channel-wise way. Similar to M-Net, the implementation of C-Net can be also divided into the channel-wise expansion, channel-wise squeeze, and channel-wise excitation in turn, described in the following.

Channel-wise expansion step. It uses the convolutional layer to interact channel-wise information by expanding the channel information from the local view. Then the expanded features ${h_{cg}}$ can be calculated as follows,

where ${h_{cg}\in R^{d\times n_{1}\times 1}}$, ${n_{1}>n}$. It is noted that the multi-modal features are processed by M-Net and C-Net in turn. First, in M-Net, the difference of cross-modal features is relatively large, thus the expansion is large by setting multi-layers. Subsequently, in C-Net (after processed by M-Net), the difference of cross-modal features is relatively small, which only requires a relatively smaller expansion. Thus we set only one convolution layer in C-Net for lightweight designing.

Channel-wise squeeze step. It utilizes ${h_{cg}}$ to calculate the channel-wise representation ${f_{cg}}$ from the global view via the average pooling, as follows,

where, ${f_{cg}}\in R^{d\times 1}$ can be also regarded as the single-channel feature set $\{f_{cg}^{i}\in R^{1\times 1}\}_{i=1}^{d}$.

Channel-wise excitation step. It utilizes the channel-wise representation ${f_{cg}}$ to learn the Channel-wise Expansion-Squeeze-Excitation Attention (C-ESEA) ${W_{c}}$ for capturing the channel-wise dependence of features, as follows,

Finally, we get the feature updated in channel-wise way, described as follows,

By comparing with the formulations of M-Net and C-Net, modal-wise fusion can be seen as a rough fusion of multi-modal features in modal-wise way, while channel-wise fusion can be seen as a fine-grained fusion of multi-modal features in channel-wise way. Both of them attentively aggregate the discriminative information of multi-modal features based on modal and channel-wise dependencies of features.

To keep the consistency between the single-modal features and the fused multi-modal features, we design a new Multi-modal Loss (ML) to additionally measure the difference between the prediction loss on single modalities and the prediction loss on the fused modality. The key idea of multi-modal loss is that we take the minimum prediction loss on single modalities to be consistent with the prediction loss on the fused modality. Formally, we define three types of recognition losses ${\mathcal{L}}_{r}$, ${\mathcal{L}}_{s}$, and ${\mathcal{L}}_{rs}$ corresponding to the RGB modality, skeleton modality, and fused modality, respectively. Then the multi-modal loss can be described as follows,

where ${\alpha}$ and ${\beta}$ are hyper-parameters (will be discussed in Section IV-C). In this work, the forms of ${\mathcal{L}}_{r}$, ${\mathcal{L}}_{s}$, and ${\mathcal{L}}_{rs}$ are cross entropy loss.

We evaluate the performance of ESE-FN in terms of the elderly activity recognition task on the ETRI-Activity3D dataset [70]. It is the currently largest elderly activity recognition dataset collected in real-world surveillance environments, which contains 112,620 samples performed by 100 persons including RGB videos, depth maps, and skeleton sequences. All videos are grouped into 55 classes of actions, including individual activities, human-object interactions, and multiperson interactions. The splitting of training and testing sets is based on person ID, namely the samples with person ID $\{3,6,9,,\cdots,99\}$ for testing, and the samples with person ID $\{1,2,4,5,\cdots,100\}$ for training.

In the data pre-processing phase, we split each video into $T=64$ clips, and randomly select one frame for each clip. Finally, we obtain a new video set, wherein each video contains $64$ frames. Likewise, we also obtain a new skeleton set, wherein each skeleton sequence contains $64$ frames.

In the feature extraction phase, we use ResNeXt18 or ResNeXt101 [69] as the RGB backbone, and Shift-GCN [12] as the skeleton backbone. For the pre-trained ResNeXt18 and ResNeXt101, we fine tune them with the standard SGD optimizer by setting the momentum, initial learning rate, weight decay, and total epochs as 0.9, 0.1, ${10^{-3}}$, and 120, respectively. Especially, the batch size is set to 128 and 32 for ResNeXt18 and ResNeXt101, respectively. For the pre-trained Shift-GCN, we fine tune it with the standard SGD optimizer by setting the momentum, initial learning rate, batch size, and total epochs as 0.9, 0.1, ${10^{-4}}$, 32, and 140, respectively. Subsequently, we use the fine-tuned RGB and skeleton backbone to extract RGB and skeleton features, respectively. Here, to test the representation performance of ResNeXt18, ResNeXt101, and Shift-GCN, we use these three models to extract single-modal features for training a softmax independently. For example, we use ResNeXt18 to extract RGB features and feed them into the softmax. The obtained performance for three models is listed in Table I. We can see that ResNeXt101 performs better than ResNeXt18. In this paper, we choose ResNeXt101 as the RGB backbone in default.

In the training phase of ESE-FN, we use the standard SGD optimizer to train it by setting the momentum, basic learning rate, weight decay, batch size, and total epochs as 0.9, 0.1, ${10^{-4}}$, 32 and 30, respectively. All above experiments are performed via the PyTorch deep learning framework on the Linux server equipped with Titan RTX GPU. The training loss and accuracy of ESE-FN as the number of epochs are shown in Figure 6. We can see that the training loss and accuracy of ESE-FN reach a steady state after about $30$ epochs. And then, the overall loss is consistent with the other sub-losses (i.e., RGB modality, skeleton modality, and fused modality losses) in terms of convergence.

We conduct the diagnostic study to discuss the sensitiveness of the hyper-parameters ${\alpha}$ and ${\beta}$ in Eq. (18). Here, to enhance the efficiency of the diagnostic study, we use ResNeXt18 instead of ResNeXt101 as the RGB backbone and set $T=16$ without loss of generality. Specifically, we empirically tune them by ${\alpha}\in\{0.3,0.5,0.7,0.9\}$ and ${\beta}\in\{0,0.3,0.6,0.9\}$, respectively. The corresponding results are shown in Figure 7. It can be found that the best performance of ESE-FN is achieved when $\alpha=0.7$, and $\beta=0.3$. Thus, we set $\alpha=0.7$, and $\beta=0.3$ in experiments. Moreover, when ${\beta}$=0, the multi-modal loss ${\mathcal{L}}$ is degraded to a basic action recognition loss ${{\mathcal{L}}}_{rs}$, where the performance is degraded significantly. This illustrates that the designed Multi-modal Loss is more effective than the basic loss.

As mentioned before, Modal-fusion Net (M-Net), Channel-fusion Net (C-Net), and Multi-modal Loss (ML) are new components in the framework of the proposed ESE-FN. Thus we conduct ablation study to evaluate the superiority of M-Net, C-Net, and ML. We firstly set seven baselines, as follows,

• •

  B1: Single modal for RGB employs RGB backbone to extract the RGB features, which are directly used to train the softmax classifier. This aims to test the basic recognition performance by using the single-modal features in RGB videos.

• •

  B2: Single modal for skeleton employs skeleton backbone to extract the skeleton features, which are directly used to train the softmax classifier. This aims to test the basic recognition performance by using the single-modal features in skeleton sequences.

• •

  B3: Multi modal for RGB and skeleton employs RGB backbone and skeleton backbone to extract the RGB features and skeleton features, which are concatenated and fed into the softmax classifier. This aims to test the basic recognition performance by simply combining multi-modal features.

• •

  B4: ESE-FN w/o C-Net is a degraded version of ESE-FN by dropping out the channel-fusion module. This aims to show the superiority of M-Net.

• •

  B5: ESE-FN w/o M-Net is a degraded version of ESE-FN by dropping out the modal-fusion module. This aims to show the superiority of C-Net.

• •

  B6: ESE-FN w/o ML is a degraded version of ESE-FN by using the basic loss ${\mathcal{L}}_{rs}$ instead of ML ${\mathcal{L}}$, namely setting $\alpha=1$, and $\beta=0$. This aims to show the superiority of ML.

• •

  B7: the proposed ESE-FN.

The comparison results among all baselines are shown in Table II. Compared with single-modal and multi-modal baselines, B3-B7 (using multi-modal features) outperform B1-B2 (using single-modal features), which validates that the multi-modal features contain more complementary information than the single-modal features. In particular, B4-B7 are equipped with the newly designed components, and perform better than B3 (simply combining multi-modal features). This validates that each component, i.e., M-Net, C-Net, or ML, is superior to help improving the representation ability of multi-modal feature fusion. Here, B6 integrated with M-Net and C-Net further improves by 0.4%(1.2%) over B4(B5) only with M-Net(C-Net). Obviously, ESE-FN (i.e., B7) integrated with all new components achieves the best performance. It is further validated that M-Net, C-Net, and ML can jointly improve the representation ability of multi-modal feature fusion.

SENet is the most related work, where we have added the Expansion step based on SENet and pushed it into the multi-modal fusion task. Thus, we also conduct the ablation experiment to compare with the proposed ESE-FN and SENet by setting six baselines, as follows:

• •

  A1: SENet only for channel-wise fusion uses SENet instead of C-Net for multi-modal feature fusion only in the channel-wise way. This can be seen as simply using Squeeze-Excitation for channel-wise fusion.

• •

  A2: SENet only for modal-wise fusion uses SENet instead of M-Net for multi-modal feature fusion only in the modal-wise way. This can be seen as simply using Squeeze-Excitation for modal-wise fusion.

• •

  A3: SENet for modal and channel-wise fusion uses SENet instead of M-Net and C-Net for multi-modal feature fusion in both the modal-wise and channel-wise ways. This can be seen as simply using Squeeze-Excitation for both modal-wise and channel-wise fusions.

• •

  A4: ESE-FN only for channel-wise fusion only uses C-Net for multi-modal feature fusion in channel-wise way.

• •

  A5: ESE-FN only for modal-wise fusion only uses M-Net for multi-modal feature fusion in modal-wise way.

• •

  A6: ESE-FN (Ours) uses M-Net and C-Net for multi-modal feature fusion in both the modal-wise and channel-wise ways.

The comparison result of the ablation study is listed in Table III. We can find that 1) ESE-FN improves by 0.6% over A3 (SENet for modal and channel-wise fusion), which illustrate that the Expansion step is effective; and 2) A3 outperforms both A1 and A2, as well as A6 outperforms both A4 and A5, which illustrate that our modal and channel-wise fusion strategy is effective and flexible for the SENet family.

We evaluate the performance of the proposed ESE-FN on the elderly activity recognition task by comparing it with the currently representative multi-modal methods, including Deep Bilinear Learning [72], Evolution Pose Map [71], c-ConvNet [73], and FSA-CNN [70]. Specifically, Deep Bilinear Learning uses RGB frames, depth maps, and skeleton sequences; Evolution Pose Map uses 3D and 2D poses extracted from RGB frames and HeatMaps, respectively; c-ConvNet uses RGB frames and depth maps; FSA-CNN uses 2D and 3D skeletons extracted from RGB frames and depth maps. The comparison results of all methods are shown in Table IV, where some results have been reported in [70]. The proposed ESE-FN with the accuracy of 95.9% achieves the state-of-the-art performance, which improves by 2.2% (a relative 2.3% improvement) over the previous SOTA method, namely FSA-CNN with the accuracy of 93.7%. For a fair comparison, ESE-FN adopts the same backbone as the RGB feature extractor, and also performs better than the Deep Bilinear Learning method and c-ConvNet method, which illustrates the robustness of ESE-FN.

In addition, the confusion matrices obtained by ESE-FN, Shift-GCN and ResNeXt101 are shown in Figure 8. First, the confusion matrix of ESE-FN shows that the color of the main diagonal is lighter than other spaces, which illustrates all classes of elderly activities can be well recognized by ESE-FN. Second, compared with the confusion matrices of ESE-FN, Shift-GCN, and ResNeXt101, we can find that the color of the main diagonal in Figure 8(c) is lighter than the color of the main diagonal in Figure 8(a) and (b). This shows that ESE-FN is more effective for recognizing the confusing elderly activities by capturing more discriminative multi-modal information.

To test the generalization of the proposed ESE-FN, we also conduct comparative experiments between the proposed ESE-FN and the other advanced methods in terms of the normal action recognition task. We use the NTU RGB+D dataset [85] as the benchmark. The NTU RGB+D dataset is a large-scale dataset collected by three Kinect cameras from different views concurrently, and has been widely used for evaluating RGB-based or skeleton-based action recognition methods. It includes 56,880 skeleton sequences and 60 different classes from 40 distinct subjects. NTU RGB+D dataset provides two standard evaluation protocols, i.e., Cross-Subject (CS) setting, and Cross-View (CV) setting. In the CS setting, the training set includes 40,320 samples performed by 20 subjects, and the testing set includes 16,560 samples performed by the other 20 subjects. In the CV setting, the training set includes 37,920 samples obtained from camera views 2 and 3, and the testing set includes 18,960 samples obtained from camera view 1.

Table V shows the recognition accuracies obtained by different methods on the NTU RGB+D dataset, where the performance of the other methods have been reported in [70]. We can see that ESE-FN is comparable to the alternatives, including RNN-based methods (e.g., IndRNN [74], Beyond Joint [75]), CNN-based methods (e.g., SK-CNN [76], MANs [79], Deep Bilinear Learning [72]), and GCN-based methods (e.g., ST-GCN [11], Shift-GCN [12]). Specifically, ESE-FN improves by 0.7% and 0.2% over the SOTA methods, namely Evolution Pose Map with the accuracy of 91.7% and Shift-GCN with the accuracy of 96.5% on CS and CV settings, respectively. In addition, we also show the confusion matrices obtained by ESE-FN on NTU RGB+D (CS and CV) in Figure 9(a) and (b). These confusion matrices show that the color of the main diagonal is lighter than other spaces, which illustrates all classes of normal actions can be well recognized by ESE-FN. It can be concluded that the proposed ESE-FN is generalized to recognize normal actions.