Semantics-aware Adaptive Knowledge Distillation for Sensor-to-Vision Action Recognition

Action Recognition is an active research field and has received great attention in recent years [18]. The action recognition methods can be divided into three types: (1) handcrafted representations based [19, 20, 21, 22, 23, 24, 25, 26], (2) graph learning based [27, 28, 29], and (3) deep learning based [30, 31, 32, 33, 34, 35]. To be noticed, most of them are based on vision-sensors modality such as RGB, depth, skeleton, infrared images or videos, etc. Some representative RGB based works include IDT [36], DAG [27], 3D CNN [30], two-stream CNNs [31], C3D [32], TSN [33], TRN [2], MLGCN [28], TSM [35], etc. Yuan et al. [37] introduced a statistical hypothesis detector for abnormal RGB event detection in crowded scenes. Yuan et al. [38] proposed a memory-augmented temporal dynamic learning network to learn temporal motion dynamics and tackle unsteady dynamics in long-duration motion of videos. Li et al. [39] introduced a spatio-temporal manifold network (STMN) that leverages data manifold structures to regularize deep action feature learning. In addition, other modalities (depth [40, 41], skeleton [42, 43], infrared [44]) based methods also receive increasing attention. Zhang et al. [29] proposed semantics-guided neural network (SGN) for skeleton-based action recognition. Zhang et al. [45] presented a low-cost descriptor called 3D histograms of texture (3DHoTs) to extract discriminant features from a sequence of depth maps. Though these vision-sensors based methods have achieved promising results, their performance is easily affected by camera viewpoints, background clutter, occlusion, and illumination change. In some privacy-sensitive area such as bank and government, the difficulty of acquiring images and videos make it infeasible. Furthermore, vision-based methods usually require expensive hardware resources to run deep learning models with high computational demands.

With the popularity of the wearable devices such as smartwatches and smartphones, human action recognition based on wearable-sensors has become a key research area in human activity understanding [46, 9]. Although aforementioned visions-sensors based methods have achieved good results, they cannot be directly applied to wearable-sensor based problems due to the existence of huge modality divergence. Since wearable-sensors action data is suitable for privacy protecting and robust to variant illuminations and camera viewpoints, some specific works have been proposed recently. Jiang et al. [7] assembled signal sequences of accelerators and gyroscopes into an activity image to learn optimal representations automatically. Wannenburg et al. [8] utilized ten different classifier algorithms to classify the human actions using the accelerator signals captured by smartphones. Setiawan [47] used gramian angular field to transform one-dimensional wearable-sensor signals to two-dimensional images. Wang et al. [10] proposed an attention-based CNN framework to address weakly-supervised sensors-based action recognition problem. Fazli et al. [48] built a hierarchical classification with neural networks to recognize human activities based on built-in sensors in smart and wearable devices. Different from vision-sensor based methods, most of these wearable-sensor based action recognition methods are based on raw sensor time-series signals, which lack color and texture information and could not preserve the local temporal relationship. In addition, these methods use simple feature fusion methods to fuse the knowledge from different sensor modalities without considering intra-modality similarity, semantic embeddings, and multiple relational knowledge.

To increase the representative ability of wearable-sensors action features, we construct a Gramian Angular Field (GAF) based virtual image generation model, which transforms the one-dimensional time-series signals of wearable-sensors into two-dimensional image representations. To fully utilize the complementary knowledge from multiple wearable-sensors, we propose a Similarity- Preserving Adaptive Multi-modal Fusion Module (SPAMFM), which integrates intra-modality similarity, semantic embeddings, and multiple relational knowledge to learn the global context representation and recalibrate the channel-wise features adaptively.

Action recognition has been developed for a long period, but action recognition on multiple modalities is a relatively new topic. With the development of deep learning methods and various hardware such as cameras and wearable devices, there are some typical methods of dealing with multi-modal action recognition problems in recent years. These methods can be roughly categorized into three types: 1) cross-view action recognition, typical works [49, 50] used transfer learning methods to reduce the domain gap of action data from different camera views; 2) cross-spectral action recognition, typical works [51, 52] addressed the visible-to-infrared action recognition problems using domain adaptation methods. Yuan et al. [53] proposed a spatial-optical data organization and sequential learning framework based on spatial-optical action data; 3) cross-media action recognition, typical works [54, 55] designed specific multi-modal feature learning frameworks to address the image-to-video action recognition problems.

Different from these cross-domain action recognition problems, the multi-modal action recognition based on wearable-sensors and vision-sensors (sensor-to-vision) is essentially a heterogeneous knowledge transfer problem because there exists large modality difference between wearable-sensors and vision-sensors in data dimension, data distribution, and inherent information content, which is shown in Fig. 1. And related research about sensor-to-vision action recognition is limited. Chen et al. [56] proposed a feature fusion framework to combine signals from the depth camera and inertial body sensor. Kong et al. [5] built a multi-modality distillation model with the attention mechanism to realize adaptive knowledge transfer from sensor modalities to vision modalities. Hamid et al. [57] proposed a multi-modal transfer module to fuse knowledge from different unimodal CNNs and tested this module for three different multi-modal fusion tasks: gesture recognition, audio-visual speech enhancement, and action recognition. However, most of these methods only use raw one-dimensional time-series sensor signals to recognize actions. Since time-series data lacks local temporal relationship, color, and texture information, it may affect the representative ability of wearable-sensor signals and make existing pre-trained deep learning models (e.g. LeNet, AlexNet, VGGNet, ResNet, etc.) hard to adapt. Furthermore, the semantic relationship between wearable-sensors and vision-sensors action data is ignored in previous works, which can guide the knowledge transfer.

In this paper, we use the semantics-aware information to guide the multi-modal feature fusion, knowledge distillation, and representation learning of our SAKDN. Although method [57] built a squeeze and excitation based multi-modal feature fusion module which only used the simple concatenation of features from different modalities to learn the global context embedding without considering more diverse relation functions and the intra-modality similarity relationship. Differently, we propose a novel plug-and-play module, named Similarity-Preserving Adaptive Multi-modal Fusion Module (SPAMFM), which seamlessly integrates intra-modality similarity, semantic embeddings, and multiple relational knowledge to learn the global context representation and recalibrate the channel-wise features adaptively in each network.

Knowledge distillation is a general technique for supervising the training of student networks by capturing and transferring useful knowledge from well-trained teacher networks. Hinton et al. [15] used softened labels of the teacher with a temperature to transfer knowledge to a small student network. Attention transfer [58] designed a knowledge distillation loss based on summed $p$-norm of convolutional feature activations along channel dimension. Park et al. [59] proposed distance-wise and angle-wise distillation losses to realize relational knowledge transfer. Tung et al. [60] construct a knowledge distillation loss with the constraint that input pairs that produce similar (dissimilar) activations in the teacher network should produce similar (dissimilar) activations in the student network. Hoffman et al. [61] built a modality hallucination architecture for training an RGB object detection model using depth as side information. Garcia et al. [62] proposed a generalized distillation framework considering the case of learning representations from the depth and RGB videos, while relying on RGB data only at test time. Crasto et al. [63] introduced a feature-based loss compared to the Flow stream, a linear combination of the feature-based loss and the standard cross-entropy loss, to mimic the motion stream, and as a result avoids flow computation at test time.

Different from existing knowledge distillation methods that focus on the modality transfer task across vision-sensor based modalities, we move a further step towards knowledge transfer from wearable-sensor based modalities to vision-sensors based modalities. In this paper, we construct a novel knowledge distillation module, named Graph-guided Semantically Discriminative Mapping (GSDM), which utilizes graph-guided ablation analysis to produce good visual explanations highlighting the important regions for predicting the semantic concept and preserving the intrinsic structures concurrently. Since the semantic relationship between wearable-sensors and vision-sensors is similar, we transfer the semantics-aware attention knowledge of multiple well-trained teacher networks to guide the training of the student network.

The framework of the SAKDN is shown in Fig. 2, which is an end-to-end knowledge distillation framework seamlessly constituted by three parts: virtual image generation of wearable-sensors, training of multiple teacher networks, and multi-modality knowledge distillation from multiple teacher networks to the student network. We use wearable-sensors action data (acceleration, gyroscope, and orientation) as teacher modalities and RGB videos as student modalities. The virtual image generation part uses Gramian Angular Field (GAF) [17, 47] to encode one-dimensional time-series signals of wearable-sensor into the two-dimensional image representation. We build a novel Similarity-Preserving Adaptive Multi-modal Fusion Module (SPAMFM) to fuse intermediate representation knowledge from different teacher networks adaptively. Then we use semantic preserving loss along with cross-entropy loss for the training of multiple teacher networks. The multi-modality knowledge distillation consists of two knowledge distillation losses, one is our proposed Graph-guided Semantically Discriminative Mapping (GSDM) loss, and the other one is the soft-target knowledge distillation loss [15]. We train the student network using GSDM loss, soft-target loss, semantic preserving loss, and cross-entropy loss.

To make the one-dimensional action data of wearable-sensor preserve the local temporal relationship and facilitate its visual recognition, we build a Gramian Angular Field (GAF) based virtual image generation model which transforms the one-dimensional time-series signals into two-dimensional image representations. The Gramian Angular Field (GAF) based virtual image generation model is shown in Fig. 3. Since there are three axial time-series signals (x, y, z) of wearable-sensors action data, we denote one of the tri-axial signals as $X=\{x_{1},\cdots,x_{n}\}$. We then use min-max normalization to normalize original signal $X$ into the interval $[-1,1]$ and get normalized signal $\widetilde{X}$,

Then, we use transformation function $g$ to transform the normalized signal $\widetilde{X}$ to the polar coordinate system, which represents cosine angle from the normalized amplitude and the radius from the time $t$, as represented in Eq. (2).

After encoding the normalized time-series signals into a polar coordinate system, the correlation coefficient between time intervals can be easily obtained by trigonometric sum between points. Since the correlation coefficient can be calculated by the cosine of the angle between vectors [17, 47], the correlation between time $i$ and $j$ is calculated using $\cos(\theta_{i}+\theta_{j})$ and the Gramian Angular Field based matrix is defined as $G$:

The encoding map of Eq. (3) has two important properties. Firstly, it is bijective as cos$(\phi)$ is monotonic when $\phi\in[0,\pi]$. Given the time-series data, the proposed map produces one and only one result in the polar coordinate system with a unique inverse map. Secondly, as opposed to Cartesian coordinates, polar coordinates preserve absolute temporal relations. After encoding the scaled time-series signals into a polar coordinate system, we can easily extract the correlation coefficient between time intervals considering the trigonometric sum between points. In this way, the GAF provides a new representation style that can preserve the local temporal relationship in the form of temporal correlation as the timestamp increases. For wearable-sensor based action data, the accelerator, gyroscope, and orientation signals are in tri-axial style. Therefore, we assume that each axis sensor data with length $n$ can be transformed into a single GAF matrix with the size $n\times n$. Then, the GAF matrices of tri-axial sensor data (x-, y-, and z-axis) are assembled as a three-channel image representation $P=\{G_{x},G_{y},G_{z}\}$ of size $n\times n\times 3$. And this novel image representation is named as GAF based Virtual Image (GAFVI). GAFVI of wearable-sensors will be used as the input for teacher modalities in this paper.

Since there exist multiple teacher modalities when training the teacher networks, we need to discover and fuse representative features among these modalities. To be noticed, attention gives a feasible way to extract important features for prediction. However, most of the existing attention operations [58, 64, 65] focus on the uni-modal problem and neglect the relationship knowledge among different modalities when conducting multi-modal fusion. To fully utilize the complementary knowledge from multiple teacher modalities, we extend the attention operations from uni-modal to multi-modal feature fusion by integrating the intra-modality similarity, semantic embeddings, and multiple relational knowledge into a unified Similarity-Preserving Adaptive Multi-modal Fusion (SPAMFM) module. The simplest case of SPAMFM for two modalities is shown in as shown in Fig. 4.

Traditional knowledge distillation methods usually conduct knowledge transfer at the last fully-connected layers and ignore the intermediate layers which contain abundant essential complementary information between networks. For heterogeneous knowledge problems like sensor-to-vision action recognition, the knowledge in intermediate layers is important for efficient knowledge transfer. To mitigate the modality divergence between teacher and student modalities, we propose a novel semantics-aware knowledge distillation module, named Graph-guided Semantically Discriminative Mapping (GSDM), which works at convolutional layers and transfers the semantics-aware attention knowledge of multiple well-trained teacher networks to guide the training of student network. This module utilizes graph-guided ablation analysis to produce good visual explanations for both teacher and student modalities highlighting the important regions for predicting the semantic concept and concurrently preserving respective interrelations of data for each modality.

Since previous works [66, 67, 68] have validated that the ablation of some units of a network can be an indicator of how important a unit is for a particular class, we use ablation drop of mini-batch input features to produce visual explanations based knowledge distillation loss across domains. Different from previous methods which use global average pooled gradients and class scores for visual explanation, we use semantics-guided ablation analysis to learn the visual explanations because the similar semantic relationship between wearable-sensors and vision-sensors data can be considered as good guidance for knowledge transfer while class scores are too strict for heterogenous sensor-to-vision action recognition problem. The framework of GSDM is shown in Fig. 5.

The input mini-batch data of student network contains two parts, the first part $I\in\mathbb{R}^{b\times c\times h\times w}$ contains raw input mini-batch data, the second part $I_{a}\in\mathbb{R}^{b\times c\times h\times w}$ contains black images which is essentially the ablation of the raw input data. And the combination of these two parts $[I;I_{a}]\in\mathbb{R}^{2b\times c\times h\times w}$ is used as the input. We assume that the class score $y^{c}$ for class $c$ can be considered as a non-linear function of input data. When we set all the input mini-batch data to zeros and repeat the forward pass, we get a reduced activation score $y_{a}^{c}$ with respect to feature map $A_{p}$ of $p$-th unit. Based on these class scores $y^{c}$ and $y_{a}^{c}$, we use Glove [69] to calculate their corresponding semantic embeddings $F^{c}$ and $F_{a}^{c}$:

Since manifold learning [70] extracts intrinsic structures from data, we construct two graphs for original data $I$ and ablation data $I_{a}$, where vertexes are embedded features at the final fully connected layers and edges are the relations between features. The edge weights $W_{i,j}$ between the input data $x_{i}$ and $x_{j}$ are determined by the Gaussian similarity, $W_{i,j}=\textrm{exp}(-\frac{\|f_{i}-f_{j}\|^{2}}{2})$, where $f_{i}$ and $f_{j}$ are embedded feature vectors of $x_{i}$ and $x_{j}$. Then we apply the normalized graph Laplacians [71] on $W$, that is, $Q=D^{1/2}WD^{-1/2}$, where $D$ is a diagonal matrix with its $(i,i)$-value to be the sum of the $i$-th row of $W$. In this way, the manifold structure in the data can be well represented in graph matrix $Q\in\mathbb{R}^{b\times b}$.

To preserve the embedded low-dimensional manifold subspace structure of original modalities when conducting knowledge distillation, we first multiply the semantic embeddings with the graph matrix, which is essentially the manifold regularization. In this way, a semantic space that is robust against small perturbations can be produced [72]. Semantics propagation can be seen as a repeated random walk through the graph of features using an affinity matrix to assign the semantic embeddings of data, which can effectively preserves the manifold structure [73]. Then, we define a graph-guided slope metric $\omega_{p,l}^{c}\in\mathbb{R}^{b\times 300}$ to measure the changing rate of the transformed semantic embeddings for the p-th unit of layer $l$ and class $c$.

where $Q\in\mathbb{R}^{b\times b}$ and $Q_{a}\in\mathbb{R}^{b\times b}$ are the normalized graph similarity matrices for original data and ablation data, respectively. In this way, the intrinsic of data can be preserved and concurrently the importance value can be represented by the fraction of drop in semantic embeddings of class $c$ when the input features are removed. Then the graph-guided semantically discriminative map $M_{l}^{c}$ for the $l$-th layer of class $c$ can be obtained as a weighted linear combination of activation maps $A_{p,l}$ and corresponding weights $\omega_{p,l}^{c}$,

The dimensionality of the weight $\omega_{p,l}^{c}$ is adaptively adjusted to the dimensionality of different feature maps in the same way as the Grad-CAM [74]. The GSDM of specific layers for teacher and student networks are generated following Eq. (16). After obtaining the GSDM, the GSDM based knowledge distillation loss can be constructed. Assume that we have $m$ teacher modalities and one student modality, we use mean squared error (MSE) loss between the normalized GSDM of teachers and student to transfer knowledge:

where $M_{l_{T}}^{T_{k}}$ denotes the GSDM for teacher network $T_{k}$ of the $l$-th layer, $M_{l_{S}}^{S}$ is the GSDM for student network of the $l$-th layer, $\mathcal{L}_{distill}^{T}$ represents the group containing the choosing layers of teacher networks for knowledge distillation, $\mathcal{L}_{distill}^{S}$ is the group containing the choosing layers of student network, and $\|\cdot\|_{2}$ denotes the L2 norm. $N^{\mathcal{L}_{distill}^{T}}$ is the number of choosing layers in group $\mathcal{L}_{distill}^{T}$.

Based on the Virtual Image Generation model, Similarity-Preserving Adaptive Multi-modal Fusion Module (SPAMFM), and Graph-guided Semantically Discriminative Mapping (GSDM) knowledge distillation module, we seamlessly integrate them into a unified adaptive knowledge distillation framework, named Semantics-aware Adaptive Knowledge Distillation Network (SAKDN), to address the sensor-to-vision heterogenous action recognition problem, shown in Fig. 2. In SAKDN, we use multiple wearable-sensors as teacher modalities and use vision-sensor as student modalities. For teacher networks, the input is GAF images of wearable-sensors. And the input of the student network is the RGB video.

In this work, we conduct extensive experiments on three benchmarks for sensor-to-vision action recognition. We first introduce the three datasets and implementation details. Then, we compare our SAKDN with existing knowledge distillation and action recognition methods. In addition, we conduct ablation studies to analyze the importance of the proposed ST, SPAMFM, GSDM, and SP. Furthermore, we conduct experiments with different backbone architectures and selected transfer layers to validate whether our SAKDN could generalize to different networks and choosing layers. To show how we select the values of hyper-parameters, we use the grid-search method to conduct parameter sensitivity analysis of hyper-parameters $\alpha$, $\beta$, and $\gamma$ in three datasets. Finally, we illustrate the training curves of our method and use CAM to visualize the space-time regions that contribute to our predictions.

We compare the performance of our SAKDN with state-of-the-art knowledge distillation (KD) methods [81, 82, 15, 58, 59, 60, 83], vision-based action recognition (VAR) methods [1, 2, 35], and multi-modal action recognition (MMAR) methods [76, 79, 80, 77, 84, 12, 6, 5]. The comparison results on three datasets are shown in Table II, Table III and Table IV, respectively. For these VAR and KD methods, we use the shared codes, and the parameters are selected based on the default setting. Since [5] is the only existing multi-modal action recognition method for the MMAct dataset which uses F-measure to evaluate the performance, we also adopt the F-measure in the MMAct dataset to make a fair comparison.

In Table II and Table III, the SAKDN achieves better performance than all the comparison action recognition methods, multi-modal action recognition methods, and knowledge distillation methods. This validates that our SAKDN can effectively improve vision-sensor based action recognition performance by integrating SPAMFM, GSDM, and SP into a unified end-to-end adaptive knowledge distillation framework. From Table IV, we can see that the SAKDN performs better than most of the comparison action recognition methods, multi-modal action recognition methods, and knowledge distillation methods. To be noticed, the TSM [35] achieves a comparable performance with the SAKDN. This is because the TSM shifts part of the channels along the temporal dimension and thus facilitates information exchanged among neighboring frames. Although the MMAD [5] proposed a multi-modality distillation model to transfer the knowledge from wearable-sensors to vision-sensors, it only uses raw one-dimensional time-series sensor signals without considering virtual image generation of wearable-sensor data and the semantic relationship. When using the SAKDN, the performance is the best among all the comparison methods. This validates the effectiveness of our adaptive knowledge distillation framework for sensor-to-vision heterogenous action recognition by integrating Gramian Angular Field (GAF) based virtual image generation, SPAMFM, GSDM into a unified end-to-end deep learning framework.

To evaluate the contribution of the SPAMFM, the ST, the GSDM, and the SP, we construct four different algorithms based on the SAKDN. Student (Baseline): our student network trained with only RGB videos. Multi-Teachers: our teacher networks trained with all wearable-sensor modalities. SKDN: our SAKDN without Similarity-Preserving Adaptive Multi-modal Fusion Module (SPAMFM). KDN: our SAKDN without soft-target loss. SADN: our SAKDN without Graph-guided Semantically Discriminative Mapping (GSDM). AKDN: our SAKDN without Semantic Preserving (SP) for both teacher and student networks. SAKDN: our proposed Semantics-aware Adaptive Knowledge Distillation Networks.

The average accuracies on Berkeley-MHAD, UTD-MHAD, and MMAct datasets are shown in Table V, VI and VII, respectively. In Table V, the SAKDN for multi-teachers achieves better performance than that of the single teacher modality (Acc1-Acc6). Additionally, the SAKDN for six teacher modalities achieve better performance than that of the SKDN, these validate that our proposed SPAMFM can effectively fuse the complementary knowledge among different wearable-sensors. Without soft-target loss, the KDN performs worse than the SAKDN. This shows the importance of the ST in conducting knowledge transfer at the last fully-connected layers between teacher and student networks. In addition, the SAKDN also performs better than the AKDN, this verifies that the semantics-preserving part indeed acts as an effective guide for knowledge transfer. The student model with only RGB input (Baseline) achieves better performance than that of the teacher models because the wearable-sensors on Berkeley-MHAD dataset lack color and texture information which may degrade their representative abilities. Introducing different accelerators to the student model improves the performance from $95.32\%$ to $99.33\%$, which validates the existence of complementary knowledge between wearable-sensors and vision-sensor modalities. In SAKDN, a more significant improvement of performance is achieved than SKDN, KDN, SADN, and AKDN in video action recognition. In addition, the performance of SKDN, KDN, SADN, AKDN are all better than that of the student baseline, which verifies that the SPAMFM, ST, GSDM, and SP are complementary and essential.

From Table VI, we can see that the teacher-acc, teacher-gyo and multi-teachers using SAKDN outperform the SKDN and AKDN. This validates that the SPAMFM can fully utilize complementary knowledge from multiple teacher modalities and the SP can guide the knowledge transfer using the similar semantic relationship between teacher and student modalities. Moreover, the performance of the SKDN, KDN, SADN, and AKDN are all better than that of the student baseline, which validates the effectiveness of SPAMFM, ST, GSDM, and SP. Among SKDN, KDN, SADN, AKDN, and SAKDN, the SAKDN achieves the best performance and improves the video action recognition performance from $94.87\%$ (baseline) to $98.60\%$. This validates that our SAKDN can effectively transfer the knowledge from wearable-sensor modalities to vision-sensor modalities by integrating four complementary modules, SPAMFM, ST, GSDM, and SP.

To make a fair comparison with other methods in MMAct dataset, we use four different settings. 1) cross-subject: samples from $80\%$ of the subjects are used for training and the remaining $20\%$ for testing; 2) cross-view: samples from $3$ views are used for training and the remaining one for testing; 3) cross-scene: samples from the scenes without occlusion are used for training and the samples with occlusion scene for testing; 4) cross-session: samples from top $80\%$ sessions in ascending order for each subject are used for training and the remaining sessions for testing. The results for different settings on the MMAct dataset are shown in Table VII. In cross-view and cross-scene settings, the multi-teachers achieves better performance than that of the baseline. This is because wearable-sensor based action data are more robust to the occlusion and appearance variations caused by the camera viewpoint and scene change. Since appearance and texture information are important for action recognition in cross-subject and cross-session settings, the baseline model fully utilizes appearance information and performs better than the multi-teachers which lack texture and color information to discriminate different human subjects. Base on these observations, we can validate that wearable-sensors and vision-sensors modalities are related and complementary. To be noticed, the performance drops significantly compared with that of the SAKDN when using SADN (without GSDM), this validates that the GSDM is significantly important for knowledge transfer from wearable-sensors to vision-sensors. In the cross-view setting where the appearance of human actions varies a lot because of the changing camera views, the performance of SADN (without GSDM) and the AKDN (without SP) both perform worse than the baseline. While in SAKDN, the performance increase by $3.33\%$. This shows that the GSDM and SP are both important for addressing the cross-view challenge. More importantly, our SAKDN outperforms both the multi-teacher model and the baseline model under all settings, which verifies that the SAKDN can effectively exploit complementary knowledge between wearable-sensors and vision-sensors action data and then improve the video action recognition performance in the wild. Among all the settings, the SAKDN performs significantly better than that of the SKDN, KDN, SADN, and AKDN, which demonstrates that the SPAMFM, ST, GSDM, and SP are all complementary and essential.

To get an intuitive understanding of the effectiveness of “ablation” video frames and what Graph-guided Semantically Discriminative Mapping (GSDM) learns, the GSDM generated by the student network (video modality) in the UTD-MHAD dataset is visualized in Fig. 6. To facilitate visualization, the GSDM of the layer inception5b is resized to the input size and then used to weight each channel of the corresponding input video frames. These examples in Fig. 6 indicate that the GSDM can effectively highlight the important regions for predicting the semantic concept, which can be considered as an effective intermediate visual pattern for knowledge distillation.

To validate whether our SAKDN could generalize to different selected layers between teacher and student networks, we evaluate the performance of the SAKDN in the UTD-MHAD dataset using different combinations of $\mathcal{L}_{distill}^{T}$ and $\mathcal{L}_{distill}^{S}$ in Eq. (21) and (22). In addition, we use different backbones (BNInception, ResNet18, and ResNet50) for the student network to measure the generalization ability of SAKDN in different student backbones, as shown in Table VIII. For BNIneption, the performances of using different transfer layers are all better than that of the baseline ($94.87\%$). This validates that our SAKDN can conduct knowledge distillation across different layers between teacher and student networks. To be noticed, for the same backbone, the performance gap of using different numbers of transfer layers is marginal. This shows that our SAKDN can generalize well to different levels of transfer layers. And the performance is the best when we use $\mathcal{L}_{distill}^{T}=\{\textrm{c}_{1}^{2},\textrm{c}_{2}^{2},\textrm{c}_{3}^{3},\textrm{c}_{4}^{3},\textrm{c}_{5}^{3}\}$ and $\mathcal{L}_{distill}^{S}=\{\textrm{c}2,\textrm{I}{3c},\textrm{I}{4c},\textrm{I}{5a},\textrm{I}{5b}\}$. When using different backbones, ResNet18, ResNet50 and BNInception all perform better than the baseline method, which verifies the generalization ability of the SAKDN in different backbones. Among the ResNet18, ResNet50 and BNInception, both the ResNet50 and BNInception achieve better performance than that of the ResNet18, which attributes to the simpler architecture of the ResNet18 for representation learning.

There are three hyper-parameters $\alpha$, $\beta$ and $\gamma$ in Eq. (25). To learn how they influence the performance, we conduct the parameter sensitivity analysis on Berkeley-MHAD, UTD-MHAD, and MMAct datasets (cross-scene) using the grid-search method. The parameter $\alpha$ get its values in $\{0.01,0.1,1\}$, and parameters $\beta$ and $\gamma$ get their values in $\{0.1,1\}$. Table IX shows the performance of our SAKDN by allocating different values to the parameters $\alpha$, $\beta$, and $\gamma$. To have a more intuitive understanding of how we choose the optimal values for these parameters, we transform the table representation of parameter sensitivity in Table IX to a four-dimensional scatter diagram with the polar coordinate, shown in Fig. 7. From Table IX and Fig. 7, we can see that the optimal values for Berkeley-MHAD dataset are $\{0.1,0.1,1\}$, while for both UTD-MHAD and MMAct datasets, the optimal values are $\{0.1,1,1\}$. This means that the semantics-preserving should be attached more importance than the soft-target and GSDM, because the semantic relationship contributes a lot to multi-modal feature fusion, knowledge transfer, and representation learning for our SAKDN. For parameter $\alpha$, its optimal values are $0.1$ for all datasets. This is because the soft-target loss only conducts knowledge distillation at the last fully-connected layers, while for the GSDM and SP loss, they contribute to the knowledge transfer throughout all layers. For parameter $\beta$, the optimal value for the Berkeley-MHAD dataset is $0.1$, while for both UTD-MHAD and MMAct datasets, the optimal values are $1$. Since the Berkeley-MHAD dataset has six teacher modalities, which is larger than that of the UTD-MHAD and MMAct datasets. Therefore, $\beta$ should be set to a moderate value to avoid overfitting to any one of the teacher modalities when conducting knowledge distillation. The value of $\gamma$ is set to $1$ for all datasets.

To show the training curves of the proposed method, we plot the training loss and accuracy for Berkeley-MHAD, UTD-MHAD, and MMAct datasets, as shown in Fig. 8. The training loss steadily decreases and converges to a smaller value with the training iteration increasing. At the same time, the training accuracy increases with the number of training iterations increasing. This validates that our proposed loss function is effective with good convergence.

To gain a better understanding about which space-time regions contribute to our predictions, we follow the Class Activation Map (CAM) technique [85] to visualize the energy of the last convolutional layer in the student network, before the global max and average pooling. We show some true examples and fail examples on Berkeley-MHAD, UTD-MHAD, and MMAct datasets to further analyze our proposed method. Fig. 9 depicts computed heat maps superimposed over sample video frames. In most cases, these examples portray a strong correlation between highly activated regions and the dominant movement in the scene, even when performing complex interactive actions in the presence of significant camera motion, background clutter, occlusion, and appearance variation. For example, in the fourth row, our model can capture dominant moving regions under different camera views. In the fifth row, with significant background clutter, occlusion, and appearance variation, our model can still concentrate on the body motion. These visualization results validate that our proposed model takes full advantage of both the vision-sensor modality and wearable-sensor modality in addressing the problem of background clutter, occlusion, and appearance variation, and thus is able to focus on the salient movement under these challenges. To be noticed, in some extremely challenging cases, our model fails to capture true moving regions in the video frames. For example, in the false examples, our model mistakes some background objects (pot plant, heavy luggage) as the body subjects due to the high appearance similarity, severe background clutter, and occlusion.