Cross-Layer Distillation with Semantic Calibration

Nowadays complex deep neural networks with millions of parameters or an ensemble of such models has achieved great success in various settings [27, 28, 29, 30, 31], but it is still a huge challenge to efficiently deploy them in applications with limited computational and storage resources [8, 2]. There are some pioneering efforts, collectively known as model compression, proposing a series of feasible solutions to this problem, such as knowledge distillation [13, 9], low-rank approximation [8, 32], quantization [33, 11] and pruning [10, 34]. Among these techniques, knowledge distillation is perhaps the easiest and the most hardware-friendly one for implementation due to its minimal modification to the regular training procedure.

Given a lightweight student model, the aim of vanilla KD is to improve its generalization ability by training to match the predictions, or soft targets, from a pre-trained cumbersome teacher model [9]. The intuitive explanation for the success of KD is that compared to discrete labels, fine-grained information among different categories in soft targets may provide extra supervision to improve the student model performance [9, 14]. A new interpretation for this improvement is that soft targets act as a learned label smoothing regularization to keep the student model from producing over-confident predictions [15]. To save the expense of pre-training, some online variants have been proposed to explore cost-effective collaborative learning [35, 36, 37]. By training a group of student models and encouraging each one to distill the knowledge in group-derived soft targets as well as ground-truth labels, these approaches could even surpass the vanilla KD in some teacher-student combinations [36, 37].

Rather than only formalizing knowledge in a highly abstract form like predictions, recent methods attempted to leverage information contained in intermediate layers by designing elaborate knowledge representations. A bunch of techniques have been developed for this purpose, such as aligning hidden layer responses called hints [16], encouraging the similar patterns to be elicited in the spatial attention maps [17] and maximizing the mutual information through variational inference [19]. Since the spatial dimensions of feature maps or their transformations from teacher and student models usually vary, a straightforward step before the subsequent loss calculation is to match them by adding convolutional layers or pooling layers [16, 17, 19]. Some other parameter-free alternatives capture the transferred knowledge by crude pairwise similarity matrices [18] or hybrid kernel formulations built on them to avoid mismatch, such as cosine-based and t-student transformations [20]. With these pre-defined representations, all above approaches perform knowledge transfer with certain hand-crafted layer associations [16, 18, 17, 19, 20, 21]. Unfortunately, as pointed in the previous work [20], these hard associations would make the student model suffer from negative regularization, limiting the effectiveness of feature-map distillation. We will discuss this negative regularization in more depth in Section 4.3.1.

Although a large number of approaches explored various knowledge representations over the past few years [16, 18, 17, 19, 20, 21], very few have considered to leverage these representations in a better fashion. The most related approach to ours, which is proposed recently, works on channel or group-wise association within given teacher-student layer pairs [21, 38]. The main difference is that we exploit feature maps from a more general cross-layer level rather than channel or group level.

Learning layer association has also been studied in transfer learning. A previous work learns the association weights between source and target networks by additional meta-networks with bi-level optimization [39]. Our approach differs from this one since feature maps of teacher-student layer pairs are both incorporated to derive the association weights while only those of the source network are used in [39]. Another advantage is that our association weights are trained end-to-end with the original student network, i.e. no need for carefully tuned bi-level optimization.

Feature embeddings are good substitutes for feature maps to distill knowledge in intermediate layers. Compared to high dimensional feature maps, these vectorized embeddings obtained at the penultimate layer are generally more tractable. Meanwhile, feature embeddings preserve more structural information than the final predictions used in the vanilla KD. Therefore, a variety of feature-embedding distillation approaches have been proposed recently [40, 41, 42, 43, 44, 45]. In these approaches, a commonly used procedure is to organize all instances by a relational graph for knowledge transfer [40, 43, 41, 42]. The main difference among them lies on how the edge weights are calculated. Typical choices include cosine kernel [40], truncated Gaussian RBF kernel [43], or combination of distance-wise and angle-wise potential functions [41]. In contrast to pairwise transfer, some approaches formulate knowledge distillation from the perspective of contrastive learning to capture higher-order dependencies in the representation space [44, 45].

Although our approach mainly focuses on feature-map distillation, it is also compatible with the state-of-the-art feature embedding distillation approach for further performance improvement.

In this section, we briefly recap the basic concepts of vanilla knowledge distillation as well as provide necessary notations for the following illustration.

Given a training dataset $\mathcal{D}=\{\left(\bm{x}_{i},\bm{y}_{i}\right)\}_{i=1}^{N}$ consisting of $N$ instances from $K$ categories and a powerful teacher model pre-trained on the dataset $\mathcal{D}$, the goal of vanilla KD is reusing the same dataset to train another simple student model with cheaper computational and storage demand.

For a mini-batch of instances with size $b$, we denote the outputs of each target layer $t_{l}$ and student layer $s_{l}$ as $F^{t}_{t_{l}}\in\mathbb{R}^{b\times c_{t_{l}}\times h_{t_{l}}\times w_{t_{l}}}$ and $F^{s}_{s_{l}}\in\mathbb{R}^{b\times c_{s_{l}}\times h_{s_{l}}\times w_{s_{l}}}$, respectively, where $c$ is the channel dimension, $h$ and $w$ are the spatial dimensions, superscript $t$ and $s$ reflect the corresponding teacher and student models. These outputs are also named as feature maps. $t_{l}/s_{l}$ ranges from $1$ to the total number of intermediate layers $t_{L}/s_{L}$. Note that $t_{L}$ and $s_{L}$ may be different especially when the teacher and student models adopt different architectures. The representations at the penultimate layer from teacher and student models are denoted as $F^{t}_{t_{L}+1}$ and $F^{s}_{s_{L}+1}$, which are mainly used in feature-embedding distillation. Take the student model as an example, outputs of the last fully connected layer $g(\cdot)$ are known as logits $\bm{g}^{s}_{i}=g(F^{s}_{s_{L}+1}[i])\in\mathbb{R}^{K}$ and the predicted probabilities are calculated with a softmax layer built on the logits, i.e., $\sigma(\bm{g}^{s}_{i}/T)$ with the temperature $T$ usually equals to $1$. The notation $F^{s}_{s_{l}}[i]$ denotes the output of student layer $s_{l}$ for the $i$-th instance and is a shorthand for $F^{s}_{s_{l}}[i,:,:,:]$.

For classification tasks, in addition to regular cross entropy loss ($\mathcal{L}_{CE}$) between the predicted probabilities $\sigma(\bm{g}^{s}_{i})$ and the one-hot label $\bm{y}_{i}$ of each training sample, classic KD [9] incorporates another alignment loss to encourage the minimization of Kullback-Leibler (KL) divergence [46] between $\sigma(\bm{g}^{s}_{i}/T)$ and soft targets $\sigma(\bm{g}^{t}_{i}/T)$ predicted by the teacher model

where $T$ is a hyper-parameter and a higher $T$ leads to more considerable softening effect. The coefficient $T^{2}$ in the loss function is used to ensure that the gradient magnitude of the $\mathcal{L}_{KL}$ part keeps roughly unchanged when the temperature is larger than one [9].

As mentioned earlier, feature maps of a teacher model are valuable for helping a student model achieve better performance. Recently proposed feature-map distillation approaches can be summarized as adding the following loss to Equation (1) for each mini-batch with size $b$

leading to the overall loss as

The functions $\mathrm{Trans^{t}(\cdot)}$ and $\mathrm{Trans^{s}(\cdot)}$ in Equation (2) are used to transform feature maps into particular representations. The layer association set $\mathcal{C}$ of existing approaches are designed as one-pair selection [16, 18] or one-to-one match [17, 19, 20, 21]. As for one-pair selection, the candidate set $\mathcal{C}=\{(s_{r},t_{r})\}$ includes only one pair of a manually specified student layer $s_{r}$ and a teacher layer $t_{r}$, while for one-to-one match, the candidate set $\mathcal{C}=\{(1,1),...,\left(\mathrm{min}(s_{L},t_{L}),\mathrm{min}(s_{L},t_{L})\right)\}$ includes $|\mathcal{C}|=\mathrm{min}(s_{L},t_{L})$ elements. With these associated layer pairs, the loss in Equation (2) is calculated by a certain distance function $\mathrm{Dist(\cdot,\cdot)}$. A common practice is to use Mean-Square-Error (MSE) [16] sometimes with a normalization pre-processing [17, 18, 21]. The hyper-parameter $\beta$ in Equation (3) is used to balance the two individual loss terms. The detailed comparison of several representative approaches is provided in Table I.

As shown in Table I, FitNet [16] adds a convolutional transformation, i.e., $\mathrm{Conv(\cdot)}$ on top of a certain intermediate layer of the student model, while keeping feature maps from the teacher model unchanged by identity transformation $\mathrm{I(\cdot)}$. AT [17] encourages the student to mimic spatial attention maps of the teacher by channel attention. VID [19] formulates knowledge transfer as Mutual Information (MI) maximization between feature maps and expresses the distance function as Negative Log-Likelihood (NLL). The transferred knowledge can also be captured by crude similarity matrices or hybrid kernel transformations build on them [18, 20]. MGD [21] makes the channel dimensions of feature maps in candidate layer pairs become matched by an assignment module $\rho(\cdot)$. Note that almost twice as much computational and storage resources are needed for HKD [20]. Since it first transfers feature embeddings from the teacher to an auxiliary model, which follows the same architecture as the student but with twice parameters per layer, then performs feature-map distillation from the auxiliary to the student model.

All of the above approaches perform knowledge transfer based on fixed associations between assigned teacher-student layer pairs, which may cause the loss of useful information. Take one-to-one match as an example, extra layers are discarded when $s_{L}$ and $t_{L}$ differ. Moreover, forcing feature maps from the same layer depths to be aligned may result in suboptimal associations, since a better choice for the student layer could from different or multiple target teacher layers.

To solve these problems, we propose a new cross-layer knowledge distillation to promote the exploitation of feature-map representations. Based on our learned layer association weights, simple convolutional $\mathrm{Trans^{t}(\cdot)}$, $\mathrm{Trans^{s}(\cdot)}$ and $\mathrm{MSE}$ distance are enough to achieve state-of-the-art results.

In our approach, each student layer is automatically associated with those semantic-related target layers by attention allocation, as illustrated in Figure 1. Training with soft association weights encourages the student model to collect and integrate multi-layer information to obtain a more suitable regularization. Moreover, SemCKD is readily applicable to situations where the numbers of candidate layers in teacher and student models are different.

The learned association set $\mathcal{C}$ in SemCKD is denoted as

with the corresponding weight satisfies $\sum_{t_{l}=1}^{t_{L}}\bm{\alpha}_{(s_{l},t_{l})}=\bm{1},$ $\forall\,s_{l}\in[1,...,s_{L}]$. The weights for a mini-batch of instances $\bm{\alpha}_{(s_{l},t_{l})}\in\mathbb{R}^{b\times 1}$ represent the extent to which the target layer $t_{l}$ is attended in deriving the semantic-aware guidance for the student layer $s_{l}$. Each instance will hold its own association weight $\alpha_{(s_{l},t_{l})}[i]$ for the layer pair $(s_{l},t_{l})$, which is calculated by function $\mathcal{F}(\cdot,\cdot)$ of given feature maps

The detailed design of $\mathcal{F}(\cdot,\cdot)$ will be elaborated later. Given these association weights, feature maps of each student layer are projected into $t_{L}$ individual tensors to align with the spatial dimensions of those from each target layer:

with $F^{s^{\prime}}_{t_{l}}\in\mathbb{R}^{b\times c_{t_{l}}\times h_{t_{l}}\times w_{t_{l}}}$. Each function $\mathrm{Proj(\cdot,\cdot)}$ includes a stack of three layers with $1\times 1$, $3\times 3$ and $1\times 1$ convolutions to meet the demand of capability for effective transformation¹¹1In practice, a pooling operation is first used to reconcile the height and weight dimensions of $F^{t}_{s_{l}}$ and $F^{s}_{s_{l}}$ before projections to reduce computational consumption..

Loss function. For a mini-batch of instances with size $b$, the trainable student model and fixed teacher model will produce several feature maps across multiple layers, i.e., $F^{s}_{s_{1}},...,F^{s}_{s_{L}}$ and $F^{t}_{t_{1}},...,F^{t}_{t_{L}}$, respectively. After attention allocation and dimensional projections, the feature-map distillation loss ($\mathcal{L}_{FMD}$) of SemCKD is obtained by simply using $\mathrm{MSE}$ for distance function

where feature maps from each student layer are transformed by a projection function as Equation (6) while those from the target layers remain unchanged by identity transformation $\mathrm{Trans^{t}(\cdot)}=\mathrm{I(\cdot)}$. Equipped with the learned layer association weights, the total loss is aggregated by a weighted summation of each individual distance among the feature maps from candidate teacher-student layer pairs. Note that FitNet [16] is a special case of SemCKD by fixing $\alpha_{(s_{l},t_{l})}[i]$ to $1$ for certain $(s_{l},t_{l})$ layer pair and $0$ for the rest.

Attention Allocation. Now we provide concrete steps to obtain the association weights. As pointed in previous works, feature representations contained in a trained neural network are progressively more abstract as the layer depth increases [1, 47, 48]. Thus, layer semantics in teacher and student models usually varies. Existing hand-crafted strategies, which did not take this factor into consideration, may not suffice due to negative effects caused by semantic mismatched layers [20]. To further improve the performance of feature-map distillation, each student layer had better associate with the most semantic-related target layers to derive its own regularization.

Layer association based on attention mechanism provides a potentially feasible solution for this goal. Based on the observation that feature maps produced by similar instances probably become clustered at separate granularity in different intermediate layers [18], we regard the proximity of pairwise similarity matrices as a good measurement of the inherent semantic similarity. These matrices are calculated as

where $\mathrm{R(\cdot)}:\mathbb{R}^{b\times c\times h\times w}\mapsto\mathbb{R}^{b\times(c\cdot h\cdot w)}$ is a reshaping operation, and therefore $A^{s}_{s_{l}}$ and $A^{t}_{t_{l}}$ are $b\times b$ matrices.

Inspired by the self-attention framework [49], we separately project the pairwise similarity matrices of each student layer and associated target layers into two subspaces by a Multi-Layer Perceptron ($\mathrm{MLP}$) to alleviate the effect of noise and sparseness. For the $i$-th instance

The parameters of $\mathrm{MLP}_{Q}(\cdot)$ and $\mathrm{MLP}_{K}(\cdot)$ are learned during training to generate query and key vectors and shared by all instances. Then, $\alpha_{(s_{l},t_{l})}[i]$ is calculated as follows

where $\mathcal{F}(\cdot,\cdot)$ for calculating the attention weights is named as Embeded Gaussian operation in the non-local block [50]. Attention-based allocation provides a possible way to suppress negative effects caused by layer mismatch and integrate positive guidance from multiple target layers, which is supported by the theoretical discussion in Section 3.4 and empirical evidence in Section 4.3. Although our proposed SemCKD distills only the knowledge contained in intermediate layers, its performance can be further boosted by incorporating additional regularization, e.g., feature-embedding distillation. The full training procedure with the proposed semantic calibration formulation is summarized in Algorithm 1.

In this section, we present a theoretical connection between our proposed attention allocation and the famous Orthogonal Procrustes problem [25, 26, 51].

The key design of attention allocation is function $\mathcal{F}(\cdot,\cdot)$, which measures the strength of association weights and should satisfy non-negative constrain. In our approach, this constrain is naturally ensured since each used feature maps have passed a ReLU activation function ($\max(0,x)$). Thus, we can safely replace the Embedded Gaussian with a computation-efficient Dot Production operation in Equation (10), which has achieved empirical success in the previous work [50]. Additional, we omit the constant normalization term suggested by [50] in the following discussion for simplicity.

Suppose that the $\mathrm{MLP}(\cdot)$ in Equation (9) equals to identity transformation²²2A similar result will be obtained for the case with the original $\mathrm{MLP}(\cdot)$., and the attention weight $w_{(s_{l},t_{l})}$ for candidate layer pair $(s_{l},t_{l})$ is calculated by averaging the weights of all instances

where $\mathrm{R}(F^{t}_{t_{l}}),\mathrm{R}(F^{s}_{s_{l}})\in\mathbb{R}^{b\times(c\cdot h\cdot w)}$³³3We suppose the vectorized feature maps from teacher and student models sharing the same dimension $(c\cdot h\cdot w)$ in what follows.. From the last Equation of (11), we can see that $w_{(s_{l},t_{l})}$ actually measures the extent to which the extracted feature maps from teacher and student models are shared or similar. We then establish a connection between this weight and the optimal value of the Orthogonal Procrustes problem.

Proof. Since $\sigma_{i}\geq 0$, the inequality $\left(\sum_{i=1}^{p}\sigma_{i}\right)^{2}=\sum_{i=1}^{p}\sigma_{i}^{2}+2\sum_{i=2}^{p}\sum_{j=1}^{i-1}\sigma_{i}\sigma_{j}\geq\sum_{i=1}^{p}\sigma_{i}^{2}$ always holds, which implies that $\|\mathrm{X}\|_{\mathrm{F}}\leq\|\mathrm{X}\|_{*}$. Based on the Cauchy-Schwarz inequality $\left(\sum_{i=1}^{p}\sigma_{i}\delta_{i}\right)^{2}\leq\left(\sum_{i=1}^{p}\sigma_{i}^{2}\right)\left(\sum_{i=1}^{p}\delta_{i}^{2}\right)$ and let $\delta_{i}\equiv 1$, we can obtain that $\|\mathrm{X}\|_{*}\leq\sqrt{p}\|\mathrm{X}\|_{\mathrm{F}}$. ∎

The Orthogonal Procrustes problem. This optimization problem seeks an orthogonal transformation $O$ to make the matrix $A\in\mathbb{R}^{m\times n}$ align with the matrix $B\in\mathbb{R}^{m\times n}$ such that Frobenius norm of the residual error matrix $E$ is minimized [25, 26]. The mathematical expression is presented as follows

Since $||B-AO||_{\mathrm{F}}^{2}=||B||_{\mathrm{F}}^{2}+||A||_{\mathrm{F}}^{2}-2\mathrm{Tr}(B^{\mathrm{T}}AO)$ and $||B||_{\mathrm{F}}^{2}$, $||A||_{\mathrm{F}}^{2}$ are constants, the objective of Equation (13) can be rewritten as an equivalent one

By applying the method of Lagrange multipliers [52], we can convert this constrained problem into an unconstrained one to find the solution $\hat{O}=UV^{\text{T}}$, where $U$ and $V$ are obtained by singular value decomposition $A^{\mathrm{T}}B=U\Sigma V^{\mathrm{T}}$ [51]. Thus, the maximum objective of Equation (14) is

Our crucial observation is that if we interpret ${E^{\prime}}^{2}$ as a measurement of inherent semantic similarity for feature maps, our designed layer association weights will naturally achieve semantic calibration.

Given vectorized feature maps from the teacher and student models $\mathrm{R}(F^{t}_{t_{l}})$ and $\mathrm{R}(F^{s}_{s_{l}})$, the irreducible error of orthogonal transformation from $\mathrm{R}(F^{s}_{s_{l}})$ to $\mathrm{R}(F^{t}_{t_{l}})$, as discussed in the above, is related to ${E^{\prime}}^{2}$, or $\|\mathrm{R}(F^{t}_{t_{l}})^{\mathrm{T}}\cdot\mathrm{R}(F^{s}_{s_{l}})\|_{\mathrm{*}}^{2}$. The larger ${E^{\prime}}^{2}$ means that these two representations are more similar from the perspective of orthogonal transformation. Since the lemma 1 implies that increasing $||X||_{\mathrm{*}}$ tends to increase $||X||_{\mathrm{F}}$ and vice versa, $\|R(F^{t}_{t_{l}})^{\mathrm{T}}\cdot\mathrm{R}(F^{s}_{s_{l}})\|_{\mathrm{F}}^{2}$ will tend to be larger if ${E^{\prime}}^{2}$ becomes larger. That is to say, attention allocation achieve semantic calibration by assigning larger weight $w_{(s_{l},t_{l})}$ for a certain teacher-student layer pair once the generated feature maps of student layer $\mathrm{R}(F^{s}_{s_{l}})$ can be orthogonally transformed into those of the associated target layer $\mathrm{R}(F^{t}_{t_{l}})$ with relatively lower error.

A simple yet effective variant to further improve the performance of SemCKD is presented in this section. Similar to softening predicted probabilities in the vanilla KD as Equation (1), we find that softening attention in Equation (10) is also helpful. But the reason behind these two seemingly similar steps are completely different. The former is to capture more structural information among different classes while the latter is to smooth the gradient direction guided by different target layers for each student layer in training.

In this variant SemCKD_τ, we replace Equation (10) as

Note that $\tau$ is set as a constant to counteract extremely small gradient during training in the original self-attention framework [49], but it is regarded as a new hyper-parameter in SemCKD_τ. We will discuss the improvement over SemCKD with various $\tau$ in more detail in Section 4.5.

Table II and III give the Top-1 test accuracy (%) on CIFAR-100 based on twelve network combinations, which consists of two homogeneous settings, i.e., the teacher and student share similar architectures (VGG-8/13, ResNet-8x4/32x4), and ten heterogeneous settings. Each column apart from the first and last rows includes the results of a certain student model trained by various approaches under the supervision of the same teacher model. The results of the vanilla KD are also included for comparison. There are two combinations where the MGD [21] is not applicable and denoted as ’—’.

From Table II and III, we can see that SemCKD consistently achieves higher accuracy than state-of-the-art feature-map distillation approaches. The distributions of relative improvement are plotted in Figure 2. On average, SemCKD shows significantly relative improvement (60.03%) over all these approaches. Specifically, the best case happens in the “VGG-8 & ResNet-32x4” setting where the $RI$ over AT [17] is 242%. When compared to the most competitive one named as HKD [20], which relies on a costly teacher-auxiliary-student paradigm as discussed in Section 3.2, the $RI$ becomes rather small on two settings (3.93% for “ShuffleNetV2 & ResNet-32x4”, 9.98% for “MobileNetV2 & WRN-40-2”). But in general, SemCKD still relatively outperforms HKD for 45.82%, showing that our approach indeed make better use of intermediate information for effective distillation.

We also find that none of the compared approaches can consistently beats the vanilla KD on CIFAR-100, which probably due to semantic mismatch among associated layer pairs. This problem becomes especially severe for one-pair selection (FitNet method fails in 7/12 cases and MGD method fails in 8/12 cases), the situation where the number of candidate layer $s_{L}$ is larger than $t_{L}$ (4/6 of methods fail in the “ShuffleNetV2 & VGG-13” setting) and MobileNetV2 is used as the student model (5/6 or 4/6 of methods fail in the “MobileNetV2 & ResNet-32x4/WRN-40-2” settings). Nevertheless, our semantic calibration formulation helps alleviate semantic mismatch to a great extent, leading to satisfied performance of SemCKD. Similar observations are obtained in Table IV for the results on a large-scale image classification dataset.

In this section, we experimentally study the negative regularization caused by manually specified layer associations and provide some explanations for the success of SemCKD by the proposed criterion and visual evidence.

In Table VI, we present the evaluation results of five kinds of ablation studies based on the “VGG-8 & ResNet-32x4” combination. Specifically, the first kind confirms the benefit of adaptively learning attention allocation for cross-layer knowledge distillation, the second to the fourth ones confirm the benefit of each individual component for obtaining attention weights, and the last one confirms the benefit of assigning each instance with independent weights.

(1) Equal Allocation. In order to validate the effectiveness of allocating the personalized attention of each student layer to multiple target layers, equal weights assignment is applied instead. This causes a lower accuracy by 2.33% (from 75.27% to 72.94%) and a considerably larger variance by 0.74% (from 0.13% to 0.87%).

(2) w/o Projection. Rather than projecting the feature maps of each student layer to the same dimension as those in the target layers by Equation (6), we add a new $\mathrm{MLP}_{V}(\cdot)$ to project the pairwise similarity matrices of teacher-student layer pairs into another subspace to generate value vectors. Thus the Mean-Square-Error among feature maps in Equation (7) is replaced by these value vectors to calculate the overall loss, which reduces the performance by 2.76%.

(3) w/o $\mathrm{MLP}$. A simple linear transformation is used to obtain query and key vectors in Equation (9) instead of the two-layer non-linear transformation $\mathrm{MLP}(\cdot)$, which includes “Linear-ReLU-Linear-Normalization”. The 2.49% performance drop indicates that the usefulness of $\mathrm{MLP}(\cdot)$ to alleviate the effect of noise and sparseness.

(4) w/o Similarity Matrices. We skip the similarity matrices calculation in Equation (8) and directly employ feature maps of candidate layer pairs to obtain the query and key vectors. Specifically, we replace Equation (9) with $Q_{s_{l}}[i]=\mathrm{MLP}_{Q}(F^{s}_{s_{l}}[i])$, $K_{t_{l}}[i]=\mathrm{MLP}_{K}(F^{t}_{t_{l}}[i]).$ The 0.52% performance decline indicates the benefit of similarity matrices for the subsequent attention calculation. Another reason for not using feature maps is that this will cause a considerable memory cost to generate query and key vectors due to the larger spatial dimensions ($c_{s_{l}/t_{l}}\cdot h_{s_{l}/t_{l}}\cdot w_{s_{l}/t_{l}}\gg b$).

(5) Shared Allocation. Instead of learning an independent attention allocation across teacher-student layer pairs for each training instance, we make all instances share the same association weights during the whole training process. This will incur 0.40% accuracy drop.

In this section, we conduct experiments on a large number of teacher-student combinations to verify the effectiveness of softening attention in SemCKD.

As shown in Figure 5, we test the performance of each student model with different softness $\tau$ and draw an orange curve for the results of SemCKD_τ. To preserve as much information as possible in the teacher model during knowledge transfer, previous works attempt to adjust the distillation position to the front of the ReLU operation [64, 21], which is called pre-activation distillation. We also combine this operation into SemCKD_τ and name this variant as SemCKD_τ+Pre. Additionally, the results of original SemCKD are plotted as the horizontal lines for comparison.

In most cases, we can see that softening attention indeed improves the performance of SemCKD by a considerable margin. For example, when $\tau=4$, the absolute accuracy boost is 0.47% and 0.86% in the “VGG-8 & ResNet-32x4” and “MobileNetV2 & WRN-40-2” settings, respectively. Another evidence to support the necessity of softness is that there will be a significant performance degeneration when $\tau$ is less than 1. The reason is that the attention weights will be sharpened in this situation and make the overall target direction become largely influenced by a certain component. Generally, $\tau=2$ or $4$ is a satisfying initial choice in practice for a given teacher-student combination. We also find that SemCKD_τ+Pre could only outperform SemCKD_τ slightly but will be inferior to SemCKD distinctly when “MobileNetV2” is used for the student model.

Knowledge transfer based on feature embeddings of the penultimate layer is another alternative to improve the generalization ability of student models. We compare the performance of several newly proposed approaches on the same twelve network combinations as Table II and Table III. Additionally, we report the results of student models trained by simply adding the loss of CRD [44] into the original one of SemCKD without tuning any hyper-parameter.

From Table IX, we can observe that the performance can be further boosted by training with “SemCKD+CRD” in all cases, which confirms that our approach holds a very satisfying property to be highly compatible with the state-of-the-art feature-embedding distillation approach.

Finally, we evaluate the impact of hyper-parameter $\beta$ on the performance. We compare three representative knowledge distillation approaches, including logits distillation (KD [9]), feature-embedding distillation (CRD [44]) and feature-map distillation (SemCKD). The range of hyper-parameter $\beta$ for SemCKD is set as 100 to 1100 at equal interval of 100, while the hyper-parameter $\beta$ for CRD ranges from 0.5 to 1.5 at equal interval of 0.1, adopting the same search space as the original paper [44]. Note that the hyper-parameter $\beta$ always equals to $0$ for the vanilla KD, leading to a horizontal line in Figure 7.

It is seen that SemCKD achieves the best results in all cases and outperforms CRD at about 1.73% absolute accuracy for the default hyper-parameter setting. Figure 7 also shows that the performance of SemCKD keeps very stable after the hyper-parameter $\beta$ is greater than 400, which indicates that our proposed method works reasonably well in a wide range of search space for the hyper-parameter $\beta$.