Interactive Knowledge Distillation for Image Classification

In general, the conventional, feature-based knowledge distillation methods [19, 25, 13, 18, 15] decompose the student network and the teacher one into $M$ modules. The student network is trained to minimize the following loss function:

where $\gamma(\cdot,\cdot)$ is the task-specific loss. $y$ represents the ground truth. $\hat{y}$ is the predicted result of the student network. $F_{m}^{S}$ and $F_{m}^{T}$ denote the transformed output of the student network and the one of the teacher network produced by the $m$-th module. $\lambda_{m}$ is a tunable balance factor to balancing the different losses. $\varphi_{m}(\cdot,\cdot)$ is the $m$-th distillation loss to narrowing down the difference between $F_{m}^{S}$ and $F_{m}^{T}$. $\varphi_{m}(\cdot,\cdot)$ varies from method to method because of different definitions of knowledge. As can be seen in Eq. 1, the conventional knowledge distillation methods essentially aim at forcing the student to mimic the teacher’s representation space.

To achieve the interaction between the student network and the teacher network, our IAKD randomly swaps in the teacher blocks to replace the student blocks at each iteration. The swapped-in teacher blocks can respond to the output of the previous student block and then provide better features to motivate the next student block. To make all student blocks fully participate in the distillation process, after each training iteration, we put back the replaced student blocks to their original positions in the student network. Due to the return of replaced student blocks, the architecture of the student is kept consistent for the next swapping-in operation.

The interaction level between the teacher network and the student network is controlled by $p_{i}$. The smaller value of $p_{i}$ indicates the student network has more interaction with the corresponding teacher blocks. For simplicity, we assume $p_{1}=p_{2}=\dots=p_{i}=\dots=p_{B}$, which suggests that every hybrid block shares the same interaction rule. To properly utilize the interaction mechanism, we propose three kinds of probability schedules for the change of $p_{i}$: 1) uniform schedule, 2) linear growth schedule and 3) review schedule. These schedules are discussed in detail as follows.

To investigate the effectiveness of different probability schedules, we conduct experiments on CIFAR-10 [22] and CIFAR-100 [22] datasets. The CIFAR-10 dataset consists of 60000 images of size $32\times 32$ in 10 classes. The CIFAR-100 dataset [22] has 60000 images of size $32\times 32$ from 100 classes. For all experiments in this subsection, we use SGD optimizer with momentum 0.9 to optimize the hybrid network. We set batchsize to 128 and set weight decay to $1\times 10^{-4}$. The hybrid network is trained for 200 epochs. The learning rate starts at 0.1 and is multiplied by 0.1 at 100, 150 epochs. Images of size $32\times 32$ are randomly cropped from zero-padded $40\times 40$ images. Each image is horizontally flipped with a probability of 0.5 for data augmentation.

In this subsection, the training settings are the same with those in Sec. IV-A. Each experiment is repeated 5 times, and we take the mean value as the final result.

From the view of the conventional, non-interactive knowledge distillation methods, the end of each set of swapped-in teacher blocks can be regarded as the knowledge distillation point where the knowledge of the teacher network is transferred to the student network. Randomly swapping in the teacher blocks to replace the student blocks implicitly increases the number of knowledge distillation points. To verify more knowledge distillation points do not necessarily lead to more performance improvement and may even harm the student performance, we introduce more connections between the teacher and the student to multi-connection knowledge based methods (i.e.,AT [13] and FSP [18]). Accordingly, we modify AT to transfer attention maps at the end of each student block rather than the end of each group, and name it as 9-point AT because there are totally 9 knowledge distillation points. Similarly, we modify FSP to calculate the Gramian matrix by the input and the output of each student block instead of each group, and name it as 9-point FSP. The experiments are conducted by ResNet-44/ResNet-26 pair on CIFAR-10 and CIFAR-100 datasets.

Experimental results are shown in Tab V. It turns out, for conventional, non-interactive knowledge distillation methods, more knowledge distillation points do not necessarily lead to more performance improvement. For AT, more distillation points even make the distillation performance worse than the performance of the original AT. This is because excessive supervision causes a hard constraint for the student network. However, as shown in Tab. I and Tab. II, introducing more possible distillation points even improves the distillation performance of our proposed IAKD.

We compare our proposed IAKD-R with other representative state-of-the-art knowledge distillation methods: HKD [12], AT [13], FSP [18], Feature distillation [15], AB [19], VID-I [25] and RKD-DA [14]. As the discussion above, all these comparison methods are non-interactive. We conduct experiments on three datasets: CIFAR-10 [22], CIFAR-100 [22], and TinyImageNet [32]. TinyImageNet contains images of size $64\times 64$ with 200 classes. Each experiment is repeated 5 times, and we take the mean value as the final report result. For other compared methods, we directly use the author-provided codes if publicly available, or implement those methods based on the original paper. The hyper-parameters of contrastive methods are listed as follows.

1. 1.

   HKD [12]: temperature $\mathcal{T}$=$4$, balance factor: $\alpha$=0.9 (Eq. 20 in [16]).

2. 2.

   AT [13]: balance factor: $\beta$=1000 (Eq. 2 in [13]).

3. 3.

   FSP [18]: No balance factor for the task-specific and the distillation loss because the distillation is in the initialization stage where only the distillation loss applies. In the initialization stage, we set $\lambda_{i}$, the hyper-parameter to balance each FSP loss term in Eq. 2 in [18], to 1.

4. 4.

   AB [19]: No balance factor for the task-specific loss and the distillation loss because the distillation is in the initialization stage where only the distillation loss applies. In the initialization stage, there are three hyper-parameters: $\lambda_{1}$, $\lambda_{2}$ and $\lambda_{3}$ to balance the different terms of the distillation loss. We set $\lambda_{1}$, $\lambda_{2}$ and $\lambda_{3}$ to $\frac{1}{4}\times 10^{-3}$, $\frac{1}{2}\times 10^{-3}$, and $1\times 10^{-3}$, respectively (See [19] for explanations of these hyper-parameters).

5. 5.

   VID-I [25]: balance factor: $\lambda_{k}$=1 (Eq. 2 in [25]).

6. 6.

   RKD-DA [14]: balance factors: $\lambda_{RKD-D}$=25 and $\lambda_{RKD-A}$=50 (See [14] for explanations of these hyper-parameters).

7. 7.

   Feature distillation [15]: balance factor: $\alpha$=$10^{-3}$ (Eq. 6 in [15]). In the distillation loss function in Eq. 5 in [15], the loss is multiplied by 2 when the spatial size of the feature maps is decreased by twice as the input moves from shallow to deep layers.

For fair comparison, the optimization configurations are identical for all methods on one dataset.

For CIFAR-10 and CIFAR-100, we use the same optimization configurations as described in Sec. IV-A. Specially, for two-stage methods like FSP and AB, 50 epochs are used for initialization and 150 epochs are used for classification training. The learning rate at the initialization stage is 0.1 and 0.001 for AB and FSP, respectively. For AB and FSP, the initial learning rate at the classification training stage is 0.1, and is multiplied by 0.1 at 75, 110 epochs. For TinyImageNet, we employ ResNet-80/ResNet-26 as the T/S pair. Horizontal flipping is applied for data augmentation. We optimize the network using SGD with batchsize 128, momentum 0.9 and weight decay $1\times 10^{-4}$. The number of total training epochs is 300. The initial learning rate is 0.1 and is multiplied by 0.2 at 60, 120, 160, 200, 250 epochs. Specially, for FSP and AB, 50 epochs are used for initialization and 250 epochs are used for classification training. The learning rate at the initialization stage is 0.1 and 0.001 for AB and FSP, respectively. For AB and FSP, the initial learning rate at the classification training is 0.1, and is multiplied by 0.2 at 60, 120, 160, 200 epochs.

For the proposed IAKD, we use the review schedule as the final probability schedule setting. $p_{start}$ is set to 0.9 for comparison on CIFAR-10, and is set to 0.1 for comparison on CIFAR-100 and TinyImageNet. The configurations are based on the experimental results and arguments in Sec. IV-A3.

In Tab. VI, we list the comparison results on CIFAR-10, CIFAR-100, and TinyImageNet by different methods. As we can see, the proposed IAKD improves the performance of the student network more effectively than most contrastive methods. Meanwhile, the distillation result of IAKD is competitive with HKD on CIFAR-10. We then combine HKD with our proposed IAKD through adding distillation loss to Eq. 5, and denote it as IAKD-R+HKD. Surprisingly, IAKD-R+HKD even outperforms the performance of the original teacher on CIFAR-10. On CIFAR-100 and TinyImageNet, IAKD-R+HKD outperforms all other contrastive methods. IAKD-R also shows highly competitive results on CIFAR-100 and TinyImageNet. In addition, our proposed method, robust as we can see, consistently achieves significant performance improvement on three datasets while most other contrastive methods do not.

As mentioned in Sec. I, our proposed IAKD discards the cumbersome knowledge transformation process and gets rid of the feature extraction of the entire teacher network, we compare the training time of different knowledge distillation methods, using ResNet-80/ResNet-26 teacher-student pair on TinyImageNet dataset. We conduct training time comparison on a single NVIDIA RTX 2080Ti GPU with Intel Core i7-8700 CPU and PyTorch framework. As shown in Tab. VII, our proposed method gives a highly competitive result in contrast to other methods. It only needs 4 hours and 40 minutes to complete the training process of the student network, which reflects that the high training efficiency of our proposed IAKD.

To ensure the applicability of IAKD, we explore more architecture combinations on CIFAR-100, as shown in Tab. VIII. We replace the “Conv3$\times$3-BN-ReLU” with “group Conv3$\times$3-BN-ReLU-Conv1$\times$1-BN-ReLU” to achieve the cheap ResNet, as suggested by [33]. The experimental settings are the same as those described in Sec. IV-D. As we can see from Tab. VIII, in all the architecture combinations, IAKD-R and IAKD-R+HKD can consistently achieve performance improvement over the student network. On the other hand, contrastive methods give different knowledge distillation performance in different settings. In the case of (a), only IAKD-R and IAKD-R+HKD can successfully achieve performance improvement over the student cheap ResNet-26. AT and RKD-DA fail to improve the student performance in the case of (b). RKD-DA and AB fail to achieve performance improvement in the case of (d). Therefore, the proposed IAKD, a novel knowledge distillation method without additional distillation losses, can also achieve promising performance improvement in different teacher/student settings.