Distilling the Undistillable: Learning from a Nasty Teacher

Knowledge Distillation (KD): KD methods train a smaller student network, $\theta_{s}$, with the outputs of a typically large pre-trained teacher network, $\theta_{t}$ alongside the ground-truth labels. Given an input image x, the output logits of student given by $\textbf{z}_{s}=\theta_{s}(\textbf{x})$ and teacher logits given by $\textbf{z}_{t}=\theta_{t}(\textbf{x})$, a temperature parameter $\tau$ is used to soften the logits and obtain a transformed output probability vector using the softmax function:

where $\textbf{y}_{s}$ and $\textbf{y}_{t}$ are the new output probability vectors of the student and teacher, respectively. The final loss function used to train the student model is given by:

where KL stands for Kullback-Leibler divergence, $(\mathcal{L}_{CE})$ represents standard cross-entropy loss, and $\lambda,\alpha$ are two hyperparameters to control the importance of the loss function terms ($\lambda=\tau^{2}$ generally).

KD-based Stealing: Given a stealer (or student) model, denoted by $\theta_{s}$, and a victim (or teacher) $\theta_{t}$, the stealer is said to succeed in stealing knowledge using KD if by using the input-output information of the victim, it can grasp some additional knowledge which is not accessible in the victim’s absence. As stated in [22], this phenomenon can be measured in terms of difference in maximum accuracy of stealer with and without stealing from victim. Formally, stealing is said to happen if:

where the left expression refers to the accuracy with stealing, and the right one refers to accuracy without stealing.

Defense against KD based Stealing: Following [22], we consider a method $M$ as defense, if it degrades the student’s tendency (or accuracy) of stealing. Formally, considering the accuracy of stealer without defense $M$ as $Acc_{w}(KD(\theta_{s},\theta_{t}))$ and with defense as $Acc_{wm}(KD(M(\theta_{t},\theta_{t})))$, $M$ is said to be a defense if:

Nasty Teacher(NT)[22]: The Nasty Teacher methodology transforms the original model to a model which has accuracy as high as the original model (to ensure model usability) but whose output distribution (or logits) significantly camouflages the meaningful information.

Formally, given a teacher model $\theta_{t}$, they output a nasty teacher model $\theta_{n}$ trained by minimizing cross-entropy loss $\mathcal{L}_{CE}$ with target labels $y$ (to ensure high accuracy) and also by maximizing KL-Divergence $\mathcal{L}_{KL}$ with the outputs of the original teacher (to maximally contrast or disturb from the original and create a confusing distribution). This can be written as:

where $\omega$ is a weighting parameter, and $\tau_{A}^{2}$ is a temperature parameter. Figure 1(a) provides a visual illustration of Nasty Teacher’s outputs after softmax. We see that when compared to a normal teacher model in Figure 1(e), it maintains correct class assignments but significantly changes the semantic information contained in the class distribution.

Proposition. Transform the output to reduce the degree of confusion in them.

Method. We hypothesize that distillation from defenses such as Nasty Teacher can be improved by increasing the relative importance of low-scoring incorrect classes and including their presence in the output. Increasing the importance of low-scoring incorrect classes makes the high-scoring incorrect classes lesser important, thus reducing confusion. We note that this idea can be used generically, even independent of the Nasty Teacher’s defense. To this end, we first soften the teacher’s outputs with a high temperature $\tau$ ($\tau$ $>$ 50 in our case). Figure 2b shows how this reduces the relative disparity among the scores and brings them closer, thus helping reduce confusion. From the softmax outputs in Figure 2d, we further see that this not only allows the other incorrect classes to be viewed in the output but also gives rise to relationships (or variations) which were earlier not visible. Formally, the above operation can be written as:

Although we get a much more informative output, the above transformation does also reduce the relative peak of the correct class, which for distillation may not be ideal. We overcome this by using a convex combination of $\textbf{y}_{nasty}$ with the one-hot target vector to obtain the final output $\textbf{y}_{net}$, which makes this strategy more meaningful (see Figure 2)):

In the above discussion, we propose to create our own training targets which satisfy the two properties to learn despite the Nasty Teacher’s defense: (i) has a high peak for the correct class; and (ii) has the rich semantic class score information. Finally, to learn from this teacher and match its distributions, we minimize the cross-entropy lpss between student probabilities ($\textbf{s}=softmax(\textbf{z}_{student})$) and HTC teacher ($\textbf{y}_{net}$) targets as:

Combining the above with Eqn 7, $\mathcal{L}_{CE}$ as cross-entropy loss and $\mathcal{L}_{KL}$ as KL-Divergence, we can now write the above as:

In Equation 9, we see that $\mathcal{L}_{CE}$ is replaced with $\mathcal{L}_{KL}$. Here, we use the fact that cross-entropy loss and KL-divergence differ by a term $\sum_{i}\textbf{y}_{nasty,i}\log{\textbf{y}_{nasty,i}}$, which remains a constant for student gradient computation because of fixed teacher outputs $\textbf{y}_{nasty}$. $\mathcal{L}_{\mathcal{HTC}}$ is thus a KD-loss similar to what was discussed in Sec 3.1. To adjust the extent of knowledge transfer, we finally re-weight the KL-divergence term with a hyperparameter multiplier $m$. Conceptually, $m$ does not make a difference to the idea, but we observe this to be useful while training. Figure 3a provides a visualization for this approach.

Motivation: Nasty Hides Essential Information. As Nasty Teacher selectively causes some incorrect classes to exhibit peaks while inhibiting scores for others, it hides certain inter-class relationships to protect against KD-based attacks.

Proposition.: Transform the output to extract/recover the essential class relationships or possibly the entire original teacher distribution

Method.: To begin with, we ask the question - what would happen if we used Eqn 5 as is with the given nasty teacher $\theta_{n}$? We would expect to see a model with accuracy as high as the teacher but with presence of class distribution peaks different from it. If we perform this operation for “k” such sequential stjpg, we can expect to obtain “k” potentially different output distributions. Building on this thought, we introduce a Sequential Contrastive Training strategy, wherein we form a sequence of “k” contrastive models by training each model to contrast with the model just before it. Formally, taking the nasty teacher $\theta_{n}$ as the starting point of the sequence, $G$ as the method for generating the next contrastive item in the sequence, which in our approach is the same as the one described in Eqn 5, the sequence $S^{k}$ with $k$ as the sequence length can be written as:

Thus, while maximising KL-divergence between models, each model learns a different probability distribution and has its own unique set of confusing class relationships. We then take an ensemble of their outputs, which is denoted as $\textbf{z}_{ens}$ and use this alongside the ground truth labels $y$ to train the student (or stealer) model, i.e.

where hyperparameters $m$, $\alpha$, $\tau$ have the same meaning as in Eqn 9. The core intuition behind SCM is that a relationship may be important in its own distribution but only the essential relationships will be important across distributions. Therefore, by taking the ensemble, we not only capture such relationships but also obtain a distribution closer to the original teacher. This idea is visualized in Figure 1 where (b) and (c) represent two successive checkpoints in the sequence, and (d) represents their ensemble. It can be clearly noted that different items in the sequence consistently output diversity in class relationships, and further, their ensemble illustrates a class distribution closer to the original teacher (Figure 1e).

For all the experiments, we follow the same models and training configurations as used in the code provided by [22]²²2https://github.com/VITA-Group/Nasty-Teacher.

Datasets and Network Baselines: For datasets, we use CIFAR-10 (C10), CIFAR-100 (C100) and TinyImageNet (TIN) datasets in our evaluation. For teacher (or victim) networks, we use ResNet18 [13] for CIFAR-10, both Resnet18 and ResNet50 for CIFAR-100 and TinyImageNet. For student (or stealer) networks, we use MobileNetv2 [33], ShuffleNetV2 [23], and Plain CNNs [26] in CIFAR-10 and CIFAR-100 and MobileNetV2, ShuffleNetV2 in TinyImageNet. For baselines, Vanilla in Table 1 refers to the cross entropy training, Normal KD/Nasty KD refer to the distillation (or stealing) with original/nasty teacher, Skeptical refers to the recent work [19], and HTC, SCM refers to our approaches.

Training Baselines: We follow the parameter choice from [22]. For generating Nasty Teacher in Eq. 5, $\tau_{A}$ is set to 4 for CIFAR-10 and 20 for CIFAR-100/TinyImageNet, correspondingly $\omega$ to 0.04, 0.005, 0.01 for CIFAR-10, CIFAR-100, TinyImageNet respectively. For both Nasty KD and Normal KD, ($\alpha$, $\tau$) parameters of distillation in Eq. 2 are set to (0.9, 4) in plain CNNs and (0.9, 20) in MobileNetV2, ShuffleNetV2. Plain CNNs are optimized with Adam[18] (LR=1e-3) for 100 epochs while MobileNetV2, ShuffleNetV2 are optimized with SGD (momentum=0.9, weight decay=5e-4, initial LR=0.1) for 160 epochs with LR decay of 10 at epoch [80, 120] in CIFAR-10 and for 200 epochs with LR decay of 5 at epoch [60, 120, 160] in CIFAR-100. Training settings in TinyImageNet are same as used in CIFAR-100. Moreover, all the experiments use batch size of 128 with standard image augmentations.

Training HTC and SCM : For both HTC and SCM, we search $m$ in {1, 5, 10, 50}, $\tau$ in {4, 20, 50, 100}, $\alpha$ in {0.1,….0.9}. For TinyImageNet, we include $\tau=200$ also in the earlier set of $\tau$. While any number of sequence models can conceptually be chosen for SCM , our experiments only leverage a max. of 4 such Sequence Contrastive models. Note, for all these we choose the search space rather intuitively and finally present their impact in Sec. 4.3.

We report the results in Table 1 and clearly observe the degradation of student performance while learning from Nasty teacher. Subsequently, our approaches: HTC and SCM increase learning from Nasty by upto 58.75% in CIFAR-10, 68.63% in CIFAR-100 and 60.16% in TinyImageNet. We significantly outperform the recent state of the art Skeptical [19] and unlike them we also consistently outperform the Vanilla training. Moreover, many times we achieve very close performances and other few times even better performance than Normal-Teacher (i.e stealing from unprotected victim model), see the $§$ marked cells in table 1. Further, we combine our training methods HTC and SCM with the novel stealing architecture Skeptical Student [19] and report improvements in Table 1.

Choice of $\tau$, $m$ in HTC: HTC (Sec 3.3) depends on softening temperature ($\tau$) which adjusts the optimal representation, and multiplier $m$ which adjusts the optimal weight to transfer this knowledge. Thus, we now study each of these parameters separately. We also note that $\alpha$ primarily controls the ground truth signal, hence, we omit varying $\alpha$ here and set it to 0.9. We vary the $\tau$ as 10, 20, 50, 100 with $m$ = 50 on ShuffleNetV2 and present results in Fig. 4 (a),(b) to demonstrate the effect of setting temperature to an optimal higher value. We then vary $m$ keeping the $\tau$ at 50 to observe the motivation of its careful selection in 4(d).

Choice of Architecture and $k$ in SCM : Given a Nasty Teacher, we currently use the same architecture type to generate ${S}^{1},{S}^{2},..{S}^{k}$. However, we now explore if SCM generalizes to different architecture choices in Table 4(a). Rather than obtaining seq. items as ResNet18 $\rightarrow$ ResNet18 i.e ${S^{i}}_{ResNet18}=G({S^{i-1}}_{ResNet18})$, we hereby do ${S^{i}}_{ShuffleNetV2}=G({S^{i-1}}_{ResNet18})$, and finally ensemble generated ShuffleNetV2 ($S^{2})$ and original ResNet18 ($S^{1}$) to train the students (or stealers). Table 4 further shows that even with different architecture, SCM can extract information from Nasty Teacher. Moving ahead, we now ablate on sequence length $k$. From Table 4(b), we observe improvement in performance as we increase number of sequence items. This can be intuitively linked to getting better estimate of essential relationships with more number of models, hence improved distillation.

Similarity of SCM and Original Teacher: As with SCM we seek to potentially recover the original teacher distribution, thus in addition to the visualization we present in 1 we hereby conduct this study to quantitatively estimates the closeness of the SCM ensemble and normal teacher. We use KL-Divergence as our metric and present results in Table 5. It can can be clearly seen that compared to Nasty Teacher, SCM ensemble consistently results in low KL-Divergence, thereby lending support to our motivation in recovering original teacher.

No Label Available: Here, we consider no access to the labels for data used in 4.1. Table 7 includes the results against this setting.

Limited Data Available: Here, we evaluate the learning by varying the percent of data used [10%, 30%, 50%, 70%, 90%]. From Fig. 4(c), we observe to better extract knowledge in all data-subset fractions. Specifically, performance difference becomes even more notable when very low fraction is used (like in 10% setting, we perform approx. 15% better than Nasty Teacher). In addition, we also observe that we always remains significantly closer to the original teacher.

No Data Available: Here, we consider the setting where no data is available. More precisely, we take the existing data-free distillation method [5], and add our method to it. Table 7 demonstrates consistent improvement while learning from nasty in this setting.

Previously, we discuss the vulnerabilities of Nasty Teacher [22]. We now discuss this section in regards to improve “Nasty Teacher” defense.

Nasty Teacher derives its efficacy from the peaks created for the incorrect classes