Progressive Knowledge Distillation for Unsupervised Domain Adaptation

In this section, we describe some popular knowledge distillation techniques that can be potentially combined with potentially any domain adaptation technique. The paper of Hinton[HIntonKD] shows that it’s possible to obtain a smaller student model trained on the soft-label provided by a teacher model. This technique uses a temperature version of soft-max that provides softer output for the student model to learn from. From here several techniques have been proposed in order to solve different problems in knowledge distillation. Recently, the overhaul feature distillation paper[Overhaul] proposes to force feature similarity at intermediate layers between the teacher and student network in order to maximize the information transfer between the teacher and the model. In this paper, feature similarity between student and teacher is enforced by minimizing a partial L2 distance, which is a L2 norm except that if the value of student is smaller than the teacher and both are negative then the result is zero, between student and teacher after using a Margin ReLU (use of a margin $m$ instead of $0$). The paper of [TeachingAssistantKD], proposes to solve the problem of the gap between a converged teacher model and a student model by adding a teaching assistant model which is a model of intermediate complexity between student and teacher. Another paper [RelationalKD] focuses on the actual knowledge transfer in order to improve performance, the paper argue that current existing knowledge transfer method, only transfer knowledge as independent individual output as knowledge whereas relational data between data is also an important information to be taught to the student. This paper[RelationalKD] proposes to transfer the relation knowledge between data samples using the distance between samples and their angles, the knowledge distillation is then the minimization of this relational knowledge between teacher and student. While there has been a lot of interest in knowledge distillation in general, the combination of knowledge distillation and unsupervised domain adaptation, with goal of obtaining a compact model for target domain, remains mostly unexplored.

As for UDA techniques, based on the survey of [DA_Survey], there are several main categories of : 1) Finding domain-invariant features 2) Domain Mapping 3) Ensemble Methods 4) Statistic normalization and 5) Target discriminate method. The first method can learn domain invariant features either by domain confusion[GRL, ADDA], minimizing a distance between distribution [MMD_ICLR]. Both of [GRL, ADDA] makes use of a domain classifier (or discriminator) in order to encourage domain confusion, [GRL] uses a gradient reversal layer in order to maximize the domain classification loss. As for [ADDA], the domain confusion is done by using an adversarial losses on the discriminator. Domain Mapping focuses on finding a mapping either from the source domain to the target domain or vice-versa like [DomainMapping_Pixel_Level, ConGAN_DA_MAP]. Currently, most of the domain mapping based techniques are based on the use of GAN. The paper of [DomainMapping_Pixel_Level] proposes a to use a pair of discriminator and generator in order to map a source image and vector of noise to the target domain. This mapping is learnt at the same time as a task-specific loss (i.e. classification loss) on both transformed image and non transformed-images and the overall optimization is done alternatively between generator and discriminator-task. The paper of [DomainMapping_CADA] goes further by not only doing domain adaptation at pixel-level but also integrate feature-level domain adaptation using another discriminator. There are also others methods like ensemble methods that use multiple models or the same model at different time(self-ensembling) in order to either produce more reliable pseudo-label on data by a voting system. The paper of [Tri_net_DA_Ensemble] proposes to use three network, that shared a feature extractor, for domain adaptation. In this model, two of the network are responsible to label target domain data when both of them provide the same label, while the third network learns from these pseudo-labels. Others methods like normalization statistic assume that the task knowledge is learned and the only adaptation needs to done is on batch norm statistics[DA_Normalization_statistics]. The last method works with the assumption that data points are in separate cluster and the decision boundary is in the lower density region, these methods work by trying to move the decision boundary to lower density region using adversarial losses. For example, the paper of [Wei2018GenerativeAG], proposes to add to Domain Adversarial Training[GRL] a generative model in order to guide the domain adaptation. By incorporating a generator that can generate target data and adding a ”fake” class to the discriminator, the author pushed the decision boundary toward the target domain. Among these methods, we chose to go with the first categories because of its simplicity and also because it has also been used in other problems like detection which can have more interest in the community.

There are several scenarios for a combination of knowledge distillation and domain adaptation: 1) Domain Adaptation directly on compact model 2) Domain Adaptation then Knowledge Adaptation 3) Joint Domain Adaptation and Knowledge Distillation at the same time. There’s actually a forth scenario which is Knowledge Distillation then Domain adaptation, this is actually the same scenario as the first one since we can just start directly from a compact model. In this paper, we focus on the third scenario while proving that the two other scenario does not work in their current trivial form. In the first scenario, one of the main problem is the generalization capacity of the compact model for the domain adaptation task which can prove limited, through our experiments, we prove that this is indeed the case. As for the second scenario, the problem of unsupervised knowledge distillation remain quite a challenge since ground truth is important in traditional knowledge distillation in order to force student consistency, this point is also going to be proven in our experiments. Currently, the third scenario seems to be the perfect choice of knowledge distillation during domain adaptation, we would take advantage of the domain adaptation loss to force consistency on the student model during knowledge distillation. Compared to the first scenario, since the teacher and the student domain adapted at the same time, the student would learn from the teacher step by step instead of trying to learn directly from a finished teacher model. The difference here is the different in learning loss, in the first scenario, the student model would learn to minimize a domain adaptation loss while in the third scenario the student model would learn only from the knowledge distillation loss.

In this paper, we chose two basics techniques in order to do knowledge distillation and domain adaptation. For knowledge distillation, we chose the technique based on Hinton’s paper, however any technique can be replaced here. As for domain adaptation, we chose a basic domain adaptation technique based on MMD from [MMD_ICLR, TCP] but it can also be generalized for any other domain adaptation techniques. We start by defining the domain adaptation loss for teacher model:

With $D_{s}$ the source domain dataset which contains $N_{s}$ samples and labels, $D_{t}$ the unlabeled target domain dataset of $N_{t}$ data samples, $\phi_{T}$ the teacher function that maps an input to a feature map, $\mathcal{H}$ the Reproducing Kernel Hilbert Space(RKHS) with gaussian kernel. Same with [MMD_ICLR, TCP], we add to this a supervised loss on the source domain in order to have the final domain adaptation loss for the teacher:

$\mathcal{L}_{sup}$ a supervised loss of the teacher model on the source domain and $\gamma$ a trade-off hyper-parameter that follows the same variations as [TCP]. The next step is to transfer the target domain knowledge from the teacher to the student, we use a modified version of the knowledge distillation loss from the work of Hinton:

In this equation, $S$ and $T$ represent respectively the output of student network and teacher network with a soft-max based on a temperature $\tau$ in order to soften the output and $L_{distill}$ is a KL divergence loss in our case but can be replaced with a mean squared loss or cross-entropy. This loss differs from the original paper because, we had to remove the cross-entropy loss between the student model output and the ground truth since we are working on an unsupervised domain adaptation. Trivially, this should be enough for joint knowledge distillation and domain adaptation since we only want to have target domain knowledge and to ensure the consistency of the model w.r.t to a common representation, we proposed to add consistency loss to ensure that the student model can learn a better common representation from source and target domains.

The equation 4 is the student knowledge distillation loss, with $\alpha$ a hyper-parameter to balance between the knowledge distillation and the cross entropy loss of the output of the student model and the ground truth on the source domain. The final loss of our models is then:

We added the $\beta$ hyper-parameter in order to balance out the importance between domain adaptation and knowledge distillation. Since we are doing jointly knowledge distillation and domain adaptation, in beginning, the teacher would still be learning from the domain adaptation, this meant that there’s not much to be learn for the student model beside the source representation which can be learned from the knowledge distillation loss, in light of this, we propose to start by giving more important to domain adaptation in the beginning and gradually transfer the importance to knowledge distillation basing $\beta$ on a exponential growth function between $[0.1,1]$.

For this paper, we use two popular datasets for domain adaptation in literature: Office31 and ImageClef-DA. The Office31 dataset contains three subsets of dataset which are Webcam (W), DSLR (D) and Amazon (A) with 31 classes. The ImageClef domain adaptation contain four subset which are taken from Imagenet (I), Pascal-Voc (P), Caltech (C) and Bing (B). Each of these subsets contains 600 images for 12 classes. For this dataset, we compare to others techniques using six scenarios: $I\xrightarrow{}P$, $P\xrightarrow{}I$, $I\xrightarrow{}C$, $C\xrightarrow{}I$, $C\xrightarrow{}P$, $P\xrightarrow{}C$. For each of these experiments, we evaluate several baseline scenarios: Only Domain Adaptation on Target dataset, Knowledge Distillation then Domain Adaptation, Domain Adaptation then Knowledge Distillation and the result from the Transfer Channel Pruning paper (cite here). For each dataset, we use two student model architecture, which are: Resnet34 (12% FLOPS reduction from Resnet50) and Resnet18(56% FLOPS reduction from Resnet50) which are closely similar to the FLOPS reduction of TCP.

For the implementation of this algorithm, we used two separate optimizer in order to optimize the domain adaptation loss and the knowledge distillation loss. For the domain adaptation optimizer we use a exponential learning rate decay and for the knowledge distillation loss, the learning rate remains the same. For the hyper-parameters $\beta$, we use a starting value of $0.1$ and an end value of $0.9$, for the knowledge distillation hyper-parameters, we use a temperature $\tau=20$ and $\alpha=0.8$. Overall, we use a learning rate for optimizer starting at $0.001$, a momentum of $0.9$ and $400$ epochs. Our implementation can be found at: (link here)