Evaluation of Deep Neural Network Domain Adaptation Techniques for Image Recognition
^††thanks: *Equal contribution.

DeepCORAL [3] is a domain adaptation method based on a previously proposed method called Correlation Alignment (CORAL) [16].

CORAL attempts to increase performance in domain shift by aligning the second order statistics between both data distributions. This method consists of 1) feature extraction, 2) apply a linear transformation and 3) implement an SVM classifier.

DeepCORAL extends the capacities of CORAL by learning a non-linear mapping using convolutional neural networks and a differentiable loss function that attempts to minimize the Frobenius norm between the feature covariance matrices of source and target distributions. This method works in an unsupervised way where the target dataset has no labels.

Recall that the main goal is to learn representations that allow to achieve good performance in both data distributions (image recognition, for example). To achieve this, DeepCORAL intializes the network parameters from pre-trained AlexNet model while at the same time minimizing the loss mentioned above (a.k.a CORAL loss). The architecture of DeepCORAL is shown in the figure below.

Note that DeepCORAL takes the last fully connected layer of AlexNet (fc8) and computes a classification and CORAL loss.

Deep domain confusion is another domain adaptation method that uses a deep neural network to learn a non-linear transformation. It is similar to DeepCORAL in the sense that it introduces a transfer loss to learn features for both data domains. [9]

In their paper, Tzeng et al. show the concept of biased datasets and how trained classifiers don’t always transfer well to target domains. Their approach is to maximize a loss function that minimizes classification error and maximizes domain confusion.

To minimize the distance between data distributions, DDC uses a standard distribution distance metric called Maximum Mean Discrepancy (MMD). In this case, this distance is computed with respect to an activation layer $\phi(\cdot)$ acting on source data points $x_{S}$ $\in$ $X_{S}$ and target data points $x_{T}$ $\in$ $X_{T}$. The approximation to this distance is as follows

Training a model with MMD and a classification loss will result in a strong classifier ready to transfer accross domains. Thus, the total loss is

Here $l_{C}$ corresponds to a classification loss on the available source labelled data $X_{L}$ and the ground truths $y$. The hyperparameter $\lambda$ acts in a similar way a regularizer as in DeepCORAL.

The architecture proposed in DDC is similar to DeepCORAL; it intializes its weights with a pre-trained AlexNet and weights are shared between both data domains. The difference is that it has an extra bottleneck layer fc adapt that is meant to regularize the training of the source classifier and prevent overfitting to particular nuances of the source distribution. The loss is computed after this bottleneck layer. The next figure shows the architecture of DDC.

Adversarial domain adaptation [17] [18] [19] consists of domain adaptation and adversarial learning which is very similar to Generative Adversarial Networks (GANs) [20]. In adversarial domain adaptation, a deep neural network model learns image representations by minimizing a classification loss and simultaneously a domain discriminator learns how to distinguish between source and target domain. There are however two pitfalls for adversarial domain adaptation methods. In the first place they cannot handle datasets which have complex multi modal distributions. And secondly we cannot rely on the uncertain information difference between source and target data to model the domain discriminator.

These two pitfalls can be solved by conditional domain adaptation techniques. Conditional domain adaptation is made possible due to the research carried out in Conditional Generative Adversarial Networks (CGANs) [21]. CGANs proved a point that domain shift between distributions of real and synthetic images can be reduced by conditioning network using discriminative information.

CDAN [2] is an adversarial domain adaptation technique motivated by CGANs and it makes use of the discriminative information obtained from a deep classifier network. The key to CDAN models is a new conditional domain discriminator conditioned to the covariance of representations of domain-specific resources and classifier predictions. Let us consider $D_{s}=\{(x_{i}^{s},y_{i}^{s})\}_{i=1}^{n_{s}}$ is a source domain data with $n_{s}$ labeled examples and $D_{t}=\{(x_{j}^{t})\}_{j=1}^{n_{t}}$ is a target domain dataset with $n_{t}$ unlabeled examples. Source and target datasets are drawn from $P(x^{s},y^{s})$ and $Q(x^{t},y^{t})$ which are not i.i.d. i.e., $P\neq Q$. The aim of CDAN is to design a network $G:x\to y$ that can reduce the domain shift between source and target datasets. $f=F(x)$ and $g=G(x)$ are feature representation and classifier predictions made from the network G respectively.

Considering all the things mentioned above, CDAN is formalized as a minmax optimization problem. In minmax optimization, two terms are considered as players, namely $E(G)$ corresponding to source classifier G and minimized to guarantee lower source risk and $E(D,G)$ on G and the domain discriminator D on cross domains. $E(D,G)$ is minimized over D and maximized over $f$ and $g$

where L(.) is the cross entropy loss. Now CDAN is described in minmax optimization problem as

where $\lambda$ is a hyper-parameter that balances the domain adversary and source risk.

In Fig. 7 D is the domain discriminator conditioned using a multilinear map $f\otimes g$, multilinear mapping is explained in Section 3.2 of [2]. Thus CDAN enables domain adaptation over cross domains using minmax optimization.

CDAN+E is an variant of CDAN. In CDAN+E we will have additional involvement of Entropy Conditioning which improves the difference between source and target domains. In CDAN, the domain discriminator allots equal weight to each and every example irrespective of examples with uncertain predictions from G. So uncertainity corresponding to classifier predictions are quantified using entropy criteria explained in Eq. 10.

where C is number of classes and $g_{c}$ is probability of predictions.

Now training examples are prioritized by an entropy-aware weight given by

So CDAN+E minmax optimization is formulated as

where h =(f,g) is a joint variable.

Thus CDAN+E encourages the certain predictions which are achieved through entropy minimization principle [22].

Office-31 dataset was first introduced in [14]. It contains a total of 4652 images collected from three different sources. The three different sources from which images are collected are online web (amazon), digital SLR (dslr) camera and webcam. Office-31 dataset have 31 classes ¹¹1The 31 classes are: backpack, bike, bike helmet, bookcase, bottle, calculator, desk chair, desk lamp, computer, file cabinet, headphones, keyboard, laptop, letter tray, mobile phone, monitor, mouse, mug, notebook, pen, phone, printer, projector, puncher, ring binder, ruler, scissors, speaker, stapler, tape, and trash can. in total.

• •

  Images from the web: These are collected from amazon website. We call this dataset as amazon dataset (A). It has around 2800 images and with an average of 90 images per each class. The resolution of the images are each 300x300 pixels.

• •

  Images from a dslr camera: These are captured from dslr camera in real environments. We call this dataset as dslr dataset (D). It has total of 498 images and with an average of 16 images per each class. The resolution of the images are 1000x1000 pixels.

• •

  Images from a webcam: These are collected from a webcam with lower resolution and will have more noise. From now this is called as webcam dataset (W). It has around 795 images and with an average of 25 images per each class. The resolution of the images is HxW where H and W are varying between 400 and 600.

The method is evaluated with the benchmark Office dataset described above. We have carried out 6 different experiments for all the domain shifts available (given the tree datastes cotained in Office).

CORAL loss is applied to the last fully connected layer (fc8). The weights of this layer are initialized following a normal distribution $\mathcal{N}(0,0.005)$ and the weights from the other layers are initialized with the pre-trained AlexNet. We used Stochastic gradient descent (SGD).

DeepCORAL was trained using batches of 128 images (both domains), a fixed learning rate of $10^{-3}$, weight decay set to $5x10^{-4}$ and momentum to 0.9. The $\lambda$ factor is set in such a way that both losses are approximately the same. In practice, given $t$ epoch iterations, $\lambda$ is $\frac{1}{t}$. Furthermore, during training only the source dataset is labelled, this is how we compute the classification loss.

In Table 1 we show the obtained accuracies for each method for all 6 experiments. These values correspond to the image classification accuracy on the target dataset. From here, we see that DeepCORAL obtained the best result in 4 out of 6 experiments. This method outputs accuracy values in the range between 38% to 89%. The fluctuation of accuracies between domains has to do directly with the chosen source and target distributions. It is important to note that the method does not output accuracies greater than 50% in most of the cases. We’ve compared our values with the original paper and although lower by a small percentage, these follow the same trend as the original results.

For a better visualization of the method’s performance we have added to visualize accuracy and loss as a function of epochs for the case Amazon $\,\to\,$ Webcam. Figure 8a) shows an increase of almost 10% in recognition accuracy when including CORAL loss. Figure 9a) shows the behavior of the loss functions, note that they tend to become equal due to the lambda factor.

Training for this method is very similar to DeepCORAL. The main differences are 1) it trains with MMD loss, 2) adds the bottleneck layer and 3) it uses a schedule learning rate which decreases as a function of epochs. Fig. 8b) shows that accuracy with MMD (or DDC) doesn’t have a very noticeable increase compared to training without MMD, however, it seems to be more stable and shows an steady increase. Due to time constrains, we were only able to run one experiment for this method. However, it is consistent with the accuracy reported on the original paper.

Experiments for CDAN and CDAN+E are conducted with same set of hyperparameters and the only difference between both is the transfer loss function that is involved. Hyperparameters used in the experiments for CDAN and CDAN+E are:

• •

  Epochs 100

• •

  Source batch size 10

• •

  Target batch size 10

• •

  Learning rate 1e-3

• •

  Momentum 0.9

• •

  Weight decay 5e-4

Dataset used for all the experiments are office-31 dataset [14] Backbone used in this experiment is pretrained AlexNet with a small modification. There is a bottleneck layer introduced before last fully connected layer. Bottleneck layer has the dimension 256 (used in the original paper [2]) which is also a user defined shape. Optimizer used is Stochastic Gradient Descent (SGD) in which parameters of deep neural network G and domain discriminator D are added to optimize. During the training phase each batch from source and target data loader are passed through deep neural network G and features f from bottleneck layer and fully connected layer output are obtained from the deep neural network G. Thus obtained features f and final layer output of source and target datasets are joined together. Concatenated final layer outputs are passed through softmax layer in order to get network predictions g.

As we obtained features f and predictions g, classification loss for source data can be calculated using $g_{s}$ and ground truth labels of source data. Transfer loss is calculated based on domain adaptation method selected. If CDAN is selected, joint variable h=(f,g) which have both source and target data features and predictions are used. Thus transfer loss is calculated using domain discriminator network whose input is h which is explained in above Sections on CDAN. If CDAN+E is selected, along with the features and predictions we calculated entropy from class predictions which is explained in above Section. CDAN+E CDAN+E. This calculated entropy is used to perform weighted operations based on uncertainty of the predictions. Total loss to do back propagation is calculated using below equation.

where $\lambda$ is a hyperparameter which is described in the above Sections. During the experiment it is observed that transfer loss obtained is very small which resulted in very very small gradients of order $10^{-7}$. Which in turn did not update the weights of domain discriminator and thus the transfer loss is fluctuated with very small variance. In Table I we can observe that CDAN+E only performed better in 2 cases out of 6. It is also observed that transfer loss is reducing till $20^{th}$epoch in most of the cases. In Fig. 9(c) and 9(d), training and testing accuracy as well as in Fig. 10(c) and 10(d), transfer loss and classification loss of A$\to$W domain adaptation is shown respectively. In our opinion smaller batch size also played a key role in results of CDAN and CDAN+E methods. Smaller batch size is chosen considering the limitations of local machine. Although the results are not matched with original paper we observe that there is variation in accuracy and loss when transfer loss is included.