Exploring Dropout Discriminator for Domain Adaptation

The deep learning based models have achieved considerable success in the Visual recognition domain. These models are trained on very large annotated datasets such as Imagenet [1]. The deployment of these generically trained models require them to adapt to work in specific settings (for instance with catalog images in E-commerce websites). This problem is recognized as one of dataset bias. The context for this problem in vision was nicely demonstrated through the work of [2]. However, the requirement of a large annotated dataset becomes a bottleneck for training networks in deep learning frameworks. The typical solution is to further fine-tune these networks on a task-specific dataset. This approach is satisfactory if there is a sufficiently large target dataset on which to fine-tune. In this paper, we solve the problem of adapting classifiers to work on datasets that do not have any labeled information. This problem is one of unsupervised domain adaptation and is a more general setting.

Ganin and Lempitsky [3] proposed a method to solve unsupervised domain adaptation through back-propagation. In this method, the domain adaptation problem is solved by using a discriminator that ensures domain invariance of learned representations used for classification. Sometime this discriminator may introduce the mode collapse problem in features. There have been several methods [4, 5, 6, 7] proposed for improving the discriminator. However, most of these involve an increase in the number of parameters. For instance a recent work by Pei et al (MADA) [8] addresses this issue through class-specific discriminators. This leads to a linear increase in the number of parameters with the number of classes in dataset. In contrast, we propose the use of curriculum-based dropout discriminator to obtain improved performance of the domain adaptation task without increasing the number of parameters. It makes our model’s applicability comprehensive as it can also adapt to datasets with a large number of classes.

Specifically, in this paper, we propose Curriculum based Dropout Discriminator for Domain Adaptation (C$\text{D}^{3}$A) and compare it with a variant, Dropout Discriminator for Domain Adaptation ($\text{D}^{3}$A). It is a novel approach that solves the above problem through an adversarial dynamic dropout based ensemble of discriminators. where we consider dropout as being a source of an ensemble of domain classifiers [9]. The proposed model also allows the discriminators to reduce the prediction variance, remove overfitting, and average out the bias.

In [10], we initiated the work by introducing a dropout discriminator in curriculum fashion in domain adaptation, whereas in this work, we further explored the dropout discriminator by considering the fixed number of discriminators samples. We analysed the impact of curriculum learning by changing the curriculum criteria in the experiments in this submission. We also provide the analysis of experiments by reducing the source samples and how it affects the domain adaptation problems. We are able to show that the dropout discriminator works well even in these challenging setting. In this paper we further provide a visualization of adapted features for different transfer tasks. The performance analysis of the discriminator is provided.

The idea for this discriminator is illustrated in Figure 1. The initial discriminator by Ganin and Lempitsky [3] suggests the use of a single binary discriminator and MADA [8] extends it to class-specific cues. In contrast, C$\text{D}^{3}$A obtains a discriminator distribution that provides a much-improved feedback for improving the feature extractor. The performance of any adversarial learning method largely depends upon the capability of the discriminator network. The ensemble method [9] improves the discriminator’s performance and makes it robust. We show that this indeed helps in an improved domain adaptation (around 5.3% improvement in Amazon-DSLR adaptation) with much fewer parameters ($\sim$59M) than MADA ($\sim$98M). More importantly, our method does not increase the number of parameters as the number of classes increase, making it scalable to datasets with a large number of classes. Through this paper we make the following main contributions:

• 1.

  We propose a method to obtain a dropout based discriminator that provides a distribution based discrimination for every sample ensuring a more robust feature adaptation

• 2.

  We adopt a curriculum based dropout model, C$\text{D}^{3}$A, that ensures gradual increase in the number of samples as the adaptation progresses to ensure better adaptation in contrast to a fixed number of samples based dropout distribution (D³A).

• 3.

  We provide a thorough empirical analysis of the method (including statistical significance, discrepancy distance) and evaluate our approach against the state-of-the-art approaches.

• 4.

  We also experiment with sensitivity to source data size by evaluating the method on half the amount of source data to verify the effect on the method. This experiment tests the resilience of the method.

Domain Adaptation: In the domain adaptation setting a basic common structure that has been followed is the Siamese architecture [11] with two streams, representing the source and target models. It is trained with a combination of a classification loss and the other being one of discrepancy loss or an adversarial loss. The classification loss depends on the source data label, while the discrepancy loss reduces the shift between the two domains. A discrepancy based deep learning method is that of deep domain confusion (DDC) [12]. The loss between a single FC (fully connected) layer of source and target feature extractor network is used to minimize the maximum mean discrepancy (MMD) between the source and the target. This approach is further extended by deep adaptation network (DAN) [13]. Recently, a number of other methods have been proposed which use discrepancy of domain [14, 15, 16, 17, 18, 19, 20, 21].

Adversarial Learning: In the domain adaptation setting, an adversarial network provides domain invariant representations by making the source and target domain indistinguishable by the discriminator. Adversarial Discriminative Domain Adaptation [4] uses an inverted label GAN loss to split the optimization into two independent objectives. One such method is the domain confusion based model proposed in [22] that considers a domain confusion objective. Domain-Adversarial Neural Networks (DANN) [3] integrates a gradient reversal layer into the standard architecture to promote the emergence of the learned representations that are discriminative for the main learning task on the source domain and non-discriminative concerning the shift between the domains. Recently, some works have been proposed which use an adversarial discriminative approach in solving the domain adaptation problem [23, 24, 25, 26, 27, 28, 29].

Similarly, the model proposed in [30, 31, 32] exploits GANs with the aim to generate source-domain images such that they appear as if they were drawn from the target domain distribution. The closest related work to our approach is the work by [8] that extends the gradient reversal method by a class-specific discriminator. In iCAN [33], a domain-collaborative and domain adversarial training of neural networks has been presented for unsupervised domain adaptation. They integrated the losses from different domain classifiers at different blocks such that the model can learn the domain invariant features from higher block and domain variant features from the lower blocks of the network.

Similar to guided dropout [34], the adversarial guided dropout [35] adversarially disconnects dominated neurons that are used for the prediction. Adversarial dropout regularization (ADR) [36] avoids to generated the target features near class boundaries using the dropout regularization. Dropout regularization has also been applied in robust speech recognition [37].

Ensemble and Curriculum learning: Ensemble methods [38] can capture the uncertainty of the neural network (NN). Gal et.al. [39] use dropout to obtain the predictive uncertainty and apply Markov chain Monte Carlo [40] also known as MCMC at the test time to deal with intractable posterior. In discriminator based approaches, ensembles can be considered as multi-discriminator or multi-generator architecture. Multi discriminator approach has also been proposed by [41, 42, 43] to learn the data distribution more effectively. In Bayesian GAN [44], dropout in the discriminator is used which can be interpreted as an ensemble model [39]. The curriculum learning [45] enhances model’s performance and its generalization capability. The performance of the GAN is also improved through the curriculum learning of the discriminator [46]. It has been shown that dropout can also work with curriculum learning [47]. In domain adaptation, a curriculum style learning approach has been applied in  [48] to minimize the domain gap in semantic segmentation. The curriculum domain adaptation first solves easy tasks such as estimating label distributions, then infers the necessary properties about the target domain. The theoretical framework for curriculum learning in transfer learning is proposed in [49]. Recently other curriculum learning based domain adaptation methods have been proposed in Transferable Curriculum Learning [50].

There are other cluster based alignment techniques that are also used to tackle the domain adaptation problem. The authors in  [51] applied the center-based discriminative feature learning methods to minimize the dataset bias. Similarly in [52], the authors align the cluster with a teacher model. KNN based alignment is used in the [53] for the domain adaptation. Further, a correlation based adversarial learning framework for domain adaptation is proposed by [54]. The Bayesian uncertainty matching is applied in [55] for adapting the classifier.

Recently, GSP [56] demonstrated a generative cross-domain learning framework via structure-preserving. They developed a cross-domain graph alignment to capture the intrinsic relationship across two domains. In DTA [57], adversarial dropout to enforced the cluster assumption on the target domain. It is based on the fact that the decision boundaries should be placed in low density regions in the feature space.

In contrast to all the previous works, the main contribution of the present work is to propose a curriculum based dropout discriminator. We show that through the proposed method, we are able to outperform state of the art domain adaptation techniques in a scalable way by using fewer number of parameters as compared to techniques such as MADA [8] and similar number of parameters as GRL [3].

In the adversarial domain adaptation problem, the previous methods have used classical statistical inference in the discriminator. A single discriminator learns the source and target domain classification. Our hypothesis is that it may lead to overconfident inference and decisions which in turn may lead to challenges in learning invariant features. In the domain adaptation problem, data is generally structured in a multimodal distribution. Thus, a multiple discriminator approach is compelling [8], due to its capacity to capture multiple modes of the dataset. It also leads to solving the perennial problem of mode collapse (which GANs are infamous for) as multiple discriminators now learn to distinguish classes with different modes. The diversity of an ensemble of such discriminators reduces the random errors in prediction. The performance of an ensemble model rests on the number of entities in the ensemble. However, as the number of entities increase, the model parameters and complexity will increase. This is one of the primary bottlenecks of the ensemble based methods. The number of parameters in an algorithm is a significant factor in determining model efficiency.

To tackle the above problems, we propose a novel and efficient discriminator architecture by using Monte Carlo (MC) sampling [58]. We incorporate Bernoulli dropout in a single adversarial discriminator network, by dropping out a certain number of neurons from our discriminator with some probability d. This gives rise to a set of dynamic discriminators for every data sample. The main idea behind our method is to construct a training regime for the feature extractor in domain adaptation that consists of increasingly challenging tasks to generate domain invariant features. This allows the sophistication of the feature extractor to gradually increase throughout training, rather than aiming for full sophistication at the outset. This method is similar to that of curriculum in supervised learning, where one orders the training examples to be presented to a learning algorithm according to some measure of difficulty [45]. Despite the conceptual similarity, the methods are quite different. Under our approach, it is not the difficulty of the training examples presented to either network, but rather the capacity, and hence strength, of the discriminator network that is increased as the training progresses. The idea behind the use of a curriculum based dropout discriminator is to exploit the characteristics of several independent discriminators by consolidating them in order to achieve higher performance.

We do a curriculum based learning on these dropout discriminators. As the training proceeds, the number of discriminators sampled, increase, thereby boosting the variance of our model’s prediction. The proposed approach enforces the feature extractor network not to constrain the learned representations to satisfy a single discriminator, but, instead, to satisfy an ensemble of dynamic discriminators (composition is different across different discriminators). Instead of learning a point estimate (in case of MADA [8]), the feature extractor network of our proposed model learns a distribution, due to the ensemble effect of feedback from a set of dynamic discriminators. This approach leads to a more generalized feature extractor, promoting resemblance in learned representations of a class from different domains. The instinct behind incorporating dropout in our model is to warrant that neurons are not exclusively reliant on a precise set of other neurons to determine their outputs. Instead, each neuron relies on the agglomerate behavior of several other neurons, promoting generalization. By applying dropout on the discriminator, we obtain a set of entirely dynamic discriminators and hence the feature extractor cannot use the trick of relying on a specific type of discriminator or ensemble of discriminators to learn to generate representations to deceive the discriminator. Instead, it will now have to genuinely learn domain invariant representations. Thus, the feature extractor network is now guided by diverse feedback given to it by an ensemble of dynamic discriminators. All this increase in performance is obtained without compromising on the scalability and complexity front through our proposed model.

The other motivation behind the curriculum learning on the discriminator is based on the fact that each data point has a hidden hierarchy in class label. The data points can also be clustered in the form of their parental class label. For example, in the Ofice-31 dataset, the class category mobile phone and calculator can share the same cluster, as their visual appearance is similar as compared to other classes. Thus, in the proposed method, we increase the number of discriminators in curriculum fashion such that it starts to capture the top parent model (domain); later, it increases the discriminator to capture the more complex child class modes (domain with class category). The curriculum-based paradigm in the sampling of discriminators also enables the model to learn domain invariant representations systematically.

In the unsupervised domain adaptation problem, we consider that the source dataset $\mathcal{D}_{s}$ has access to all its labels while there are no labels for the target dataset $\mathcal{D}_{t}$ at the training time. We assume that $\mathcal{D}_{s}$ comes from a source distribution $\mathcal{S}$ and $\mathcal{D}_{t}$ comes from a target distribution $\mathcal{T}$. We assume that there are $N_{s}$ source data points and $N_{t}$ unlabeled target data points. So $\mathcal{D}_{s}={(x_{i}^{s},y_{i}^{s})}_{i=1}^{N_{s}}\in\mathcal{S}$ has $N_{s}$ labeled examples and the target domain $\mathcal{D}_{t}={(x_{i}^{t})}_{i=1}^{N_{t}}\in\mathcal{T}$ has $N_{t}$ unlabeled examples. Our underlying assumption is that both distributions are complex and unknown. Our model provides a deep neural network that enables learning of transferable feature representations $f(x)$ and an adaptive classifier $y=C(f(x))$ to reduce the shift in the joint distributions across domains, such that the target risk $Pr_{(x,y)\sim q}[C(f(x))\neq y]$ is minimized by jointly minimizing source risk and distribution discrepancy by adversarial domain adaptation where $q$ is assumed to be the joint distribution of target samples.

In this work, we employ a variant of GRL[3], where discriminator is modeled as an MC-dropout based ensemble. The feature extractor network consists of convolution layers to produce image embeddings. Both source and target feature extractors share the same parameters. The classifier network consists of fully connected layers. Only source embeddings are forwarded to the classifier network to predict the class label. The classifier network parameters ($\theta_{c}$) are updated only by the loss from source data samples. The discriminator receives both source and target embeddings. The parameters of the MC-dropout discriminator ($\theta_{d}$) are updated with domain classification loss. The feature extractor parameters ($\theta_{f}$) are updated by the gradients from the classifier network as well as by the reverse gradient of both source and target data samples from the dynamic set of the ensemble of discriminators. Detailed architecture is presented in Figure 2.

For the adaptation task, the feature extractor learns domain-invariant features with the help of MC-Dropout based discriminator. For each data sample that goes to the discriminator, we obtain the domain classification loss. These losses are backpropagated through respective Monte Carlo sampled dropout discriminators followed by gradient reversal layer. Hence, for every input, we obtain a distribution of gradients. The feature extractor is updated by a gradient from this distribution to generate domain invariant features. In a binary discriminator [3], we obtain a point estimate of the gradient for specific input. In the case of multi discriminator [8], we obtain an ensemble of the point estimates of gradients. The advantage of obtaining a distribution of gradients is that we get generalized learned representations robustly leading to domain invariant features. We propose Curriculum based Dropout Discriminator (C$\text{D}^{3}$A), where we increase the number of MC samples as training proceeds in a paradigm similar to curriculum learning. However, in the other variant ($\text{D}^{3}$A), we maintain a fixed number of MC sampled discriminators throughout the training.

In this paper, we provide a simple approach to obtain an improved discriminator for adversarial domain adaptation. We specifically show that the use of sampling-based ensemble results in an improved discriminator without increasing the number of parameters. The main reason for this improvement is that the features are made domain invariant based on a distribution of observations as against a single point estimate. Our approach based on curriculum dropout suggests that we are able to obtain an improved discriminator that is stable and improves the feature invariance learnt. We compare our method with standard baselines and provide a thorough empirical analysis of the method. We further observe through visualization that domain adapted features do result in domain invariant feature representations. Using the discriminator obtained through curriculum based dropout to solve domain adaptation is a promising direction, which we have initiated through this work.