Adaptively-Accumulated Knowledge Transfer
for Partial Domain Adaptation

The cross-domain data distribution discrepancy, known as domain shift, is the main challenge of domain adaptation. Plenty of works exploits the potential of deep neural networks to capture explanatory attributes and domain-invariant features in recent years, which is conducive to mitigating domain shift while transferring underlying knowledge across domains in domain adaptation tasks (Bengio et al., 2013; Donahue et al., 2014; Yosinski et al., 2014). Compared to traditional machine learning based domain adaptation solutions, introducing deep architecture into domain adaptation promotes the generalization of frameworks dramatically (Hoffman et al., 2014; Oquab et al., 2014). Some researchers integrate high-order statistical properties of different domains into a unified framework, such as maximum mean discrepancy (MMD), to align the data distribution across domains, which successfully eliminate domain shift and achieve promising classification performance on the target domain (Long et al., 2015, 2016). By virtue of generative adversarial techniques, some works involve a domain discriminator into the game to distinguish which domain the sample belonging to while optimizing the generator and discriminator in an adversarial manner (Ganin et al., 2016; Tzeng et al., 2015; Li et al., 2019b). Moreover, the latest works rethink the domain adaptation problem from various perspectives and propose dual-classifiers-based frameworks that seek to align not only domain-wise data distributions but also classifier-class-specific boundaries (Saito et al., 2018b; Lee et al., 2019b; Zhang et al., 2019).

Unfortunately, realistic application scenarios hardly satisfy the standard domain adaptation assumption that the source and target domains share identical label space. A more common situation is the source domain subsumes the target domain label space, which means the source domain includes samples from more different categories except for the shared ones with the target domain. Such a novel challenge, named as partial domain adaptation (PDA), attracts substantial attention in transfer learning and brings out many inspiring works on this topic. Selective Adversarial Network (SAN) explores multiple adversarial networks to weight and select out the outlier categories source samples and down their transferring weights (Cao et al., 2018a). Partial Adversarial Domain Adaptation (PADA) extends SAN and pays more attention to class-level transferability weighting on the source classifier (Cao et al., 2018b). Similarly, Importance Weighted Adversarial Nets (IWAN) considers the sigmoid output of an auxiliary domain classifier as the indicator to measure the probability of each source sample comes from the target domain (Zhang et al., 2018a). Example Transfer Network (ETN) further explains the discriminative information as the transferability quantification of the source domain samples, through which the irrelevant examples from outlier categories are down-weighted for both the task-specific classifier and domain discriminator (Cao et al., 2019). All the pioneering efforts achieve impressive performance improvements over conventional domain adaptation approaches on PDA tasks.

However, although most existing PDA solutions seek to mitigate the negative transfer caused by outlier source classes by re-weighting samples’ importance to reduce the distraction, they still train and predict the entire source domain label space, which dilutes the contribution of discriminative information within the shared categories across domains. Besides, some of them regard the prediction of the target samples as pseudo labels to align cross-domain conditional distribution, which would involve severe classification errors and mislead the optimizing direction of the model, especially at the initial stage of training when the classifier cannot handle the differently distributed unlabeled target domain samples.

Unlike previous efforts, our proposed Adaptively Accumulated Knowledge Transfer framework (A²KT) can simultaneously align the data distribution inter-class center-wise and intra-class sample-wise, both within and across domains. Exploiting prototype classifier and adaptively optimization strategy makes for eliminating the distraction triggered by the misclassified target domain samples.

Given source domain $\mathcal{D}_{s}=\{\mathbf{X}_{s},\mathbf{Y}_{s}\}=\{(\mathbf{x}_{s}^{1},\mathbf{y}_{s}^{1}),\cdots,(\mathbf{x}_{s}^{n_{s}},\mathbf{y}_{s}^{n_{s}})\}$ with labels, and unlabeled target domain $\mathcal{D}_{t}=\{\mathbf{X}_{t}\}=\{\mathbf{x}_{t}^{1},\cdots,\mathbf{x}_{t}^{n_{t}}\}$, where $\mathbf{x}_{s/t}^{i}\in\mathbb{R}^{d}$ is a $d$-dimension source/target sample and $y_{s}^{i}$ is the known label corresponding to the source sample. $\mathcal{D}_{s}$ and $\mathcal{D}_{t}$ are drawn from distribution $P_{s}$ and $P_{t}$ respectively, while $P_{s}\neq P_{t}$. Since the source domain label space $\mathcal{C}_{s}$ subsumes the target domain label space $\mathcal{C}_{t}$, i.e., $\mathcal{C}_{s}\supset\mathcal{C}_{t}$, partial domain adaptation attempts to predict unlabeled target samples with the relevant source knowledge out of the entire well-labeled source domain.

To eliminate the influence of irrelevant source categories, existing partial domain adaptation models mainly design a weighting strategy to select the relevant source categories for effective cross-domain alignment with discrepancy loss (Zhang et al., 2018a) or adversarial loss (Cao et al., 2018a). To mitigate the conditional distribution mismatch across two domains, most of them rely on the pseudo labels of target samples assigned from a source-supervised neural network classifier. Due to the cross-domain distribution gap, such pseudo labels are not reliable, which would further hurt the cross-domain alignment, since the neural network classifier fits perfectly for the source distribution while not for target distribution.

To address these issues, we consider not only to detect the irrelevant source categories to eliminate the negative influence but also select the most confident target samples during cross-domain alignment. Thus, our proposed model can adaptively select out a subset of the target domain samples that are highly affiliated to the source domain and corresponding categories to align across domains. Moreover, the prototype classifier (Snell et al., 2017) is adopted to annotate the target samples via source prototypes, since it can capture the intrinsic structure and semantic knowledge across source and target domain. Exploring the dual classifier architecture consisting of two different types classifiers, prototype classifier and multilayer perceptron classifer, extends the ability of the proposed model to reveal the task-specific knowledge from various perspectives.

The proposed framework, which is shwon as Figure 1, consists of three modules: 1) domain-invariant feature generator $G(\cdot)$, 2) fully-connected multilayer perceptron classifier $C_{N}(\cdot)$, 3) prototype classifier $C_{P}(\cdot)$. $G(\cdot)$ takes the source and target data as input and maps them into a shared embedding space. The extracted features are denoted as $\mathbf{z}_{s/t}$ for the source and target domain respectively.

Entropy minimization regularization is adopted to eliminate the side effect caused by the uncertainty of classifiers, due to the large domain shift and samples which are hard to transfer. Especially during the early training stage, the target domain samples are easy to be assigned to wrong categories and may deteriorate the optimization procedures. We also explore the entropy minimization regularization as:

where $C$ is the number of categories in source domain label space, $N_{t}$ is the number of samples from the target domain.

To sum up, we propose our overall objective function as:

The whole framework consists of a feature generator $G(\cdot)$, a multilayer perceptron classifier $C_{N}(\cdot)$, and a prototype classifier $C_{P}(\cdot)$. As $C_{P}(\cdot)$ is non-parameter, so only $G(\cdot)$ and $C_{N}(\cdot)$ are optimized with the objective as Eq. (8). Specifically, $\mathcal{L}_{y}$ is calculated on the source domain data, while $\mathcal{L}_{em}$ is based on the whole target domain. However, $\mathcal{L}_{intra}$ and $\mathcal{L}_{inter}$ are only based on the filtered out target data, as well as the corresponding source data from the same categories as the filtered target samples pseudo labels.

Office-31 (Saenko et al., 2010) consists of more than 4,000 images from 31 categories office common objects. The dataset includes 3 different domains: Amazon, Webcam, and DSLR. Following the protocol of (Cao et al., 2018a), 9 different partial domain adaptation tasks are explored. For each target domain, we select the 10 shared categories across Office-31 and Caltech-256 (Griffin et al., 2007) dataset and denoted as A10, W10, and D10. The source domain data takes the whole domain data space and denoted as A31, W31, and D31.

In this section, we will comprehensively evaluate our proposed model with several baselines on Office-31 and Office-Home benchmarks in terms of the target samples labels prediction accuracy to manifest the effectiveness of our model.

Specifically, we observe that PDA methods (IWAN, SAN, PADA, DRCN, and ETN) achieve better performance than standard DA efforts such as DAN, DANN, ADDA, and RTN. ETN achieves much greater improvement because it introduces a method to quantify the source samples transferability. Our proposed method can still outperform all compared baselines on most partial domain adaptation tasks and obtain the best average performance.

Table 1 reports the classification accuracy on the Office-31 dataset obtained by all baselines and our model with ResNet-50 as the backbone of feature extractor. It is noteworthy that the prototype classifier $C_{P}(\cdot)$ always generates better performance than the conventional multilayer perceptron classifier $C_{N}(\cdot)$. From the results, the prototype classifier achieves the best performance on 5 out of 6 tasks, compared to all the other baselines. To be specific, the average classification accuracy reaches the best performance $\mathbf{97.72\%}$, and reaches $\mathbf{100\%}$ accuracy on W31 $\rightarrow$ D10 and D31 $\rightarrow$ W10.

Moreover, we also explore the VGG network as the feature extractor backbone on Office-31 dataset and report the results in table 2. Our proposed model achieves the best average performance compared with other baselines. Specifically, compared to the best baseline performance on task A31$\rightarrow$W10, PADA, $C_{N}(\cdot)$ and $C_{P}(\cdot)$ improve the accuracy over 2% to 88.44% and 4% to 90.48%, respectively. It is noteworthy that the improvements of performance with VGG networks as backbone is more significant than using ResNet-50, because the ResNet-50 is more advanced deep convolutional neural networks model, which can generate more task specific discriminate features than VGG networks.

Experiment results on the Office-Home dataset are stated in Table 3. Both $C_{N}(\cdot)$ and $C_{P}(\cdot)$ obtain better performance against other baselines with significant improvements on average classification accuracy ($\mathbf{1.84\%}$ and $\mathbf{2.33\%}$). Moreover, our proposed method achieves more than $\mathbf{5\%}$ accuracy increase compared to the state-of-the-art baseline, e.g., Ar $\rightarrow$ Pr, Cl $\rightarrow$ Pr, etc.

First, visualize the generator $G(\cdot)$ output features before and after the domain adaptation process on task Ar$\rightarrow$Cl on Office-Home, and A$\rightarrow$W on Office-31 dataset in the Fig. 2 (a) and (b). From the results, we observe that our proposed method aligns the source and target domain samples with respect to categories, and tights the compactness of the embedding features to each class centers.

Secondly, we evaluate the contribution of every loss term in Eq. (8) by removing each specific term while keeping other terms as the original framework. The results are shown in Fig. 3. It is noteworthy that both $\mathcal{L}_{intra}$ and $\mathcal{L}_{inter}$ make crucial contribution to the PDA tasks because these two terms are aligning the data distribution inter-classes and intra-class. $\mathcal{L}_{y}$ keeps the model performance on the source domain stable, while it has limited contribution to the PDA process, but cannot be ignored. $\mathcal{L}_{em}$ helps to mitigate the negative transfer influence of the multilayer perceptron classifier $C_{N}$, especially at the beginning of the training stage.

Then, we monitor the training and optimization process of our model. Fig. 4 illustrates the process of the adaptively-accumulated knowledge transfer process. We choose case A31 $\rightarrow$ W10 of Office-31 dataset and show the changing of the filtered out high prediction confidence categories used to align the data distribution across domains. In the beginning, high prediction target samples only spread in only 6 classes, but then more and more categories are involved, and the number finally reaches 11, while the total number of the target domain categories is 10. Although there is an incorrect outlier class involved, the adaptive optimization strategy still significantly narrows the range of the target domain label space.

Moreover, we implement several ablation experiments on the Office-Home dataset with different training details to explore the contribution of our proposed model and optimization strategy, the results are reported in Table 4. ”No Adaptive” denotes the results without the adaptively accumulating knowledge transfer and target samples filtering out process. From the results, compared to our complete A²KT model results, we notice how important the adaptively accumulating knowledge strategy is. ”$C_{N}$ Guide” are the results when we use the $C_{N}$ probabilistic prediction to filter out high confidence target samples for domain alignment, instead of $C_{P}$. The way to decide the threshold is the same as when we use $C_{P}$. The results prove that the multilayer perceptron classifier $C_{N}$ and the prototype classifier $C_{P}$ have different classification philosophy, and using $C_{P}$ probability prediction to accumulate can boost the performance significantly. Finally, we explore the motivation of adopting two different type dual classifiers framework in our model by setting $C_{N}$ and $C_{P}$ both same structure multilayer perceptron classifiers, all other settings and training strategies are the same as before, and the results are reported in ”Same $C_{N}\&C_{P}$”. From the results, we observe that for some cases two same multilayer perceptron classifiers can get slightly better performance than our model, e.g., Ar $\rightarrow$ Rw and Cl $\rightarrow$ Rw. However, for most cases and the average performance, our model with different type classifiers outperforms much more. All the results with different training strategies in Table 4 demonstrate the effectiveness and motivation of our model and optimization strategies.

We present the parameter sensitivity analysis in Fig. 5. We vary $\lambda_{1}$ from 0.0001 to 0.05 and $\lambda_{2}$ from 1 to 3 on four cases on the Office-Home dataset (Ar $\rightarrow$ Pr, Ar $\rightarrow$ Rw, Pr $\rightarrow$ Ar, Rw $\rightarrow$ Cl) to analyze if the model is sensitive to the change of the hyper-parameters. The results in Fig. 5 shows that our model has great stability across cases of the two parameters $\lambda_{1}$ and $\lambda_{2}$.

Finally, we select several representative target samples from task Pr$\rightarrow$Rw on Office-Home dataset and show the predictions of $C_{N}(\cdot)$ and $C_{P}(\cdot)$ in Fig. 6. We notice that some cases only $C_{N}(\cdot)$ or $C_{P}(\cdot)$ can handle, or even neither can predict correctly, which demonstrates the motivation of combine two different type classifiers $C_{N}(\cdot)$ and $C_{P}(\cdot)$ in our proposed model. Besides, we operate the image retrieval task by giving specific labels to retrieve the target samples. The 5 target images with the highest $C_{P}(\cdot)$ prediction confidence and 5 with the lowest in the retrieved images are shown in Fig. 7. The different samples retrieved by $C_{N}(\cdot)$ and $C_{P}(\cdot)$ demonstrate the motivation of integrating various classifiers.