Dual-Correction Adaptation Network for
Noisy Knowledge Transfer

In UDA, both source instances $\mathbf{X_{\emph{S}}}={\{x_{i}\}}_{i}^{N_{\emph{S}}}$ and labels $\mathbf{Y_{\emph{S}}}={\{y_{i}\}}_{i}^{N_{\emph{S}}}$ are given, each instance $x_{i}\in\mathbb{R}^{d}$ and $y_{i}\in\{1,...,K\}$, where $N_{\emph{S}}$ is the number of source instances, $d$ is the feature dimension, and $K$ is the class number. Target data $\mathbf{X_{\emph{T}}}={\{x_{i}\}}_{i}^{N_{\emph{T}}}$ is an unlabeled set with $N_{\emph{T}}$ instances. The target distribution is commonly different from source distribution in UDA, i.e., $p_{S}(x)\neq p_{T}(x)$ or $p_{S}(x|y)\neq p_{T}(x|y)$. In real-world adaptation tasks, data collected is usually corrupted with both feature and label noises, thus we usually encounter noisy source data. Specifically, assuming a noise ratio $p_{noise}$, features of a clean instance $x_{i}^{CL}$ will be corrupted by noise $e$ with a probability of $p_{noise}$, i.e, $p(x_{i}=x_{i}^{CL}+e)=p_{noise}$ and $p(x_{i}=x_{i}^{CL})=1-p_{noise}$. At the same time, clean label $y_{i}^{CL}$ is corrupted according to a noise transition matrix $\mathcal{T}\in\mathbb{R}^{K\times K}$, where $\mathcal{T}_{kl}=p(y_{i}=l|y_{i}^{CL}=k)=p_{noise}$ denotes the probability that instances in the k-th class is incorrectly labeled as $l$.

Previous noisy UDA learning methods commonly adopt a signle-directional adaptation, i.e., transfer knowledge only from source domain to target to boost target learning. Moreover, they aim to filter out the noisy instances by some criterion, such as the small loss metric, or prediction discrepancy between different classifiers. Then only clean or trusted source data $\{\mathbf{\bar{X}_{\emph{S}}},\mathbf{\bar{Y}_{\emph{S}}}\}$ are selected for adaptation across domains, as shown in Fig. 1 (a). While in this paper, we propose dual-directional transfer and correction between domains, which is shown in Fig. 1 (b). Specifically, source knowledge is first transferred to target with a forward ST adaptation, together with noise correction for target pseudo-labels $\mathbf{\hat{Y}_{\emph{T}}}$, in order to alleviate the effect of both domain discrepancy and source noises. After that, target knowledge is reversely transferred to source by a backward TS adaptation, in order to further correct both feature and label noises in source domain for better source learning. With such dual adaptation and correction repeated, noises in both domains are corrected mutually and the learning performance of both domains can be promoted gradually.

Moreover, both feature and label noises are addressed in DualCAN, since intensive feature noise may also have negative effect to learning performance. Those noisy instances are identified and further corrected, in order for a recycling of noisy data in adaptation. The details of the proposed DualCAN will be introduced in details in the next section.

The network architecture of DualCAN is shown in Fig. 2. Specifically, the source and target domains share the same feature generator $G$ while different classifiers denoted as $\theta_{\emph{S}}$ and $\theta_{\emph{T}}$, respectively. First, noisy source data $\{\mathbf{X_{\emph{S}}},\mathbf{Y_{\emph{S}}}\}$ and target data $\mathbf{X_{\emph{T}}}$ are inputted into the network, and the initial feature generator and source classifier are obtained by source classification. Then in the ST task, source knowledge is transferred to target by assigning the target pseudo-labels, and those pseudo-labels are further corrected by the NIC module to generate $\mathbf{\overset{\frown}{Y}_{\emph{T}}}$. The corrected target pseudo-labels, along with self-supervised knowledge over target domain, are adopted to train target classifier. Reversely in the TS task, both source features and labels are corrected by NIC, then feature generator $G$ is updated so that target classifier can correctly classify the corrected source data $\{\mathbf{\overset{\frown}{X}_{\emph{S}}},\mathbf{\overset{\frown}{Y}_{\emph{S}}}\}$, in this way, transfer target knowledge to source and generate domain-invariant representation. Finally, a dual adaptation and correction framework is formed, so as to mutually transfer knowledge and correct noise across domains. Further in NIC, there are actually three steps. First, the small-loss criterion is adopted to roughly separate data into trusted and untrusted parts. Secondly, a semi-supervised clustering method is adopted to identify feature and label noises. Finally, those noises are further corrected for full knowledge transfer.

In DualCAN, there is dual-directional knowledge transfer across domains. That is, knowledge transfer iterates between the forward source-to-target (ST) task and backward target-to-source (TS) task, so as to mutually promote learning in each domain.

The initial feature generator and source classifier are obtained by source classification. Then in the forward ST task, knowledge transfer is conducted from label-rich source domain to the label-scared target. Target pseudo-labels $\mathbf{\hat{Y}_{\emph{T}}}={\{\hat{y}_{i}\}}_{i}^{N_{\emph{T}}}$ are first predicted by source classifier $\theta_{\emph{S}}$, i.e.,

In the backward TS task, the learning goal is to transfer knowledge from corrected target domain to weak noisy source. First, both features and labels of untrusted source data are corrected by NIC, generating $\{\mathbf{\overset{\frown}{X}_{\emph{S}}},\mathbf{\overset{\frown}{Y}_{\emph{S}}}\}$ to make it closer to a noise-free distribution. Then, with the target classifier $\theta_{\emph{T}}$ fixed, both feature generator $G$ and source classifier $\theta_{\emph{S}}$ are updated by correctly classifying source instances $\{\mathbf{\overset{\frown}{X}_{\emph{S}}},\mathbf{\overset{\frown}{Y}_{\emph{S}}}\}$, in order to boost the target classifier on source domain to generate domain-invariant representation. The final learning objective is

Finally, the learning algorithm of DualCAN is shown in Algorithm 1 below, and the NIC module will be described in details in the next sub-section.

Due to domain discrepancy and source noise, there will be an estimation bias in the predicted target pseudo-labels, which can also be considered as noisy labels. Those target noises may hurt the source performance in the reverse adaptation. Moreover, there are actually both feature and label noises in the source domain, it will aggravate the target noises, further leading to noise amplification across domains. Thus in DualCAN, there is an extra NIC module for noise identification and correction (NIC) in each iteration. It aims to detect untrusted data, identify them as feature or label noises, and then correct them to achieve full knowledge utilization. NIC actually learns in three steps.

In the first step, both source and target data $\{\mathbf{X_{\emph{S/T}}},\mathbf{Y_{\emph{S/T}}}\}$ are roughly divided into trusted data $\{\mathbf{\overline{X}_{\emph{S/T}}},\mathbf{\overline{Y}_{\emph{S/T}}}\}$ and untrusted data $\{\mathbf{\widetilde{X}_{\emph{S/T}}},\mathbf{\widetilde{Y}_{\emph{S/T}}}\}$ by small-loss criterion, considering that clean instances have smaller losses than noisy instances, i.e.,

In the second step, since the trusted set contains both clean instances and noises, including feature and label noises, those different kinds of instances are further identified to reduce the negative effect. Referring to semi-supervised learning, a deep clustering method is adopted here. Specifically, the trusted data by the previous step are viewed as labeled data, while the untrusted data are treated as unlabeled data. $K$ initial clusters $C_{k}\ (k=1,...,K)$ are first formed over trusted data in terms of their class labels, i.e., the k-th cluster consists of $\overline{N}_{k}$ trusted instances that belong to the k-th class,

For a source instance $\widetilde{x}_{i}$ in the untrusted set it will be identified as,

Finally, those noisy instances are corrected after identification. Specifically, feature noises increase the difficulty of representation learning and usually give the classifier wrong prediction results, while in many cases they do not affect the whole vision in human view. To utilize the useful information in feature noises, we take the class center $\mu_{k_{i}^{*}}$ as disturbance feature, and perform weighted disturbance on noisy features to correct instances at the feature level, i.e.,

The whole learning process of the NIC module is summarized in Algorithm 2 below.

In this sub-section, we give a theory justification for DualCAN, based on a noisy version of generalization bound for target domain.

From proposition 1, The target risk in noisy UDA is bounded by three items. The first item is the empirical risk of noisy source data. In DualCAN, a reverse knowledge transfer from target to source is adopted, the aligned target data can actually be treated as augmented source data after adaptation, thus the target data structure can help correct source noises and reduce source risk. At the same time, noisy source instances are corrected and recycled in DualCAN, which can further reduce the source risk, especially under extremely noisy environment. In fact, a better source learning guarantees better target learning, and reversely, it helps correct source noises and promote source performance as well, finally a virtuous circle is formed. The second item is the distribution discrepancy across noisy source domain and target domain, and the feature noise in source domain will affect the discrepancy [10]. Considering that, feature noises in source domain are also addressed in DualCAN for better domain alignment.

Further, for any labeling functions $f_{1}$, $f_{2}$ and $f_{3}$, we have [32]

The first and second items denote the disagreement between $h$ and the source labeling function $f_{S}$ on soure and target data, respectively. The third item is the gap of source and target functions over target instances, which is actually fixed with given data. Since the source instances are labeled, the first item can be very small. As a result, we focus on the second item $\epsilon_{T}(h,f_{S})=\mathbb{E}_{x\sim\mathbf{X_{\emph{T}}}}[l(h(x),f_{S}(x))]$ for smaller $\lambda$, where $l(\cdot,\cdot)$ is the 0-1 loss. By reversly transfer knowledge from target to source, it is helpful to reduce the discrepancy between $h$ and $f_{S}$ over the target data.

In summary, by dual-directional knowledge transfer between domains, as well as correction and recycling for both feature and label noises, it can be expected to achieve better performance by DualCAN.

Datasets. Office-31 [33] is a standard domain adaptation dataset containing 4652 images with 31 classes. It consists of three domains: Amazon (A), Webcam (W) and DSLR (D). Office-Home [34] has 15599 images with 65 classes. It contains 4 domains with large domain gaps: Artistic (Ar), Clip Art (Cl), Product (Pr) and Real-World (Rw). To introduce noise in those two clean datasets, we follow the protocol in [10] to create corrupted counterparts in 3 different ways: label corruption, feature corruption, and mixed corruption. Label corruption changes the label of each image to other random classes uniformly and equally with probability $p_{noise}$. Feature corruption corrupts image pixels with probability of $p_{noise}$ by Gaussian blur and salt-and-pepper noise. Mixed corruption refers that each image is corrupted by label corruption and feature corruption with probability $p_{noise}/2$ independently. Bing-Caltech [35] is a real noisy dataset consisting of Bing (B) and Caltech-256 (C). The Bing dataset is created by collecting images from the Bing image search engine with class labels in Caltech-256, which natually contains label and feature noises. We take Bing as the noisy source domain and Caltech-256 as the clean target domain.

Noisy UDA. Tables 1 show the results on Office-31 under 40$\%$ label corruption, feature corruption, and mixed corruption, respectively. In mixed corruption case, 40$\%$ means 20$\%$ feature corruption and 20$\%$ label corruption independently. From those tables, we can find that

• •

  Through addressing label noise in learning, both SPL and MentorNet obtain better performance than ResNet, however, since they directly apply source classifier over target instances without taking the domain discrepancy into account, the target performance can be further boosted.

• •

  The UDA methods, such as MDD and CST, consider the domain alignment in learning, whereas there is noise in source domain, and the target learning will be affected by the source noise, leading to negative transfer. As a result, their performance can be further boosted, and it is important to correct the source noise in domain adaptation for better knowledge transfer.

• •

  TCL and RDA consider both source noise and domain discrepancy in learning. They filter out noise instances with small-loss criterion to reduce the negative impact of noise in adaptation, thus achieve better performance than those noise processing and UDA methods.

• •

  Our DualCAN outperforms the other methods in almost all cases, including TCL and RDA, indicating the effectiveness of the proposed dual correction and adaptation.

Table 2 shows the results in extremely noisy environment on Office-Home with 160$\%$ mixed corruption, including 80$\%$ label corruption and 80$\%$ feature corruption independently, as well as naive noisy environment over Bing-Caltech. From this table, we have a similar conclusion that DualCAN outperforms the other methods, since standard UDA methods usually overfit the source noise, while standard noise processing methods do not take the discrepancy across domains into account. Further, DualCAN adopts a dual correction and adaptation strategy, instead of only allowing clean samples to participate training, the noise instances are corrected and recycled in DualCAN. Finally, DualCAN achieves better performance than both TCL and RDA, demonstrating the robustness of our proposal in high noisy environment.

Ablation Study. In order to investigate the contribution of each component in DualCAN better, we perform an ablation study on Office-31 with 40$\%$ mixed corruption by deleting some components in DualCAN, and the results are reported in Table 5. From Table 5, it can be found that abandoning either label correction or feature correction leads to performance degradation, thus it is reasonable to address both label and feature noises in learning. At the same time, label noise has a greater impact to performance, demonstrating the necessary for different treatment of label and feature noises. Furthermore, the performance of DualCAN is better than source correction or target correction alone, which indicates that the dual correction is indeed helpful for alleviating the negative effects of noisy instances, and improving the quality of pseudo-labels, thus boosts both source and target learning.