Source-Free Unsupervised Domain Adaptation with Hypothesis Consolidation of Prediction Rationale

Unsupervised domain adaptation aims to transfer knowledge learned from a labeled source domain to an unlabeled target domain. Various approaches have been proposed to address this task, including discrepancy minimization [42, 40, 41], adversarial learning [36, 22, 37, 38], and contrastive learning [39, 28]. Recently, self-training using labeled source data and pseudo-labeled target data has emerged as a prominent approach in unsupervised domain adaptation (UDA) research [43, 44, 45, 46, 47]. However, these methods typically rely on access to the source data, making them inapplicable when source data is unavailable.

Source-free unsupervised domain adaptation involves adapting a pre-trained model from a source domain to a target domain without access to source data+labels or target labels. Existing SFUDA methods can be broadly categorized into two classes: i) Label Refinement: Methods such as SHOT [2], G-SFDA [5], NRC [6], and GPL [13] focus on refining pseudo labels. SHOT generates pseudo labels using centroids obtained in an unsupervised manner. G-SFDA, NRC, and GPL refine pseudo labels through consistent predictions and nearest neighbor knowledge aggregation from local neighboring samples. ii) Contrastive Feature Learning: Approaches like HCL [9], C-SFDA [12], AdaContrast [10], GPL [13], and DaC [11]. HCL and C-SFDA use a contrastive loss similar to moco [3], where positive pairs consist of augmented query samples and negatives are other samples. AdaContrast and GPL exclude same-class negative pairs based on pseudo labels. DaC divides the target data into source-like and target-specific samples, computes source-like class centroids, and generates negative pairs using these centroids. These methods aim to tackle SFUDA by refining pseudo labels or leveraging contrastive feature learning, demonstrating the potential of different strategies in addressing the challenges of adapting models without access to labeled source data or target label.

The first step of our approach is to make an initial adaptation to reduce the domain gap. We empirically find such a step can be beneficial for the following steps. We develop a pre-adaptation strategy by encouraging a smooth prediction on the data manifold. ¹¹1Other pre-adaptation approaches may also work, such as the method in [2], please refer to Sec. IV-F for more experimental evidence. Specifically, we create a memory $Q\in\mathbb{R}^{N_{q}\times d}$ to store $N_{q}$ randomly sampled image embedding and update it after each batch training. Then for each target sample $x_{i}$, we find the $z$-nearest neighbor $\mathcal{NN}(x_{i})$ and $z$-samples $\mathcal{FN}(x_{i})$ that are furthest to $x_{i}$ based on the Euclidean distance between the image embedding of $x_{i}$ and embedding in $Q$ ($d=256$ and $z=3$ in our implementation). Then we optimize the following objective:

where KL represents Kullback-Leibler divergence and $p$ denotes the posterior probability predicted from the source model. $N_{B}$ is the number of samples within a mini-batch. The first term is used to ensure similar samples have similar predictions. However, using the first term alone may lead to a trivial solution that assigns identical prediction for every instance. Thus we use the second term to counter-act it as it ensures that the least similar samples should have divergent posterior probabilities, i.e., the inner product between posterior should close to zero.

After pre-adaptation, the model generally exhibits improved adaptation to the target domain. However, there may still be instances where the model produces incorrect predictions, making it challenging to rectify misclassifications solely based on predicted posterior probabilities. Therefore, in the second step, we explore a more robust methodology for analyzing predictions.

We begin by considering multiple prediction hypotheses for each individual instance. Specifically, for each instance, we consider the top $\tilde{k}$ classes with the highest posterior probabilities as potential prediction hypotheses, denoted as $(x_{i},y^{h}_{ik})$, $k\in$ top $\tilde{k}$. In other words, we acknowledge the correct class label could exist within one of these top $\tilde{k}$ classes, even though we do not know which one.

To further analyze each hypothesis $(x_{i},y^{h}_{ik})$, we calculate the GradCAM [1] to identify the regions that contribute to supporting the prediction for $y^{h}_{ik}$, resulting in a representation called the rationale representation $a_{ik}$. This rationale representation encodes the evidence supporting the corresponding hypothesis. Drawing inspiration from prior work [48, 49], we formally calculate $a_{ik}$ using following equation:

where $a_{ik}\in\mathbb{R}^{d^{\prime}}$, $\phi(x_{i})\in\mathbb{R}^{H\times W\times d^{\prime}}$ is the feature map of the last convolutional layer of the network with $H$ height, $W$ width, and $d^{\prime}$ channels. $[\phi(x_{i})]_{m,n}\in\mathbb{R}^{d^{\prime}}$ is the feature vector located at the $(m,n)$-th grid. $logit(y^{h}_{ik})$ is the logit for class $y^{h}_{ik}$, $[\cdot]_{+}=max(\cdot,0)$. $\left[\frac{\partial logit(y^{h}_{ik})}{\partial[\phi(x_{i})]_{m,n}}^{\top}[\phi(x_{i})]_{m,n}\right]_{+}$ is equivalent to GradCAM value at the $(m,n)$-th grid. Essentially, the calculation of $a_{ik}$ performs weighted average pooling over $\phi(x_{i})$ according to the GradCAM. Figure 1 shows the GradCAM calculated from different hypotheses for the same image. Upon observation, we notice that even if the ground-truth class is not ranked as the top prediction by the model, its associated rationale remains reasonable and similar to the common rationale patterns for the corresponding class. This inspires us to leverage this observation to analyze the model’s current predictions. For example, if an instance has a prediction hypothesis that exhibits a rationale similar to the corresponding class’s common rationale but is not ranked as the top prediction, then the top prediction may not be correct.

Formally, we calculate the class-wise rationale centroid as the average rationale representation from each hypothetical class, representing the common rationale for each class:

where $c$ represents a class and $\mathds{1}(y^{h}_{ik}=c)=1$ if $k=c$. The idea of using multiple hypotheses with the rationale representation is illustrated in Figure 2.

Next, we generate a ranking index $r_{ik}$ for each prediction hypothesis $(x_{i},a_{ik},y^{h}_{ik})$ by ranking the Euclidean distance between $a_{ik}$ and its corresponding rationale centroid $\bar{a}_{y^{h}_{ik}}$, i.e., the centroid for class $y^{h}_{ik}$, in the ascending order. For each instance $x_{i}$, we obtain $\tilde{k}$ ranking indices ${r_{ik}}$, $k\in$ top $\tilde{k}$ classes, one for each hypothesis. Then, a hypothesis $\{x_{i},y_{ik^{\prime}}\}$ is considered reliable if it satisfies the following two conditions: (1) $r_{ik^{\prime}}<\tau_{1}$, indicating the rationale for $\{x_{i},y_{ik^{\prime}}\}$ is typical as its rationale representation is close to the rationale centroid. (2) $r_{ij}>\tau_{2}~{}~{}\forall j\neq k^{\prime}$, where $\tau_{2}>\tau_{1}$ are two predefined ranking thresholds. The second condition ensures that there are no conflicting hypotheses, i.e., no other hypothesis is likely to be true for the same instance as their rationale appears to be unusual.

With those criteria, we can collect a set of reliable hypotheses $\mathcal{P}$ as samples with their corresponding hypothetical labels. Representative examples of this procedure are depicted in Figure 3. It is important to note that in the second step, we aim to select the most reliable hypothesis rather than correcting hypotheses. This is because we believe that the task of correcting predictions or hypotheses can be better accomplished through the use of semi-supervised learning, which allows for the gradual propagation of pseudo-labels.

By focusing on identifying the most reliable hypothesis based on the proximity of the rationale representation to the rationale centroid and the absence of conflicting rationales, we can create a high-quality set of pseudo-labeled samples (see Section IV-E). These pseudo-labels can then be used in a semi-supervised learning framework to refine the model’s predictions and gradually improve its performance.

After completing the second step of hypothesis consolidation, we obtain a reliable pseudo-label set $\mathcal{P}$, while the remaining samples are treated as the unlabeled set $\mathcal{U}$. At this stage, we are ready to apply a semi-supervised algorithm to perform the final step of adaptation. For this purpose, we utilize one of the state-of-the-art semi-supervised methods, FixMatch [34], which combines consistency regularization and pseudo-labeling to address this task.

Specifically, we start by sampling a labeled mini-batch $\mathcal{B}_{l}$ from the reliable pseudo-label set $\mathcal{P}$ and an unlabeled batch $\mathcal{B}_{u}$ from the unlabeled set $\mathcal{U}$. We then optimize the following objective function using these batches:

where $\hat{y}_{u}=\mathop{\arg\max}\limits_{c}p(y=c|\mathcal{A}_{w}(x_{u}))$. $\mathcal{A}_{w}(\cdot)$ and $\mathcal{A}_{s}(\cdot)$ are the weakly-augmented and strongly-augmented operations, respectively. $\tau$ is the threshold defined in FixMatch to identify reliable pseudo-label (we set the same with FixMatch as 0.95), and $CE$ is the cross-entropy between two probability distributions.

We present the overall training process of our proposed SFUDA method in Algorithm 1.

Office-Home [31] consists of 15,500 images categorized into 65 classes. It includes four distinct domains: Real-world (Rw), Clipart (Cl), Art (Ar), and Product (Pr). To evaluate the proposed method, researchers perform 12 transfer tasks on this dataset, involving adapting models across the four domains. The evaluation reports each domain shift Top-1 and the average Top-1 accuracy. Originally, the DomainNet dataset [29] consisted of over 500,000 images, including six domains and 345 classes. For our evaluation, we follow the approach described in [21] and focus on four domains: Real World (Rw), Sketch (Sk), Clipart (Cl), and Painting (Pt). We assess our proposed method on seven domain shifts within these four domains. VisDA-C [30] contains 152,000 synthetic images from the source domain and 55,000 real object images from the target domain. It consists of 12 object classes, and there is a significant synthetic-to-real domain gap between the two domains. Our evaluation reports per-class Top-1 accuracies, as well as the average Top-1 accuracy on this dataset.

To ensure fair comparisons with previous work [2, 10, 12], we employ the ResNet-50 [3] as the network backbone for the Office-Home and DomainNet datasets, and ResNet-101 for the VisDA-C dataset. The network architecture follows the same configuration as SHOT [2]. Specifically, we replace the original fully connected (FC) layer in ResNet-50/101 with a bottleneck layer of 256 dimensions and apply batch normalization [32]. This modified setup serves as the feature extractor+projector head, producing feature representations and embedding of dimensions $d^{\prime}=2048$ and $d=256$, respectively. Additionally, we include an extra fully connected layer with weight normalization [33] as a task-specific classifier.

In the first step of model pre-adaptation, we use a batch size of 64. The value of $\lambda$ is set as $\lambda=\lambda_{0}\cdot(1+10\cdot p^{\prime})^{-5}$, where $\lambda_{0}=1$, and $p^{\prime}$ represents the training progress variable ranging from 0 to 1, calculated as $\frac{iter}{max\_iter}$. In the second step of hypothesis consolidation, we set the number of nearest/furthest neighbor per instance $z$ as 3, and set hypothesis per instance $\tilde{k}$ as 4, respectively. The ranking thresholds $\tau_{1}$ and $\tau_{2}$ are determined as a percentage of the total number of samples on the three datasets, specifically set at 0.8% and 1.6%. In the third step of semi-supervised learning, we set the size of $\mathcal{B}_{l}$ and $\mathcal{B}_{u}$ to 64.

We use the SGD optimizer with a momentum of 0.9 and a weight decay of $1e^{-3}$ for all datasets. The learning rate is set as $1e^{-4}$ for all datasets, except for the bottleneck layer and the additional fully connected layer, where it is set as $1e^{-3}$. We train for 40 epochs on the Office-Home and DomainNet datasets, where 9 epochs are dedicated to the model pre-adaptation. For the VisDA-C dataset, we train for 15 epochs, with 7 epochs allocated for the model pre-adaptation. All images from the datasets undergo augmentation, including weak and strong augmentation. Weak augmentation involves a standard flip-and-shift augmentation strategy, while strong augmentation is similar to the approach used in the work of [34].

In this section, we assess both the quality and quantity of pseudo-labels generated by each component of our method, comparing them with the source model alone and SHOT. Pseudo-label quantity is measured by the ratio of selected samples to the total samples, while pseudo-label quality is defined as the precision of the selected samples. The results are shown in Table X. As seen, using the original source model generates good pseudo-label quality within the selected group, but only a small number of samples satisfy the high confidence condition. On the other hand, SHOT and PA select a large number of samples but with a relatively poor quality of approximately 80%. In comparison, PA+HCPR achieves both good pseudo-label quality (90.76%), and a substantial quantity of pseudo-labels (24.65%). When comparing HCPR only and PA only, we observed that PA generates nearly four times as many pseudo-labels as HCPR but with lower quality. This suggests the presence of significant noise in the pseudo-labels generated by PA.

The training progress to both the quantity and quality of pseudo-labels can be shown in Figure 9. Our findings revealed that in the initial step with PA (0-9th epochs), there is a significant increase in the quantity of pseudo labels, albeit accompanied by a gradual decrease in their quality. However, with the assistance of HCPR (after 9th epch, before 10th epoch), the quality of pseudo-labels experiences a significant increase, accompanied by a substantial quantity. In the subsequent third step involving FM (10-40th epochs), the quality of pseudo labels has a gradual improvement, which subsequently stabilizes at a consistent level.

The proposed method can be seamlessly integrated into existing network architectures, such as SHOT [2] and AaD [51]. Specifically, we replace the pre-adaptation phase in our first step with SHOT and AaD, resulting in the combined approach referred to as “SHOT+Ours” and “AaD+Ours”. The integration process can be summarized as follows: first, pseudo labels are generated using SHOT’s unsupervised nearest class centroid approach and AaD’s feature clustering and cluster assignment approach. Then, to refine these pseudo labels and address potential noise and inaccuracies, we utilize hypothesis consolidation of prediction rationale. The refined pseudo-label set is used as the labeled dataset, while the remaining samples are treated as unlabeled. Consequently, the SFUDA problem is transformed into a semi-supervised learning problem. The experimental results, as shown in Table VII, demonstrate the superiority of the proposed method integrated into the SHOT and AaD objectives. Across the Office-Home (Avg. $\uparrow$ 1.6% and $\uparrow$ $0.7\%$), VisDA-C (Avg. $\uparrow$ 3.9% and $\uparrow$ $0.4\%$), and DomainNet-126 (Avg. $\uparrow$ 2.8% and $\uparrow$ $3.7\%$) datasets, the integrated approach consistently outperforms the baseline of SHOT and AaD. This indicates that our method complements existing SFUDA baselines and consistently improves their performance by incorporating our approach as a replacement for the model pre-adaptation phase.

In t-SNE visualization, we compare the results with the state before adaptation by examining three approaches: source model only, AaD [51], and our method shown in Figure 10. The source model only demonstrates shortcomings, experiencing false predictions within each class and struggling to establish clear intra-class boundaries. While AaD generally achieves accurate predictions within each class, it falls short in generating clear intra-class boundaries. In contrast, our method excels in achieving accurate predictions within each class and successfully generates distinct intra-class boundaries, which showcases its ability to enhance prediction accuracy and produce well-defined intra-class boundaries.