Cross-domain Contrastive Learning for
Unsupervised Domain Adaptation

State-of-the-art contrastive learning frameworks typically use the N-pair loss [44], also known as the InfoNCE [36, 12] and NT-Xent loss [13], to minimize the distance of a positive pair relative to all other pairs. More formally, let $\bm{u}$ and $\bm{v}$ denote $\ell_{2}$-normalized feature representations of a pair and the loss function is then defined as:

where $\bm{v}^{+}\in V^{+}$ and $\bm{v}^{-}\in V^{-}$ represent the positive and negative samples with respect to $\bm{u}$, and $\tau$ is a temperature parameter that is manually set. In practice, a positive pair is derived by two random data augmentations (i.e., blurring and color jittering, e.t.c.) operated on the same image sample, resulting in two correlated views. In contrast, in domain adaptation, it remains unclear how to obtain positive and negative pairs for feature alignment.

We now introduce how to form pairs to learn domain-invariant features with contrastive learning. Since samples from the source domain and the target domain belong to the same set of classes in current UDA settings, we build upon this assumption to reduce domain shift. More specifically, we hypothesize that samples within the same category are close to each other while samples from different classes lie far apart, regardless of which domain they come from. More formally, we consider the $\ell_{2}$-normalized features ${\bm{z}}^{i}_{t}$ from the $i$-th sample ${\bm{x}}_{t}^{i}$ in the target domain as an anchor, and it forms a positive pair with a sample in the same class from the source domain, whose features are denoted as ${\bm{z}}_{s}^{p}$, we formulate the cross-domain contrastive loss as:

where $I_{s}$ denotes the set of source samples in a mini-batch and $P_{s}(\hat{y}^{i}_{t})=\{k\ |\ y_{s}^{k}=\hat{y}^{i}_{t}\}$ indicates the set of positive samples from the source domain that share the same label with the target anchor $x_{t}^{i}$. Since we do not have access to labels of target samples, we use estimated pseudo labels $\hat{y}^{i}_{t}$ (as will be introduced below) to generate pairs. The cross-domain loss forces intra-class distance to be smaller than inter-class distance for samples from different domains so as to reduce domain shift. It is worth pointing out that compared to the standard InfoNCE loss, we sum over all samples in a mini-batch from the source domain that belong to the same category as the anchor $x_{t}^{i}$, which could reduce sampling variance.

In Eqn. 2, we consider samples from the target domain as anchors. Alternatively, we can use source samples as anchors and compute $\mathcal{L}_{CDC}^{s,i}$ similarly by setting $P_{s}(y^{i}_{s})=\{k\ |\ \hat{y}_{t}^{k}=y^{i}_{s}\}$. Then, we combine $\mathcal{L}_{CDC}^{s,i}$ with $\mathcal{L}_{CDC}^{t,i}$ to derive the cross-domain contrastive loss as follows:

The cross-domain contrastive loss aligns features in a bi-directional manner by using anchors from both domains for improved performance. Finally, combining the cross-domain contrastive loss with a standard cross-entropy loss $\mathcal{L}_{CE}$ enforced on the source domain, we have the final objective function for training:

where $\lambda$ controls the trade-off between the two loss terms and ${\bm{\theta}}$ denotes the parameters to be optimized.

Ground-truth labels from the target domain are not available during training, and thus we leverage k-means clustering to produce pseudo labels [22, 15], forming pairs for cross-domain contrastive learning. Since K-means is sensitive to initialization, using randomly generated clusters fails to guarantee related semantics with respect to predefined categories. To mitigate this issue, we set the number of clusters to the number of classes $M$ and use class prototypes from the source domain as initial clusters. The benefits of initializing the cluster centers with class prototypes are twofold: i) source class prototypes can be seen as the approximation of target class prototypes, since features used are high-level and contain semantics information (ii) with the alignment of samples in the same category by CDCL, this approximation will be more accurate as the training continues. More formally, we first compute the centroid of source samples in each category as the corresponding class prototype and the initial cluster center ${\bm{O}}_{t}^{m}$ for the $m$-th class is defined as:

Given features from the target domain, we then perform spherical K-means clustering using these carefully initialized centers. When determining the assignment of each target sample, cosine similarity is adopted to measure the distance between the target feature ${\bm{z}}_{t}^{i}$ and the $m$-th cluster center ${\bm{O}}_{t}^{m}$. Once clustering is finished, each sample in the target domain ${\bm{x}}_{t}^{i}$ is associated with a pseudo label $\hat{y}_{t}^{i}$. To reduce the noise in target pseudo labels, we remove the ambiguous samples far from its assigned clustered centers. Concretely, one target sample will be removed when the cosine similarity between its feature and its assigned cluster center is below a manually set threshold $d$.

In this section, we demonstrate that CDCL can be easily adapted to a newly introduced source data-free setting [15], where a model trained on the source domain is provided yet source data are unavailable due to corruption or privacy concerns. Formally, the goal is to learn a model $f_{t}:X_{t}\to Y_{t}$ and predict $\{y_{t}^{i}\}_{i=1}^{N_{t}}$ with only unlabeled target data $D_{t}$ and a pre-trained source model $f_{s}:X_{s}\to Y_{s}$. The pre-trained source model is obtained by minimizing the cross-entropy loss on the source samples.

It is difficult for most previous UDA methods to adapt to source data-free UDA. For discrepancy-based methods, the predefined domain discrepancies are statistics that should be measured between source samples and target samples. Similarly, for adversarial-based methods, the adversarial discriminators need to be trained with source samples and target samples. Without access to source data, both strategies are not applicable to perform adaptation to the target domain. Besides, for the standard UDA setting, many UDA methods assume that the same feature encoder is shared on the source and target domains. However, this constraint may be hard to implement under source data-free setting since the feature encoder can not be trained on the source and target domain simultaneously. Some UDA methods, e.g., DSBN [30], prove that a domain-specific module in the feature encoder can improve the performance of domain adaptation. Therefore, it is practical to remove the parameter-sharing constraint for feature encoders under source data-free setting.

For CDCL, the lack of samples from the source domain $D_{s}$ makes it challenging to (1) form positive and negative pairs and (2) to compute source class prototypes. We address this issue by replacing source samples with classifier weights from the trained model $f_{s}$. The intuition is that the weight vectors in the classifier layer of a pre-trained model can be regarded as prototypical features of each class learned on the source domain. In particular, we first remove the bias of the fully-connected layer and perform normalization for the classifier.

We use ${\bm{w}}_{s}^{m}$ to denote the weight vector of the $m$-th class in the classification layer ${\bm{W}}_{s}=[{\bm{w}}_{s}^{1},\ldots,{\bm{w}}_{s}^{M}]$ learned on the source domain. Since the weights are normalized, we use them as class prototypes. When adapting to the target domain, we freeze the parameters of the classifier layer to keep the source class prototypes and only train the feature encoder. Through replacing the source samples with source class prototypes, the cross-domain contrastive loss under the source data-free setting can be written as:

Similarly, we estimate labels for samples in the target domain with clustering. However, it is not feasible to compute class prototypes using samples anymore. Instead, we replace Eqn. 5 with class weights:

The final objective of source data-free UDA is:

Compared to Eqn. 4, the cross-entropy loss is not used since the source data are no longer available for supervised training.

We use two public benchmarks to evaluate our method for unsupervised domain adaptation under both standard and data-free settings.

VisDA-2017 [16] is a challenging large-scale benchmark including 12 classes from two domains: the source domain with 152,397 synthetic images, and the target domain contains 55,388 real-world images. Our method is evaluated on the synthesis-to-real domain adaptation task.

Office-31 [6] is a common DA benchmark which contains 4,110 images from three distinct domains, i.e., Amazon (A with 2,817 images), DSLR (D with 498 images) and Webcam (W with 795 images). Each domain consists of 31 object categories. Our method is evaluated by performing domain adaptation on each pair of domains, which generates 6 different tasks.

Compared Approaches. We first report the results of a model trained on the source domain only, and compare with the following state-of-the-art approaches: (a) DANN [5], which utilizes a domain discriminator with adversarial optimization objective to reduce the domain gap. (b) DAN [4] and JAN [7], which learn domain-invariant features by minimizing MK-MMD and Joint MMD. (c) ADR [33], which encourages the encoder to generate more discriminative features by using dropout on the classifier. (d) SAFN [29], which adapts the feature norms of different domains to a large range of values. (e) SWD [18], which measures the dissimilarity between the output of classifiers with sliced Wasserstein discrepancy. (f) MMAN [27], which introduces semantic multi-modality representation learning into adversarial domain adaptation and captures fine-grained category information by multi-channel constraint. (g) CDAN [23], which aligns the conditional distribution in adversarial learning. (h) DSBN [30], which adopts the domain-specific batch normalization in models. (i) BSP [32], which improves the feature discriminability by penalizing the largest singular values. (j) BNM [31], which achieves the discriminability and diversity of the predictions with batch nuclear-norm maximization. (k) MDD [19], which proposes a hypothesis-induced discrepancy for domain adaptation. (l) GVB-GD [24], which proposes a gradually vanishing bridge mechanism for adversarial-based domain adaptation. (m) GSDA [25], which aims to learn domain invariant representations by hierarchical domain alignment. (n) STAR [35], which tries to employ more classifiers by sampling from Gaussian distribution without more parameters. (o) CAN [22], which introduces class information into domain alignment by minimizing the contrastive domain discrepancy. (p) SHOT [15], which develops a framework for data-free UDA based on hypothesis transfer learning. (q) ModelAdapt [43], which adopts a collaborative class conditional generative adversarial networks to avoid using source data.

Among these methods, SHOT and ModelAdapt are developed under the source data-free UDA setting. We implement our method under both standard UDA and source data-free UDA settings.

Network architecture. We adopt a ResNet-50 (for Office-31) and ResNet-101 (for VisDA-2017) pre-trained on ImageNet [46] as the feature encoder in the experiments of the standard UDA task. We replace the last FC layer with the task-specific FC classifier layer. All the network parameters are shared between different domains except those of the batch normalization (BN) layers as we utilize the domain-specific BN [30]. Under the source data-free UDA setting, following [15], we use one bottleneck layer containing a FC layer and a BN layer after convolutional layers in the feature encoder module $g$. Furthermore, we remove the bias of the task-specific FC classifier layer and perform normalization for the classifier.

Training details. The network is trained by using mini-batch SGD with a momentum of 0.9. The initial learning rate $\eta_{0}$ is set as $1e^{-}3$ for pre-trained convolutional layers and $1e^{-}2$ for newly added layers. We employ the same learning rate scheduler $\eta=\eta_{0}\cdot(1+10\cdot p)^{-b}$ as [4, 7, 5], where $p$ denotes training process linearly increase from 0 to 1. Following [22], for Office-31, $b=0.75$ while for VisDA-2017, $b=2.25$. When pre-training models with source samples under source data-free setting, following [15], we randomly split the source dataset into 0.9/0.1 train-validation sets and select the optimal source pre-trained model based on the validation set. We use one RTX 3090 with 24GB for experiments.

Table I and Table II summarize the results of our approach and comparisons with state-of-the-art methods. On both datasets, We can see that all domain adaptation methods achieve significantly better results compared to the source-only method, confirming the importance of feature alignment. On VisDA, we observe CDCL achieves a mean accuracy of 88.6% across all categories, outperforming all state-of-the-art approaches and boosting the accuracy of source-only baseline by 26.2%. In particular, CDCL is better than CAN [22] by 1.4% absolute point and beats JCL [42] by 1.0%, which is notable given that VisDA is a challenging benchmark.

When the source data are no longer available, CDCL achieves a mean accuracy of 87.5%, outperforming the state-of-the-art approach SHOT [15] by 4.6% point, highlighting the effectiveness of our approach. Comparing the data-free setting and the conventional UDA setting, we see that CDCL is slightly (0.9%) worse in the data-free setting. It is noteworthy that CDCL for source data-free setting still surpasses many UDA methods for standard setting. This suggests that one can simply explore a model trained on the source domain for effective transfer without the need to access the source data.

We observe similar trends on Office-31 under both settings. In particular, in the conventional UDA setting, CDCL offers a mean accuracy of 90.6% across 6 different tasks, which is on par with the best results in the literature. Under the data-free setting, CDCL achieves a mean accuracy of 89.3%, comparable to ModelAdapt. It is worth mentioning that results are better on Office-31 compared to VisDA since the dataset is smaller. Besides, due to the small domain gap between D and W, it is easy to achieve high accuracy on Office-31 tasks D$\rightarrow$W and W$\rightarrow$D, even for source-only models. Since domains W and D contain fewer samples compared to domain A, the performance on tasks W$\rightarrow$A and D$\rightarrow$A is relatively poor.

In this section, we conduct a set of ablation experiments to justify the effectiveness of different components and provide discussions.

Positive and Negative Pairs. We mainly form positive pairs and negative pairs using cross-domain samples. We now discuss alternative ways to form pairs and report results on VisDA. In particular, we use the following approach to form pairs: (1) In-domain, where anchors come from both domains but samples that are used to form pairs are from the same domain and same category as the anchor; (2) Combined-domain, where the source and target domains are mixed and pairs are generated by simply considering label information; (3) Cross-domain ($\mathcal{L}^{s,i}_{CDC}$ only), which simply $\mathcal{L}^{s,i}_{CDC}$ in Eqn. 3 using samples in the source domain as anchors; (4) Cross-domain ($\mathcal{L}^{t,i}_{CDC}$ only), which uses samples in the target domain as anchors. From the results shown in Table III, we observe that cross-domain alignment achieves better results compared to performing a simply in-domain alignment. This suggests the importance of forming pairs from two domains in order to produce domain-invariant features. In addition, we observe that mixing both domains together is worse than CDCL, possibly due to the fact that jointly modeling intra-class and inter-class information is challenging. Moreover, the bi-directional use of anchors is better compared to using anchors simply from one domain.

The impact of hyper-parameters. We test the sensitivity of CDCL to the temperature $\tau$ on VisDA-2017 for standard UDA. As shown in Figure 4(a), the accuracies around $\tau=0.05$ are not sensitive. When the $\tau$ grows larger, the accuracy steadily increases before decreasing. Additionally, we study the effect of $\lambda$ on VisDA. Results in Figure 4(b) show that the accuracy around $\lambda=1.6$ are also not sensitive and the gap of accuracies is smaller than 0.2 when $\lambda>1.4$. Therefore, CDCL is not sensitive to its hyper-parameters.

Feature Visualization. We further use t-SNE [47] to visualize features from the source and target domain before and after alignment in Figure 2. We can see that before alignment, source and target features are separated into two clusters, which demonstrates the gap between the two domains. After alignment with CDCL, we see that features from different domains are mixed together and features from different classes are well separated.

Learned Feature Distance. We visualize in Figure 3 top-5 retrieved images given an anchor image at the beginning and the end of training. We observe that in early states, retrieved samples are similar to the anchor in shape but belong to a different category. As training continues, CDCL gradually pulls features from the same class to be closer. This highlights the effectiveness of CDCL in learning domain-invariant features.