Bures Joint Distribution Alignment with Dynamic Margin for Unsupervised Domain Adaptation

Most recent UDA methods are motivated by the seminal theoretical work of Ben-David et al. [26], which proposes the $\mathscr{A}$-distance to estimate the discrepancy of feature marginal distributions between source and target domains. Then Blitzer et al. [27] develop the $\mathscr{H}$-distance to deduce a uniform learning bound, which minimizes the convex combination of empirical risks among various domains. A thorough theoretical investigation for UDA is presented in [28] in terms of the $VC$ dimension. In light of these foundation works, minimizing the domain distribution discrepancy has been used extensively for UDA. The covariance matrix is used in several works [12, 13] to model the difference between the feature distributions of various domains. Ren et al. [6] exploit a low-rank representation to reduce the marginal distribution shift and stress the group compactness of feature representations. Li et al. [29] design an adaptive batch normalization operation to align the feature distributions of different domains. Luo et al. [30] use a sequence of latent manifolds to describe the diverse domains and employ the manifold metrics to model the domain discrepancy. While methods discussed above are mainly based on distribution discrepancy, there have been several works to address the problem of UDA by applying adversarial learning. Ganin et al. [19] introduce domain adversarial neural network (DANN) to align the feature distributions with a domain discriminator via the adversarial mechanism. Tzeng et al. [31] use separate feature extraction networks for diverse domains and train the target domain adversarially until it is adapted to the source domain. Satio et al. [32] propose maximum classifier discrepancy (MCD), which utilizes two category classifiers to simulate a domain discriminator and train the feature generator in an adversarial fashion. These approaches focus on learning a domain-invariant feature representation among the source and target domains, the available label information from the source domain is not fully exploited during the adaptation process.

Domain shift usually exists both marginal distributions and conditional distributions. Thus, it is more important to investigate the domain difference among the joint distributions. To account for the label information, Li et al. [33] propose a novel domain distance, Maximum Density Divergence (MDD), which measures both the inter-domain divergence and the intra-class density. Li et al. [34] also propose Adaptation by Adversarial Attacks (AAA), which addresses the UDA task from the view of the adversarial attack, and generalizes the UDA task to the scenario in which the source data or the target data is absent during training. Then, Faster Domain Adaptation (FDA) [35] is proposed, which adaptively selects the number of neural network layers for various samples and significantly speeds up the adaptation process. Li et al. [36] further propose Heterogeneous Domain Adaptation (HDA), which uses dictionary learning to handle heterogeneous feature spaces with different dimensions, and minimizes the marginal distribution gaps across different domains. Zhang et al. [37] use a domain discriminator to align the marginal distribution with adversarial learning, and develop a class-wise divergence to address the gap between conditional distributions. Du et al. [38] align the marginal distribution with the domain discriminator, it also utilizes the disagreement of two category classifiers to measure the discrepancy of conditional distributions. Yu et al. [39] use two domain discriminators, one for the marginal distributions, and the other for the conditional distributions. Wang et al. [40] propose Dynamic Distribution Adaptation (DDA) to address the different effects of marginal and conditional distributions in the adaptation process. Wang et al. [41] train a domain-invariant classifier in Grassmann manifold and minimizes the structural risk. Luo et al. [42] propose a category-level adversarial network to enforce local semantic consistency. Then, Luo et al. [43] introduce a novel Significance-aware Information Bottleneck (SIB) to improve the former-mentioned category-level adaptation. Luo et al. [44] further propose a novel Significance Aware Layer, which uses the channel-wise significance of each semantic feature to balance the information from different categories. Although these works take the label information into consideration, they align the marginal distributions and the conditional distributions separately.

Long et al. [22] propose Joint Distribution Adaptation (JDA) by aligning both the marginal feature distributions and the conditional distributions in a dimensionality reduction procedure. Chen et al. [23] exploit nonlinear projections to match the projected distributions. Long et al. [45] propose the Joint Maximum Mean Discrepancy (JMMD) to measure the discrepancy of the joint distributions of feature representations from different layers in the neural network between diverse domains. Li et al. [46] propose to match the marginal and conditional distributions via the maximum mean discrepancy (MMD) distance. Long et al. [47] develop a multilinear map to model the interactions across the feature and label predictions. Zhang et al. [20] propose a symmetric framework to perform domain confusion both on label-level and feature-level. Gu et al. [48] take the label information into consideration and propose to correct the pseudo-labels based on the feature distance. Ni et al. [49] propose to match both the marginal distributions and conditional distributions by training a domain-invariant kernel. Courty et al. [10] investigate an optimal transport transformation to align the joint distributions between different domains and propose joint distribution optimal transport (JDOT). Damodaran et al. [24] incorporate JDOT with deep neural networks and propose deep joint distribution optimal transport (DeepJDOT). Although these works have shown promising performance in UDA, some of them reduce the distribution discrepancy in the Euclidean space, which may ignore the nonlinear structures underlying the data, and the others usually train the deep neural networks and the optimal transport task alternately, which is unstable and hard to train.

Optimal transportation (OT) [50] has been an important topic in both the computer vision and machine learning communities. OT investigates how to effectively transport a probability distribution to another one, and the total cost during the transportation is related to a distance between diverse probability distributions. Assume that we are given two probability measures $\mu$ and $\nu$, defined respectively on some measure spaces $\mathcal{Z}_{1}$ and $\mathcal{Z}_{2}$. The cost function $D(\mathbf{z}_{1};\mathbf{z}_{2})$ defined on $\mathcal{Z}_{1}\times\mathcal{Z}_{2}$ depicts how much it costs to transport a unit of probability mass from the point $\mathbf{z}_{1}$ to the point $\mathbf{z}_{2}$, where $\mathbf{z}_{1}\in\mathcal{Z}_{1}$ and $\mathbf{z}_{2}\in\mathcal{Z}_{2}$. The transport coupling $\gamma(\mathbf{z}_{1};\mathbf{z}_{2})$ is a joint probability measure which can be interpreted as the amount of probability mass transported from the point $\mathbf{z}_{1}$ to the point $\mathbf{z}_{2}$. The optimal transport cost between the two measures $\mu$ and $\nu$ defines the Wasserstein distance [50], which is denoted as

Courty et al. [51] propose a regularized optimal transportation algorithm to perform the alignment of the feature representations among the source and target domains. Yang et al. [52] present a novel robust regression scheme by integrating OT with convex regularization terms. Li et al. [18] use an attention-aware transport distance to directly minimize the domain discrepancy. Usually, it is complicated to compute the Wasserstein distance as it requires for solving a linear programming task iteratively. Thus, it is not straightforward to incorporate the Wasserstein distance with deep neural networks. To address this issue, Bhatia et al. [53] extend the Wasserstein distance to the Bures-Wasserstein distance, which is easy and computable. Let $S^{+}(n)$ be the set of all PSD matrices of size $n\times n$. The Bures-Wasserstein distance [53] between two probability measures $\mu$ and $\nu$ is defined as

where $\mathbf{\Sigma}_{1}$, $\mathbf{\Sigma}_{2}\in S^{+}(n)$ are the covariance matrices of $\mu$ and $\nu$, respectively. The original OT problem is discussed in the Euclidean space. Recently, several works [25, 4] generalize the original optimal transport method to infinite-dimensional feature spaces. An equivalent and computable closed-form formulation [25] of the Kantorovich optimal transport problem is provided via the kernel methods, which provides a probable solution for measuring the discrepancy among diverse distributions. The theorem in [25] states that if $\mu$ and $\nu$ are two Gaussian measures with the same expectation, the Wasserstein distance is just the corresponding Bures-Wasserstein distance between their covariance matrices on the manifold of Positive Semi-Definite (PSD) matrices.

Without loss of generality, we consider a $C$-class problem as an example. Let $\Omega$ be the input space and $\mathcal{C}$ the label space, where $\mathcal{C}=\{1,\cdot\cdot\cdot,C\}$. $X_{s},X_{t}\subset\Omega$ denote the source and target samples, $Y_{s},Y_{t}\subset\mathcal{C}$ denote the label of source and target samples, where $Y^{t}$ is not available during training. We define a domain $\mathcal{D}$ as a pair consisting of a joint distribution $P$ on the input space and a labeling function $f:\Omega\rightarrow\mathcal{C}$. $P_{s}(X_{s},Y_{s})$ and $P_{t}(X_{t},Y_{t})$ denote the joint distributions of the source and target domains, respectively. Our goal is to learn effective features such that the model also makes accurate predictions on the target domain.

Inspired by recent works about the joint distribution alignment for UDA [10, 24] as well as new proceedings [25] on the optimal transport theory in RKHS, we propose to align the joint distributions of the two domains in the infinite-dimensional kernel spaces. The architecture is depicted by Fig. 2. Our model consists of a feature extractor $G$ with parameters $\theta_{G}$, a category classifier $F$ with parameters $\theta_{F}$. The feature extractor $G$ maps the samples to the representation space $\mathcal{G}$. $\mathcal{G}_{s}$ and $\mathcal{G}_{t}$ represent the subspace for $G(X_{s})$ and $G(X_{t})$, respectively. The category classifier $F$ outputs the $C$-dimensional probability prediction vector for the input representation. The optimal coupling between the joint distributions $P_{s}(G(X_{s}),Y_{s})$ and $P_{t}(G(X_{t}),Y_{t})$ is denoted as $\gamma^{\star}$, and we can get two sub-couplings $\gamma_{1}=\gamma^{\star}\big{|}_{\mathcal{G}_{s}\times\mathcal{G}_{t}}$ and $\gamma_{2}=\gamma^{\star}\big{|}_{\mathcal{C}\times\mathcal{C}}$. Since $Y_{t}$ is unavailable during training, we assign soft pseudo-labels $\hat{Y}_{t}$ to the unlabeled $X_{t}$, i.e., $\hat{Y}_{t}=F(G(X_{t}))$. We use the following theorems to model the discrepancy between the joint distributions of the two domains.

Theorem 1 [50]. Let ($\mathcal{Z}_{1},\mu$) and ($\mathcal{Z}_{2},\nu$) be two Polish probability spaces; let $a:\mathcal{Z}_{1}\rightarrow\mathbb{R}\cup\{-\infty\}$ and $b:\mathcal{Z}_{2}\rightarrow\mathbb{R}\cup\{-\infty\}$ be two upper semicontinuous functions such that $a\in L^{1}(\mu)$, $b\in L^{1}(\nu)$. Let $D:\mathcal{Z}_{1}\times\mathcal{Z}_{2}\rightarrow\mathbb{R}\cup\{+\infty\}$ be a lower semicontinuous cost function, such that $D(\mathbf{z}_{1};\mathbf{z}_{2})\geq a(\mathbf{z}_{1})+b(\mathbf{z}_{2})$ for all $\mathbf{z}_{1}\in\mathcal{Z}_{1}$, $\mathbf{z}_{2}\in\mathcal{Z}_{2}$. Then there is a coupling of $(\mu,\nu)$ which minimizes the total cost $d_{W}(\mu,\nu)$ among all possible couplings $\gamma(\mathbf{z}_{1};\mathbf{z}_{2})$.

Theorem 2 [50]. Any sub-coupling is still optimal.

Theorem 1 states the existence of an optimal coupling. Theorem 2 declares that any induced sub-coupling is also optimal.

In light of the above theorems, we can compute the squared Wasserstein distance between joint distributions as follows:

where $\mathbf{g}_{s}\in G(X_{s}),\mathbf{g}_{t}\in G(X_{t})$, $\mathbf{y}_{s}\in Y_{s}$, and $\hat{\mathbf{y}}_{t}\in\hat{Y}_{t}$.

The original OT problem is discussed in the Euclidean space. It is widely believed that the kernel methods are effective in exploring those complex data with nonlinear structures. Thus, the Bures-Wasserstein distance corresponding to the original OT problem is generalized to the infinite-dimensional settings, and the kernel Bures-Wasserstein distance [25] between the covariance operators in RKHS is induced.

Denote a nonempty set as $\mathcal{X}$, and let $\mathcal{H}$ be a Hilbert space of $\mathbb{R}$-valued function defined on $\mathcal{X}$. Let $k$ be a positive definite kernel on $\mathcal{X}\times\mathcal{X}$, and $\mathcal{H}_{\mathcal{X}}$ be the reproducing kernel Hilbert space generated by $k$. Define the implicit feature map $\phi:\mathcal{X}\rightarrow\mathcal{H_{\mathcal{X}}}$ as $\phi(\mathbf{x})=k(\cdot,\mathbf{x}),\forall\mathbf{x}\in\mathcal{X}$. Let $\mathbf{U}=[\mathbf{u}_{1},\cdots,\mathbf{u}_{m}]$ and $\mathbf{V}=[\mathbf{v}_{1},\cdots,\mathbf{v}_{n}]$ be two sample matrices from two probability measures $\mu$ and $\nu$, respectively. Let $\mathbf{\Phi}_{U}=[\phi(\mathbf{u}_{1}),\cdots,\phi(\mathbf{u}_{m})]$ and $\mathbf{\Phi}_{V}=[\phi(\mathbf{v}_{1}),\cdots,\phi(\mathbf{v}_{m})]$ be the mapped data matrices. Let $\mathbf{K}_{UU}$,$\mathbf{K}_{UV}$ and $\mathbf{K}_{VV}$ be the kernel matrices defined by $(\mathbf{K}_{UU})_{ij}=k(\mathbf{u}_{i},\mathbf{u}_{j})$,$(\mathbf{K}_{UV})_{ij}=k(\mathbf{u}_{i},\mathbf{v}_{j})$, and $(\mathbf{K}_{VV})_{ij}=k(\mathbf{v}_{i},\mathbf{v}_{j})$. Let $\mathbf{H}_{n}=\mathbf{I}_{n\times n}-\frac{1}{n}\mathbf{1}_{n}\mathbf{1}_{n}^{T}$ and $\mathbf{H}_{m}=\mathbf{I}_{m\times m}-\frac{1}{m}\mathbf{1}_{m}\mathbf{1}_{m}^{T}$ be two centering matrices. The kernel Bures-Wasserstein distance [25] is expressed as:

where $\|\cdot\|_{*}$ denotes the nuclear norm, i.e., $\|\mathbf{A}\|_{*}=\sum_{i=1}^{r}\sigma_{i}(\mathbf{A})$, and $\sigma_{i}(\mathbf{A})$ is the singular values of matrix $\mathbf{A}$. This is an equivalent and computable formulation [25] of the kernel Bures-Wasserstein distance, the form of which is fully determined by the kernel function.

In order to exploit the capability of capturing nonlinear structures of kernel methods, we propose to match the joint distributions of source and target domains in RKHS. Specifically, we incorporate the formulation of the kernel Bures-Wasserstein distance and propose the following Bures joint distribution alignment loss

where the kernel matrices follow the definition of Eq. (4), such as $(\mathbf{K}_{g_{s}g_{t}})_{ij}=k(\mathbf{g}_{i}^{s},\mathbf{g}_{j}^{t})$,$(\mathbf{K}_{y_{s}\hat{y}_{t}})_{ij}=k(\mathbf{y}_{i}^{s},\hat{\mathbf{y}}_{j}^{t})$. We use the Gauss kernel function in this paper, i.e., $k(\mathbf{g}_{i},\mathbf{g}_{j})=\exp(-\|\mathbf{g}_{i}-\mathbf{g}_{j}\|^{2}/\sigma^{2})$, where $\sigma^{2}$ is set as the mean of all the squared Euclidean distance $\|\mathbf{g}_{i}-\mathbf{g}_{j}\|^{2}$.

Our proposed Bures joint distribution alignment loss is differentiable by means of the closed form solutions, so we can directly model the joint distribution shift in the reproducing kernel Hilbert space. We also train the feature extractor $G$ and the category classifier $F$ with the cross-entropy loss on the source domain, i.e.,

The goal of UDA is to train a model which makes accurate and robust predictions for the unlabeled target data. Therefore, enhancing the discriminative ability of the feature representations between categories may be of utmost importance. Recently, some works about contrastive learning are proposed. Wang et al. [54] use self-supervised learning method to reduce the discrepancy across domains. Park et al. [55] employ the mutual information between a feature and its label to train class-level discriminative features. Both [54] and [55] are based on the InfoNCE Loss [56], which minimizes the distance of all positive pairs relative to negative pairs, and they may probably encounter the potential risk of overfitting. Vikas et al. [57] propose a method for data augmentation, which mixes samples either at the input or hidden-state levels to create positive and negative samples. But those synthesis samples will introduce the Mixup-noise [57]. Intuitively, representations belonging to the same category should be located nearby in the representation space, and representations belonging to the different categories should be pushed far away. The triplet loss [58, 59, 60, 61, 62] is effective in achieving the desired goal. Let $\mathbf{g}_{i}$ denote the feature representation of a sample $(\mathbf{x}_{i},y_{i})$, the triplet loss is defined as

where $\alpha^{\prime}$ represents a fixed margin value, $\mathbf{g}_{j}^{pos}$ denotes the feature representation that has the same category label with respect to $\mathbf{g}_{i}$, $\mathbf{g}_{j}^{neg}$ denotes the feature representation with different category label. We try to incorporate the triplet loss into our method, and train the network using the Bures joint distribution alignment loss, the cross-entropy loss, and the triplet loss. The experimental settings are described in Section IV, and is termed as BJDA($w/o$ $L_{dmc}$)+$L_{trip}$. However, the experiment results show some limitations. As shown in Fig. 3, different margin values have a significant influence on the performance. It motivates us to find a dynamic margin to avoid the tedious task of determining the margin by cross-validation. On the other hand, the fixed margin may be an overly strict objective, as it is difficult to make all the clusters keep the same margin far away from each other in the high-dimensional representation space. In this work, we propose a dynamic margin based contrastive learning phase to address these issues and preserve the category discriminative information from the source domain.

Naturally, the prototypes, which play the role of each class center, contain much more robust category discriminative information than the sample points [63]. Therefore, we use the prototypes to guide the feature extractor to generate a stable and category discriminative feature space. Specifically, the representations that have the same label should get close to the corresponding category prototype and keep a reasonable distance far away from the category prototypes that have the different labels. Therefore, we propose to learn a dynamic margin among various clusters under the guidance of category prototypes and train a more robust and category-separable representation space.

Let $\mathbf{p}^{y_{i}}$ denote the prototype of the $y_{i}$-th category, $\mathbf{p}^{c}_{neg}$ is the prototype of the category $c$, where $c$ is different from $y_{i}$ and $c\in\mathcal{C}$. We propose the following dynamic margin contrastive (DMC) loss:

where $\alpha=-\sum_{c=1}^{C}\hat{y}^{c}_{i}\log(\hat{y}^{c}_{i})$, $\hat{y}^{c}_{i}$ is the prediction probability of $F(\mathbf{g}_{i})$ for the category $c$, $\operatorname{Dist}(\cdot,\cdot)$ denotes the Euclidean distance between two feature vectors. The dynamic margin $\alpha$ will not be a fixed value, but has relation to the output of the network. When the margin $\alpha$ gets a small value, it means that the category classifier $F$ outputs a confident prediction for $\mathbf{g}_{i}$, and the feature representation $\mathbf{g}_{i}$ is far away from the decision categorical surface. It seems that $\mathbf{g}_{i}$ is a tractable sample for optimizing the representation space, thus we can assign a small margin between $\mathbf{p}^{y_{i}}$ and $\mathbf{p}^{c}_{Neg}$. On the contrary, when $\mathbf{g}_{i}$ gets an uncertain prediction, we should assign a larger margin between $\mathbf{p}^{y_{i}}$ and $\mathbf{p}^{c}_{Neg}$. The comparison between the original triplet loss and the proposed DMC loss is shown in Fig. 4. The triplet loss pulls the positive pairs (i.e., samples that have the same category label) close to each other and pushes the negative pairs (i.e., samples that have the different category label) far away from each other, but the margin is a fixed value which is hard to determine. Our DMC loss encourages the feature representations scattering around the center of the corresponding category and seeks to keep a reasonable separation between different clusters.

The prototypes are computed from the source data which have ground truth labels. To generalize the discriminative information from the source domain to the target domain, we compute the DMC loss using both the source and target data. Since labels of the target data are unavailable during training, we assign hard pseudo-labels $\hat{Y}_{t}^{{}^{\prime}}$ to the unlabeled $X_{t}$, that is,

Although both our DMC and previous works [54, 55, 57] are based on the idea of contrastive learning, their models are different. Specifically, the works [54, 55, 57] search positive samples and negative samples for each sample and compute the distance for all positive pairs and negative pairs, and they assign a fixed margin between clusters. While our DMC only needs to compute the distance between a single sample and two category prototypes, and it introduces a dynamic margin to flexibly characterize the class separability of deep features. Overall, the proposed DMC loss encourages maintaining a reasonable separation across the centers of feature representations belonging to different categories in the feature space. Since the relationship of intra-category compactness and inter-category discrepancy are taken into account, the feature learned for the target data has the better discriminative ability.

The objective function of the BJDA method is formulated as

where $\lambda_{1}$ and $\lambda_{2}$ are non-negative hyper-parameters.

We use the mini-batch stochastic gradient descent (Mini-batch SGD) [64] to solve the objective function.

The detailed training procedure is shown in Algorithm 1.

We use six benchmark datasets, i.e., ImageCLEF-DA, Office-Caltech, Adaptiope, and Refurbished Office-31.

ImageCLEF-DA: The ImageCLEF-DA dataset [45] is a commonly used dataset for UDA. It consists of images from three distinct domains, which are subsets of Caltech-256 ($C$), ImageNet ILSVRC 2012 ($I$), and Pascal VOC 2012 ($P$) datasets. It selects 12 classes common to the three public datasets. There are 50 images in each category, and each domain is comprised of the same amount of samples.

Office-Caltech10: The Office-Caltech10 dataset [66] is a dataset widely adopted by UDA methods. The whole dataset contains 2533 images, which consists of 4 domains: Amazon (images downloaded from online websites), DSLR (high-resolution images from digital SLR cameras), Webcam (low-resolution images taken by web cameras), and Caltech-10 (images from Caltech-256 dataset). We use characters $A$, $D$, $W$, $C$ to represent these domains for short, respectively. There are 10 common categories of objects in each domain, including monitor, keyboard, headphones, calculator, etc.

Adaptiope: The Adaptiope dataset [67] is a new, large, diverse classification dataset for UDA. It has 3 domains, namely Product, Real life, and Synthetic. Images for the Product domain are real article images from the e-commerce website, and images in the Real life domain are customer review images. For the Synthetic domain, data is collected from artistic websites such as $sketchfab.com$. It is noteworthy that every category has an equal number of images, and each domain contains 12300 images from 123 classes. Fig. 5 presents example images from the Adaptiope dataset. It can be seen that there are large domain gaps among various domains in this challenging dataset, and different backgrounds and colorspaces exist among images from different domains.

Refurbished Office-31: The Refurbished Office-31 dataset[67] is an upgrade version of the commonly used Office31 dataset [68]. In light of there are some systematic errors (i.e., ambiguous ground truth, domain leakage) in the Amazon domain of the original Office31 dataset [68], the author of [67] propose the Refurbished Office-31 dataset by manually cleaning a total of 834 of the 2,817 images in the Amazon domain. It is a collection of images from three different domains, i.e., The Refurbished Amazon, DSLR, and Webcam. We use $A_{ref}$, $D$, $W$ to represent these domains for short, respectively.

Office-Home: It is one of the largest datasets for the UDA tasks in the community [69]. It has $15,500$ samples with 65 categories. It consists of four different domains, namely Art, Clipart, Product, Real-World. We use $A$, $C$, $P$, $R$ to represent these domains for short, respectively.

DomainNet: It is a recently proposed benchmark [70] with 0.59 million images, it consists of six different domains, namely Clipart, Infograph, Painting, Quickdraw, Real, and Sketch[70]. Each domain contains 345 categories of objects. We use $clp$, $inf$, $pnt$, $qdr$, $rel$, $skt$ to represent these domains for short, respectively.

We compare our BJDA method against a variety of state-of-the-art UDA methods, including CORAL [12], DANN [19], DAN [71], OT-GL [51], CDAN [47], SymNets [20], ATM [33], RSDA [48]. Results of these methods are quoted from the original papers, or re-implemented under the same UDA settings. In the test phase, test samples are forwarded through the feature extractor $G$ and the category classifier $F$. The setting of Source-only is used as a baseline, where all images from source domains are used to train a classifier without matching with the target data.

To make it a fair comparison, we follow the experimental settings of UDA ([20, 71, 33] ). We perform each task for five runs and report the average accuracy and standard deviation. We use the deep features commonly used in existing works for a fair comparison with other UDA methods. Specifically, the Decaf6 [72] features (activations of the 6th fully connected layer of the convolutional neural network, the dimensionality is 4096) are used for the Office-Caltech10 dataset, the ResNet101 [73]features (activations of the penultimate fully connected layer of the convolutional neural network, the dimensionality is 2048) are used for the DomainNet dataset. In light of ResNet50 [73] has been commonly used to extract features or as the backbone of deep learning models in the literature, we use the ResNet50 features (the dimensionality is 2048) for the other three datasets in our experiments. For the feature extractor $G$, we use two-layered fully-connected networks with 1024 neurons in the hidden layers, in which the LeakyReLU is used as the activation function. We set the number of output dimensions of the feature extractor as 512. For the category classifier $F$, we use a fully-connected network. The maximum iteration $T_{max}$ is set to 300 in all experiments. All the networks are optimized by a Minibatch SGD optimizer with a weight decay of 0.0005 and momentum of 0.9, and the initial learning rate is 0.001. The values of $\lambda_{1}$ and $\lambda_{2}$ for experiments in current manuscript are set as (0.5,0.3). We follow the protocol in the community [45, 71, 33] and tune hyper-parameters using transfer cross-validation [74]. All the experiments are implemented using the PyTorch library and are trained with an NVIDIA GeForce TITAN Xp GPU.

The classification results for the ImageCLEF-DA dataset are reported in Table I. The results show that our BJDA significantly outperforms all methods on 3 out of 6 UDA tasks and achieves the best average accuracy. There are several interesting observations as shown in Table I. Firstly, BJDA achieves better performance than DeepJDOT [24] which matches the joint distributions using OT and deep neural networks in the Euclidean space. This confirms that performing domain alignment via OT in RKHS can improve the performance of UDA tasks. Secondly, several methods (i.e., MADA [75], CDAN [47], SymNets [20], RSDA [48], ATM [33], TCM [76]) that explicitly or implicitly consider the label information significantly outperform those methods that perform domain alignment with only feature marginal distribution (i.e., DDC [14],DANN [19], DAN [71], OT-GL [51]), which indicates that label information contributes to modeling the multi-modal structure of image data. Thirdly, our BJDA substantially improves the classification accuracy of difficult UDA tasks (e.g., I$\rightarrow$P, C$\rightarrow$I, C$\rightarrow$P), where there exist large domain gaps, which confirms that aligning the joint distributions among various domains is robust to large domain gaps. Finally, our BJDA achieves the best average performance, which demonstrates its effectiveness in modeling a more transferable target classifier.

Table II summarizes the classification results for the Office-Caltech10 dataset. The BJDA model substantially outperforms the comparison methods on most UDA tasks. Specifically, BJDA obtains an absolute accuracy improvement of 1.4% over the latest competitor DMP [30]. Notably, BJDA and DAN [71] yield greater improvements for this benchmark, which highlights the advantage of performing distribution alignment by exploiting the nonlinear feature mapping of RKHS. For JDA [22] and DeepJDOT [24], both of which match the joint distribution cross the two domains, BJDA outperforms them by 5.9% and 2.8%, respectively. It validates the the superiority of BJDA which directly models the joint distribution shift in the infinite-dimensional spaces. UDA tasks with the domain A are more challenging as images from domain A exhibit more diversity within each category. In this case, aligning the joint distributions can effectively improve the performance of UDA tasks. As Table II shows, our BJDA method exhibits significant advantages over other UDA methods when domain $A$ acts as the source domain or the target domain. The results show that BJDA tends to more stable when there are large domain shift between the source and target domains.

To demonstrate the applicability of the proposed BJDA on more intricate real-world problems, we evaluate BJDA on the recently proposed Adaptiope benchmark. The results compared with several state-of-the-art methods are shown in Table III. Our method outperforms the other methods on 4 out of 6 UDA tasks and achieves the best average accuracy. Especially in the most challenging task (i.e, P$\rightarrow$ S, R$\rightarrow$S, and S$\rightarrow$R), for which Source-only accuracies are below 30%, our method significantly outperforms all the previous methods. It shows the superiority of our BJDA when there are severe domain gaps between the source and the target domains. Furthermore, both BJDA and DeepJDOT [24] show significant improvements over the marginal-based methods (i.e., CORAL [12], DDC [14], DANN [19], OT-GL [51]), which demonstrates that it is very necessary to align the joint distributions rather than the marginal distributions. Our BJDA achieves 2.8% and 3.7% average accuracy improvements with respect to the recent SymNets [20] and RSDA [48], respectively. These experimental results indicate that BJDA can extract more transferable features than state-of-the-art approaches.

We report the results of the proposed BJDA method on the recently proposed Refurbished Office-31 benchmark in Table IV. It can be seen that our BJDA method outperforms all comparison methods on 3 out of 6 UDA tasks and achieves the best average results. It is interesting to see that BJDA, SymNets [20], RSDA [48] achieve significantly better results over individual UDA tasks than the other comparison methods, the reason may be that these methods take into account the label information and lead to a more discriminative target feature space. By comparing the performance of our BJDA with the DeepJDOT [24] which also considers aligning the joint distribution based on OT, we find that BJDA exceeds DeepJDOT on all UDA tasks by large margins. The results verify that it is effectual to extend the OT theory in the Euclidean space to the reproducing kernel Hilbert space. It is worth noting that our BJDA achieves a significant improvement on the hard tasks between the domain A_ref and the domain D, which demonstrates the effectiveness of our proposed approach.

To evaluate the scalability of the proposed BJDA method, We compare BJDA with several recent UDA methods on the large-scale datasets Office-Home and DomainNet. During the training process, we assign pseudo-labels for the unlabeled target samples. Those pseudo-labels may be wrong, so these noises in pseudo-labels will degrade the performance of our BJDA method. To address this issue, we select the target samples which have prediction probability greater than a confidence threshold in the training stage. We add the experiments with this selection strategy on the Office-Home and DomainNet datasets. The results are shown in Table V and Table VI, and the corresponding setting is denoted as BJDA+PL. It can be seen that the selection strategy for pseudo-labels can improve the performance of the proposed BJDA method and alleviate the impact of noisy pseudo-labels. For the Office-Home dataset, our BJDA outperforms the recently proposed ATM method [33] by 0.2% and the average accuracy is close to the state-of-the-arts (i.e., SCDA  [77] and TCM [76]). The DomainNet dataset is a large and very challenging benchmark, in which large domain gaps exist. Our BJDA obtains an average accuracy of 28.2%, which is very close to the best result 30.3% produced by SCDA [77]. The reason may be that SCDA uses multiple complicated pair-wise adversarial processes, in which the classifier first finds the region with a large discrepancy among samples with the same category, and then the feature extractor tries to suppress the dissimilar regions. TCM combines the disentangled causal mechanisms with deep neural networks and it can effectively identify the latent semantic feature and improve the discriminative ability of representations. In addition, our BJDA achieves the best accuracy on 12 out of the 30 UDA tasks on the DomainNet dataset. From these results, we find that BJDA achieves comparable accuracy compared with recent UDA methods on those large-scale benchmarks. These results verify the scalability and the effectiveness of the proposed BJDA method.

To show the effectiveness of BJDA intuitively, we visualize the deep features before and after adaptation using t-SNE [79] in Fig. 6. The experiment of tasks I$\rightarrow$P and P$\rightarrow$I in ImageCLEF-DA are selected for illustration. Features extracted from DANN [19] and DeepJDOT [24] are also shown in Fig. 6 for comparison. For clarity, we use different markers to denote different categories, and the source features are shown in blue and target in red. Compared with the features learned by the Source-only in Fig. 6(a) and 6(e), we observe that all the features from BJDA in Fig. 6(d) and 6(h) have shown good adaptation patterns, which implies that BJDA is helpful to reduce the domain shift and improve the transferability of deep features. We can see that the features of DeepJDOT [24] in the third column are more separable than those of DANN [19] in the second column, which demonstrates that it is more valuable to align the joint distributions compared with the marginal-based methods. As shown in Fig. 6, there are larger margins between different clusters corresponding to different categories in Fig. 6(d) and 6(h) compared with graphs in other columns, and the target features in the fourth column behave more clear to categorize, which indicates that BJDA can learn more discriminative features. It is shown that the features of BJDA are better aligned and more distinguishable than the features of DeepJDOT [24], which indicates the superiority of performing joint distribution alignment in RKHS.

In order to exemplify the performance of BJDA and better investigate where the improvements are coming from, we perform an ablation study over various combinations for the UDA task on ImageCLEF-DA. The BJDA algorithm reduces the differences in the joint distributions and learns a category discriminative feature space simultaneously. It mainly consists of two components: the Bures joint distribution alignment loss and the dynamic margin contrastive loss. The BJDA($w/o$ $L_{dmc}$) implements the idea of Bures joint distribution alignment but excludes the DMC loss. The BJDA($w/o$ $L_{da}$) trains the model with the cross-entropy loss and the DMC loss. The BJDA implements the proposed BJDA algorithm as Eq. (10). DeepJDOT [24] incorporates OT and deep neural networks in the Euclidean space to matches the joint distributions, we select DeepJDOT as the baseline to demonstrate the superiority of our Bures joint distribution alignment loss. In light of the original triplet loss [58] is widely used to preserve the semantic relations, we train our network using the Bures joint distribution alignment loss, the cross-entropy loss, and the triplet loss, and is termed as BJDA($w/o$ $L_{dmc}$)+$L_{trip}$. We select BJDA($w/o$ $L_{dmc}$)+$L_{trip}$ as the baseline to show the effectiveness of our DMC loss. The recently proposed TripDA [60] uses the original triplet loss to model the semantic structure of the data and incorporates the Joint Maximum Mean Discrepancy (JMMD) [45] to reduce the domain shift. We compare BJDA($w/o$ $L_{dmc}$)+$L_{trip}$ with TripDA [60] to show the effectiveness of our Bures joint distribution alignment loss. Table VII shows the results of the ablation study.

Comparing BJDA($w/o$ $L_{dmc}$) with DeepJDOT [24], it is clear that on average the BJDA($w/o$ $L_{dmc}$) performs better, which demonstrates the superiority of our Bures joint distribution alignment loss that performs domain alignment via OT in RKHS. BJDA($w/o$ $L_{dmc}$)+$L_{trip}$ outperforms the TripDA [60] by a clear margin, which verifies the effectiveness of the proposed Bures joint distribution alignment loss compared with JMMD. Our BJDA achieves 2.1% improvement on the average accuracy against BJDA($w/o$ $L_{dmc}$)+$L_{trip}$. This result shows the superiority of the DMC loss compared with the original triplet loss [58] and indicates that our DMC loss can be of help to improve the category discriminative ability of feature space. Notice that BJDA($w/o$ $L_{dmc}$) obtains an improvement of 1.6% over BJDA($w/o$ $L_{da}$), which reveals that the Bures joint distribution alignment loss makes a greater contribution to the UDA task compared with the DMC loss. From the above results, we can confirm that each component in BJDA has a specific contribution. Combining the Bures joint distribution alignment loss and the DMC loss leads to further improvement, indicating their complementarity and superiority of the BJDA algorithm.

LMMD [80] is a recently proposed divergence for conditional MMD, which models the domain divergence by aligning the marginal distributions of subdomains. We compare our BJDA with conditional MMD by substituting the Eq. (5) loss with the LMMD loss. The setting is denoted as BJDA($w/o$ $L_{da}$)+$LMMD$. It can be seen that the Eq. (5) loss performs better than LMMD, which verifies the superiority of aligning joint distributions. MDD [33] is another domain divergence, which considers the marginal distributions by minimizing the inter-domain divergence and optimizes the conditional distributions by introducing class information. We empirically evaluate the proposed BJDA by substituting the Eq. (5) loss with the MDD loss. The setting is denoted as BJDA($w/o$ $L_{da}$)+$MDD$. It can be seen that the Eq. (5) loss achieves better results compared with the MDD loss. We also add the ablation experiment with the variant which uses the Wasserstein distance on the ImageCLEF-DA dataset. We compute the Wasserstein distance follow the formula of discrete Optimal Transport from previous work [25]. The corresponding setting is denoted as BJDA($w/o$ $L_{da}$)+$WD$. The results suggest that the BJDA loss gets better performance than the Wasserstein distance. These results further verify the superiority of the proposed Bures joint distribution alignment loss in Eq. (5).

To show the stability of training, we report the classification loss values and total loss values during the epochs in Fig. 7. To reveal the effectiveness of our contribution, we compare the loss values between BJDA, BJDA($w/o$ $L_{dmc}$), and BJDA($w/o$ $L_{da}$). We take the task I$\rightarrow$P in ImageCLEF-DA as an example. It can be seen that BJDA gradually converges after 300 epochs. Fig. 7(a) show that the classification loss values of BJDA (black line) are lower than that of BJDA($w/o$ $L_{dmc}$) (blue line) and BJDA($w/o$ $L_{da}$) (red line), which indicates that the features learned by BJDA are more discriminative. Fig. 7(b) presents the comparison of total loss values, it can be seen that the values of the total loss of BJDA are much smaller, which demonstrates the superiority of BJDA, and BJDA is better towards the convergence.

The overall objective function, i.e., Eq. (10), of the proposed BJDA includes two hyper-parameters $\lambda_{1}$ and $\lambda_{2}$ to balance the Bures joint distribution alignment loss and the dynamic margin contrastive loss. To show the stability of our BJDA methods and the impact of different hyper-parameters $\lambda_{1}$ and $\lambda_{2}$, we investigate the sensitivity of the hyper-parameters and implement grid search for $\lambda_{1}$ and $\lambda_{2}$ with respect to the tasks I$\rightarrow$P and P$\rightarrow$I in ImageCLEF-DA. $\lambda_{1}$ and $\lambda_{2}$ are searched in the range of [0.1, 0.3, 0.5, 0.7, 0.9], respectively. Results are shown in Fig. 8. The brighter the color, the higher the classification accuracy is. It can be seen that the classification accuracy varies slowly among the peak area. The results show that in most cases, the performance of BJDA is stable when the values of $\lambda_{1}$ and $\lambda_{2}$ increase from 0.1 to 0.9. This indicates the robustness of performing joint distribution alignment in the RKHS. The values of $(\lambda_{1},\lambda_{2})$ in this paper are set as (0.5,0.3). The fact that $\lambda_{1}>\lambda_{2}$ indicates that the Bures joint distribution alignment phase is more important than the metric learning phase during the adaptation process.