Unsupervised Domain Adaptation Using Compact Internal Representations

Despite the remarkable advancements in deep learning, deep neural networks still face challenges with generalization when distributional gaps occur during model execution. These gaps result from “domain shift” which occurs when there are significant differences between the distribution of data in the training domain (source domain) and the testing domain (target domain). To overcome this issue and ensure effective generalization in the target domain, it is necessary to retrain the neural network using data specifically from the target domain. However, this process can be expensive and time-consuming since it requires manual data annotation and preparing a large training dataset [24, 53]. Moreover, the process becomes infeasible when the distribution changes continually [50, 56, 59], making persistent data annotation necessary.

Unsupervised Domain Adaptation (UDA) is a learning framework aimed at addressing the challenges posed by domain shift when only unlabeled data is available in a target domain. In UDA, the primary strategy involves mapping the labeled data from the source domain and the unlabeled data from the target domain into a shared latent embedding space. This shared space serves as a bridge between the two domains, facilitating knowledge transfer and adaptation  [52]. The objective in UDA is to minimize the distributional discrepancy between the source and target domains in this shared embedding space. By reducing the distance between the distributions, the effects of domain shift can be mitigated, enabling improved generalization in the target domain [35, 18, 64, 77, 7, 28]. Domain alignment allows for the training of a classifier network in the source domain using data representations obtained from this shared space. Since the data representations in the embedding space capture the common underlying structure between the domains, the classifier network can effectively learn to discriminate between the classes using the shared knowledge. As a result, it demonstrates good generalization capabilities in the target domain, despite being trained exclusively on data from the source domain. By exploiting the information contained in the unlabeled target data through the shared space, UDA provides a powerful mechanism for adapting models to new domains without requiring costly manual annotation efforts.

Most domain adaptation methods have focused on modeling the shared embedding space as the output space of a deep encoder network and training the encoder to enforce domain alignment. Various approaches have been extensively explored to achieve domain alignment. Most methods aim to enforce domain alignment through adversarial learning or direct cross-domain distribution alignment [15, 14, 67, 19, 1, 44, 47, 51, 22, 43, 69, 78]. Adapting generative adversarial networks (GANs) to UDA involves aligning two distributions indirectly within an embedding space. The shared encoder is modeled as a cross-domain feature generator network. By employing a competing discriminator network, which is jointly trained using an adversarial min-max training procedure, the source domain and target domain features are made indistinguishable. The generator networks is trained to produce samples that can “fool” the discriminator into classifying them as target domain samples or the source domain samples. When the discriminator is unable the distinguish the correct domain for the samples, the embedding space becomes domain-agnostic. This alignment procedure effectively aligns the two distributions but adversarial learning requires careful engineering, including setting the initial optimization point, designing the auxiliary networks’ architecture, and selecting appropriate hyperparameters to maintain stability. Adversarial learning also has vulnerabilities, such as mode collapse, which can impact its effectiveness. Due to these challenges, direct probability matching has been explored as an alternative approach.

Direct probability matching focuses on minimizing the distance between two domain-specific distributions directly in the embedding space [46]. This approach requires less data engineering if an appropriate probability distribution metric is chosen. However, methods based on probability matching tend to have lower average performance compared to those based on adversarial learning. One reason for this discrepancy is the challenge of measuring distances in higher dimensions, known as the curse of dimensionality. It is difficult to establish meaningful semantic similarities in the input space based solely on small distances between distributions in an embedding space. Thus, the key challenge lies in selecting a suitable probability distribution metric that can effectively measure distances in high dimensions, using empirical samples, and can be optimized efficiently. Overall, both adversarial learning and direct probability matching offer different approaches to achieve domain alignment in UDA. Adversarial learning requires careful engineering but can be effective in aligning distributions, while direct probability matching requires selecting an appropriate metric that can capture meaningful distances in high-dimensional spaces.

One promising direction to enhance the general strategy of domain alignment in UDA is to incorporate additional mechanisms that improve the separability of target domain data in the embedding space. Recent advances in UDA primarily stem from the introduction of new secondary mechanisms that enhance performance. Various approaches have been explored to improve UDA performance by leveraging secondary mechanisms. For instance, Motiian et al. [40] enforce semantic class-consistent alignment of distributions by assuming access to a few labeled target domain data. Chen et al. [5] align the covariances of the source and target domains to reduce domain discrepancy. Li et al. [30] introduce entropy-based regularization into the loss function to leverage the intrinsic structure of the target domain classes. In this work, we would like to explore the possibility of using internal representations to develop a secondary mechanism to improve UDA. In supervised classification settings, when training a deep neural network, data representations (i.e., network responses) tend to form distinct class-specific clusters in the final layer of the network. An effective strategy to improve domain alignment is to create larger margins between these class-specific clusters in the embedding space, allowing them to be more separated and distinct. Kim et al. [23] propose an approach based on adversarial min-max optimization to increase model generalizability by inducing larger margins. This strategy aligns with recent theoretical findings that demonstrate the benefits of large margin separation in the source domain for improved model generalizability in UDA [10].

In order to enhance the generalizability of models in UDA, we introduce a novel secondary mechanism. Our approach leverages the internal data distribution that emerges in a shared embedding space, which is obtained through pretraining on the source domain. The goal is to increase the margins between different visual class clusters, thereby mitigating the impact of domain shift when adapting to a target domain. The internally learned distribution in the embedding space exhibits a multimodality behavior. To effectively capture this multimodal nature, we employ a Gaussian Mixture Model (GMM) to parametrize and estimate the internal distribution. By using the GMM, we aim to create larger separations between the class clusters. To increase the interclass margins, we construct a pseudo-dataset by sampling random instances with confident labels from the estimated GMM. The model is then regularized to push the target domain samples away from the class boundaries using the generated pseudo-dataset. This is achieved by minimizing the distance between the target domain distribution and the distribution of the pseudo-dataset which is more compact than the original source domain internal distribution. The objective is to encourage clear separation between classes in the target domain. Theoretical analysis supports our method by demonstrating that it minimizes an upper bound for the expected error in the target domain. This theoretical foundation provides insight into the effectiveness of our approach. To evaluate the performance of our algorithm, we conduct experiments on standard UDA benchmark datasets. Through these evaluations, we demonstrate that our method is competitive compared to existing approaches. The results showcase the efficacy of our approach in improving model generalizability and addressing the challenges posed by domain shift in UDA.

The main challenge in the direct probability matching approach lies in selecting an appropriate probability discrepancy measure that can effectively align the distributions between the source and target domains. The chosen metric should be computationally efficient and possess desirable properties for numerical optimization. Several probability metrics have been employed for probability matching in UDA. One commonly used metric is the Maximum Mean Discrepancy (MMD), which aligns the means of the two distributions [35, 37, 33, 79]. Building upon this baseline, Sun et al. [66] improve the alignment by considering distribution correlations and leveraging second-order statistics. Other approaches include utilizing the Central Moment Discrepancy [75, 6] and incorporating Adaptive Batch Normalization [32]. While aligning lower-order probability moments is straightforward, it tends to overlook discrepancies in higher-order moments. To address this limitation, the Wasserstein distance (WD) has been employed [9, 2, 74, 12]. WD incorporates information from higher-order statistics and has been shown by Damodaran et al. [2] to enhance UDA performance compared to MMD or correlation alignment approaches [35, 66]. Hence, WD is a suitable choice for domain alignment.

Unlike more classic probability discrepancy measure such as KL-divergence or JS-divergence, WD exhibits non-vanishing gradients even when the two distributions do not have overlapping supports. Note that at the initial stage of UDA, the two distribution might have minimal overlapping supports due to distributional mismatches. This property makes WD well-suited for solving deep learning optimization problems, which are typically employed in deep learning to optimize objective functions through gradient-based methods. By leveraging the properties of WD and its compatibility with gradient-based optimization, the direct probability matching approach enables effective distribution alignment in UDA. It provides a principled framework for minimizing the distributional differences between domains, facilitating knowledge transfer and enhancing the generalization capability of models in target domains. Although using WD leads to improved UDA performance, a downside of using WD is heavy computational load in the general case compared to simpler probability metrics. The high computational load is because WD is defined as a linear programming optimization and does not have a closed-form solution for dimensions more than one. To account for this constraint, we use the sliced Wasserstein distance (SWD) [27] which we have previously been used for addressing UDA in different settings [55, 54, 13, 62, 61, 61, 63, 70]. SWD is defined in terms of a closed-form solution of WD in 2D. It can also be computed fast from empirical samples. Compared to these works that use SWD for distributional alignment, we develop a secondary mechanism that improves model generalizability.

When training deep neural network classifiers using supervised learning, the goal is to achieve a clear separation between the input data representations in an embedding space, as depicted in Figure 1. This embedding space is typically captured by the responses of the network in a higher layer. Through this transformation, the original input distribution is converted into an internal distribution that consists of multiple modes, where each mode corresponds to a specific class. This transformation is crucial for facilitating class discrimination, as the final layer of a classifier network is often a softmax layer that assumes linear separability between classes. The properties of this internally learned distribution can be effectively utilized to enhance the performance of Unsupervised Domain Adaptation (UDA) methods. An effective approach in UDA focuses on aligning the internal distributions by matching the cluster-specific means for each class across both the source and target domains [44, 8]. This class-aware domain alignment procedure addresses the challenge of class mismatch, where the class distributions in the source and target domains may differ significantly. By leveraging the learned internal distribution, UDA methods can effectively bridge the gap between different domains and improve the generalization capabilities of the model in the target domain. This alignment process ensures that similar classes from different domains are closer to each other in the embedding space, allowing for better adaptation and transferability of the learned knowledge. By considering the distributional properties of the data, UDA methods can effectively mitigate the challenges posed by domain shift and improve the overall performance and robustness of deep neural network classifiers in real-world applications [43, 31, 25, 65].

Our goal is to add a regularization component to the shared encoder network in order to maintain the robustness of the internal distribution when the input distribution undergoes perturbations [76]. This is achieved by making the clusters corresponding to different classes more compact in the embedding space. By promoting compactness in the representation of data, we introduce larger margins between classes, which in turn enhances the stability of the model when faced with domain shift. Theoretical studies conducted in few-shot learning scenarios have shown that increasing the margin between classes can effectively reduce the misclassification rate [4]. Motivated by these theoretical insights, we propose a novel approach that involves estimating the internal distribution formed in the embedding space using a parametric Gaussian Mixture Model (GMM).The GMM allows us to capture the multimodal nature of the distribution and represent it as a weighted combination of Gaussian components. By leveraging this estimated internal distribution, we can induce larger margins between the class-specific distributional modes. This means that the representations of different classes will be more separated, enabling better discrimination and improving the model’s ability to generalize to unseen data. By encouraging margin enhancement through the use of the estimated internal distribution, our approach provides a principled way to address the challenges posed by domain shift.

Consider a learning scenario where we have a target domain $\mathcal{T}$ for which we have access to unlabeled data $D_{\mathcal{T}}=(\bm{X}_{\mathcal{T}})$, where $\bm{X}_{\mathcal{T}}=[\bm{x}1^{t},\ldots,\bm{x}M^{t}]\in\mathcal{X}\subset\mathbb{R}^{d\times M}$ represents the input data points. Our objective is to train a model that can generalize well on this target domain. However, since we lack annotations for the target domain, using supervised learning to solve this task becomes infeasible. To overcome this limitation, we also assume that we have access to a source domain $\mathcal{S}$ with a labeled dataset $D_{\mathcal{S}}=(\bm{X}_{\mathcal{S}},\bm{Y}_{\mathcal{S}})$, where $\bm{X}_{\mathcal{S}}=[\bm{x}_{1}^{s},\ldots,\bm{x}N^{s}]\in\mathcal{X}\subset\mathbb{R}^{d\times N}$ and $\bm{Y}_{\mathcal{S}}=[\bm{y}^{s}_{1},...,\bm{y}^{s}_{N}]\in\mathcal{Y}\subset\mathbb{R}^{k\times N}$ represent the input data points and their corresponding one-hot labels, respectively. We assume that doth domains share the same set of $k$ semantic classes. This assumption is critical in UDA because it makes the two problems related.

The source and target data points are independently and identically drawn from their respective domain-specific distributions, denoted as $\bm{x}_{i}^{s}\sim p_{S}(\bm{x})$ and $\bm{x}_{i}^{t}\sim p_{T}(\bm{x})$. However, despite similarities between the two domains, there exists a discrepancy in their distributions, i.e., $p_{T}(\bm{x})\neq p_{S}(\bm{x})$. In the absence of labeled data in the target domain, a naive approach would be to use supervised learning in the source domain for model training. To this end, we consider a family of parameterized functions $f_{\theta}:\mathbb{R}^{d}\rightarrow\mathcal{Y}$, such as deep neural networks with learnable weight parameters $\theta$. We then search for the optimal model $f_{\theta^{*}}(\cdot)$ in this family using empirical risk minimization (ERM) on the labeled source domain data:

where $\mathcal{L}(\cdot,\cdot)$ represents a suitable point-wise loss function, such as the cross-entropy loss, and the additive terms is the emprical risk. Ideally, the model trained using ERM would generalize well on the source domain, and due to the transfer of knowledge between the two domains, it would perform better than random guessing on the target domain. However, due to the presence of a domain gap, i.e., $p_{T}(\bm{x})\neq p_{S}(\bm{x})$, the performance of the source-trained model would still be degraded compared to its performance on the source domain.

The above simple solution neglects the potential value of the unlabeled data available in the target domain. In UDA, our objective is to capitalize on the knowledge embedded in the unlabeled target data and the similarities between the two domains in order to enhance the generalization capability of the model trained on the source domain. By benefiting from knowledge transfer, we aim to bridge the gap between the source and target domains by mapping their respective data points into a shared embedding space denoted as $\mathcal{Z}$. This mapping process is facilitated by employing a shared encoder, which is designed to minimize the distribution discrepancy between the domains within the embedding space. The ultimate goal is to minimize the disparity between the domains to the extent that it becomes negligible after the transformation. Different UDA algorithms adopt diverse strategies to accomplish this task, leading to a wide range of approaches in the field. The key objective remains finding the shared embedding space that facilitates seamless knowledge transfer between the domains and enables the model to generalize well in the target domain in the presence of domain shift.

To model the shared embedding described above, we can decompose the end-to-end deep neural network $f_{\theta}(\cdot)$ into two subnetworks: an encoder subnetwork $\phi_{\bm{v}}(\cdot):\mathcal{X}\rightarrow\mathcal{Z}\subset\mathbb{R}^{p}$ and a classifier subnetwork $h_{\bm{w}}(\cdot):\mathcal{Z}\rightarrow\mathcal{Y}$, where $\bm{v}$ and $\bm{w}$ represent the learnable parameters of the encoder and classifier, respectively. This decomposition can be expressed as $f_{\theta}=h_{\bm{w}}\circ\phi_{\bm{v}}$, where $\theta=(\bm{w},\bm{v})$ represents the combined parameters. In line with the condition for good model generalization on the source domain, we assume that the classes are separable in the shared embedding space $\mathcal{Z}$ as a result of supervised pretraining on the source domain (refer to Figure 1, left). Note that this assumption is natural because if we cannot train a good model for the source domain, then by extension, we cannot adapt it to work in a target domain with unannotated data. Taking advantage of the target domain data, most UDA frameworks adapt the encoder trained on the source domain in order to match the distributions of both domains in the embedding space $\mathcal{Z}$. In other words, we aim to ensure that $\phi(p_{\mathcal{S}}(\cdot))\approx\phi(p_{\mathcal{T}}(\cdot))$. By achieving this prperty, the classifier subnetwork can generalize effectively in the target domain, despite being trained solely on the source domain data. A major class of UDA methods focus on matching the distributions $\phi(p_{\mathcal{S}}(\bm{x}^{s}))$ and $\phi(p_{\mathcal{T}}(\bm{x}^{t}))$ by training the encoder $\phi(\cdot)$ to minimize the distance between these two distributions in the embedding space, using a suitable probability distribution metric. The objective is to find an appropriate alignment between the source and target domain distributions in the shared embedding space by minimizing the probability metric in addition to minimizing the ERM loss function on the source domain:

where, $D(\cdot,\cdot)$ represents a probability discrepancy metric, and $\lambda$ serves as a trade-off parameter balancing the empirical risk and the domain alignment terms. Multiple probability discrepancy metrics exist, and various UDA methods have been developed based on different choices for $D(\cdot,\cdot)$. For this particular study, the selected metric for $D(\cdot,\cdot)$ is SWD. For details on definition and computation of SWD, please refer to the Appendix. In short, we use SWD because it can be computed efficiently from empirical samples of two distributions.

To increase the margins between classes, we leverage the internally learned distribution. Based on the aforementioned rationale, this distribution takes the form of a multi-modal distribution denoted as $p_{J}(\cdot)=\phi_{\bm{v}}(p_{\mathcal{S}}(\cdot))$, where $\phi_{\bm{v}}$ represents a mapping function applied to the source domain distribution $p_{\mathcal{S}}(\cdot)$ (refer to Figure 1, left). It is important to note that the formation of a multi-modal distribution is not guaranteed, but when we model the embedding space using network responses just prior to the softmax layer, it becomes a prerequisite for our model to learn the source domain. This assumption is not a limitation for our method because it is a pre-condition in UDA.

The means of the distributional modes (clusters) correspond to the concept of “class prototypes” which has been explored in previous works for achieving class-consistent domain alignment [44, 8]. The margins between classes can be observed in Figure 1 (left-top) as the geometric distances between the boundaries of the modes in the internally learned distribution $\phi_{\bm{v}}(p_{\mathcal{S}}(\cdot))$. Conversely, domain shift occurs when the internal distribution for the target domain $\phi_{\bm{v}}(p_{\mathcal{T}}(\cdot))$ deviates from the source-learned internal distribution $\phi_{\bm{v}}(p_{\mathcal{S}}(\cdot))$, resulting in increased overlap between class clusters in the target domain (Figure 1, left-bottom). In other words, the effect of domain shift in the input space manifests as a crossing of boundaries between certain class clusters.

Our goal is to develop a mechanism that encourages target domain data samples to move away from the interclass margins and towards the class means. This process is visualized in Figure 1 (right-top). By achieving this property, we aim to enhance the robustness of our model against domain shift in the input space, as shown in Figure 1 (right-bottom). The key idea is to make the class clusters in the embedding space more compact for the source domain, which in turn allows for greater variability in the target domain. To implement this idea, we begin by estimating the internally learned distribution in the latent space $\mathcal{Z}$. We use a parametric Gaussian Mixture Model (GMM) distribution to approximate this distribution. The GMM distribution captures the multi-modal nature of the internal distribution, representing different modes that correspond to class prototypes and interclass margins. The GMM distribution is defined as follows:

In our estimation of the internally learned distribution using a Gaussian Mixture Model (GMM), we assign the mean vectors $\bm{\mu}_{j}$ and covariance matrices $\bm{\Sigma}_{j}$ to each mode, while $\alpha_{j}$ represents the mixture weights. This probability distribution is an appropriate model for capturing the internal probability distribution because we have prior knowledge of the number of modes, denoted as $k$. As a result, estimating the GMM parameters becomes simpler compared to the general GMM estimation problem, which typically requires iterative and time-consuming procedures such as expectation maximization (EM) [39]. Since we have labeled source data points and know the value of $k$, we can specifically estimate $\bm{\mu}_{j}$ and $\bm{\Sigma}_{j}$ as the parameters of $k$ independent Gaussian distributions. The mixture weights $\alpha_{j}$ can be easily computed using a Maximum a Posteriori (MAP) estimate. Let $\bm{S}_{j}$ denote the set of training data points belonging to class $j$ in the source domain, i.e., $\bm{S}_{j}={(\bm{x}_{i}^{s},\bm{y}i^{s})\in\mathcal{D}{\mathcal{S}}|\arg\max\bm{y}_{i}^{s}=j}$. The MAP estimation for the GMM parameters can be computed as follows:

Compared to the computationally complex procedure for EM [57], the MAP estimation in Eq.(4) does not introduce a significant computational burden. The computational complexity of Eq.(4) is manageable, especially when considering a balanced source domain dataset. In the case of a balanced source domain dataset, determining whether data points $\bm{x}_{i}^{s}$ belong to $\bm{S}_{j}$ to compute $\alpha_{j}$ can be achieved with a computational complexity of $O(N)$. Here, $N$ represents the number of data points. Computing the mean vectors $\bm{\mu}_{j}$ has a computational complexity of $O(NF/k)$, where $F$ denotes the dimension of the embedding space. The computation of covariance matrices $\bm{\Sigma}_{j}$ has a complexity of $O(F(\frac{N}{k})^{2})$. Given that there are $k$ classes, the total computational complexity for estimating the GMM distribution is approximately $O(\frac{FN^{2}}{k})$. If we assume that $O(F)\approx O(k)$, which is a reasonable assumption when modeling the embedding space using network responses in the final layer, the total computational complexity becomes $O(N^{2})$. This computational overload is relatively small when compared to the computational complexity of one epoch of back-propagation, which needs to be performed multiple times during training. Since deep neural networks typically have a significantly larger number of weights compared to the number of data points $N$, we conclude that GMM estimation is not a demanding procedure.

We utilize the estimated GMM distribution to introduce larger interclass margins and improve the solution obtained through Eq.(2). Our aim is to create repulsive biases from the class margins, as depicted in Figure1 (top-right). To accomplish this, we first generate a pseudo-dataset in the embedding space with confident labels. This pseudo-dataset is denoted as $\mathcal{D}{\mathcal{P}}=(\textbf{Z}{\mathcal{P}},\textbf{Y}{\mathcal{P}})$ and is created by randomly sampling from the estimated GMM distribution. Here, $\bm{Z}{\mathcal{P}}=[\bm{z}1^{p},\ldots,\bm{z}{N_{p}}^{p}]\in\mathbb{R}^{p\times N_{p}}$ represents the set of generated embedding vectors, $\bm{Y}{\mathcal{P}}=[\bm{y}^{p}_{1},...,\bm{y}^{p}{N_{p}}]\in\mathbb{R}^{k\times N_{p}}$ which contains the corresponding pseudo-labels, and $\bm{z}_{i}$ is randomly drawn from $\bm{z}_{i}^{p}\sim\hat{p}_{J}(\bm{z})$. To ensure that these generated samples are located away from the margins and close to the respective class means in the embedding space, we pass all the initially drawn samples through the classifier subnetwork. We only include the samples for which the classifier exhibits a confidence level higher than a predefined threshold, denoted as $0<\tau<1$. This procedure helps us avoid including samples that are in proximity to the class margins. The selected samples will be part of the final pseudo-dataset $\mathcal{D}_{\mathcal{P}}$. By applying these steps, we ensure that the selected samples in $\mathcal{D}_{\mathcal{P}}$ are positioned away from the margins and closer to their corresponding class means in the embedding space, thus inducing repulsion from the interclass margins.

The process of selecting samples based on threshold values close to one, i.e., $\tau\approx 1$, indicates that the samples in the pseudo-dataset are located in the vicinity of the class means (as illustrated in Figure 1 (right-top)). Consequently, the margins between the empirical data clusters in the generated pseudo-dataset are larger compared to the empirical clusters of the source domain data in the embedding space. In other words, these samples represent a more compact representation in the embeddign space. We leverage this characteristic of the pseudo-dataset to update Eq. (2) and induce larger margins:

where, $\hat{D}(\cdot,\cdot)$ represents the empirical SWD metric. In Equation (6), we have four terms that contribute to the optimization objective. The first and second terms correspond to the ERM objective functions for the source dataset and the pseudo-dataset, respectively. These terms aim to maintain discriminative features in the embedding space for both the source and pseudo-domains, ensuring that the learned representations capture the characteristics of their respective domains. The third term serves as an alignment term, matching the source domain distribution with the empirical distribution of the pseudo-dataset. This alignment helps to align the source domain with the pseudo-domain, enabling the model to generalize well across both domains. The fourth term also acts as an alignment term, but it focuses on matching the target domain distribution with the empirical distribution of the pseudo-dataset. Since the pseudo-dataset possesses larger interclass margins, aligning the target domain with this distribution helps increase the margins in the target domain as well, enhancing the model’s generalizability in the presence of domain shift. These terms collectively contribute to increasing the interclass margins, as we discussed earlier, leading to improved model generalizability and adaptation to the target domain.

Our proposed algorithm, named Increased Margins for Unsupervised Domain Adaptation (IMUDA), is visualized and presented in Figure 1 and Algorithm 1, respectively. Figure 1 provides a visual representation of the concepts and steps involved in IMUDA, illustrating the impact of increased interclass margins on the feature representations visually. The algorithm outlines the steps involved in leveraging the estimated GMM distribution, constructing the pseudo-dataset, and incorporating the alignment terms into the optimization objective to enhance the adaptation process.

We analyse IMUDA in a standard PAC-learning setting [60]. In short, our goal is to prove that IMUDA algorithm minimizes an upperbound for the expected error on the target domain. We define the hypothesis space $\mathcal{H}$ as $\mathcal{H}=\{h_{\bm{w}}(\cdot)|h_{\bm{w}}(\cdot):\mathcal{Z}\rightarrow\mathbb{R}^{k},\bm{v}\in\mathbb{R}^{V}\}$ be the hypothesis space. This space includes all the classifiers that can be represented by the classifer network. Also, let $\hat{\mu}_{\mathcal{S}}=\frac{1}{N}\sum_{n=1}^{N}\delta(\phi_{\bm{v}}(\bm{x}_{n}^{s}))$ and $\hat{\mu}_{\mathcal{T}}=\frac{1}{M}\sum_{m=1}^{M}\delta(\phi_{\bm{v}}(\bm{x}_{m}^{t}))$ be the empirical source and the empirical target distributions in $\mathcal{Z}$. Similarly, let $\hat{\mu}_{\mathcal{P}}=\frac{1}{N_{p}}\sum_{q=1}^{N_{p}}\delta(\bm{z}_{n}^{q})$ be the empirical internal distribution. These distributions estimate the true distribution which are unkown. We denote the expected error for $h(\cdot)_{\bm{w}}\in\mathbb{H}$ on the source and the target domains by $e_{\mathcal{S}}({\bm{w}})$ and $e_{\mathcal{T}}({\bm{w}})$. Also, let $h_{\bm{w}^{*}}$ be the optimal joint-trained model, i.e., $e_{\mathcal{C}}(\bm{w}^{*})$, i.e. $\bm{w}^{*}=\arg\min_{\bm{w}}e_{\mathcal{C}}(\bm{w})=\arg\min_{\bm{w}}\{e_{\mathcal{S}}({\bm{w}})+e_{\mathcal{T}}({\bm{w}})\}$. Finally, note that we have $\tau=\mathbb{E}_{\bm{z}\sim\hat{p}_{J}(\bm{z})}(\mathcal{L}(h(\bm{z}),h_{\hat{\bm{w}}_{0}}(\bm{z}))$ because only samples with confident predicted labels are included in the generated pseudo-dataset. We offer the following theorem to justify why our algorithm is effective.

Theorem 1: Consider a UDA problem involving a source and a target domain and use algorithm 1 for model adaptation. Then, the following inequality holds:

where $W(\cdot,\cdot)$ denotes the WD distance and $\xi$ is a constant which depends on the loss function $\mathcal{L}(\cdot)$ characteristics, $\alpha$ is a constant and $\beta=(2(d+1))^{-1}$.

Proof: We base our proof on the applicability of using optimal transport for UDA [46]. We first state the following theorem developed for UDA when we use the optimal transport for domain matching.

Theorem 2 [46]: Under the assumptions described for UDA, then for any $d^{\prime}>d$ and $\zeta<\sqrt{2}$, there exists a constant number $N_{0}$ depending on $d^{\prime}$ such that for any $\xi>0$ and $\min(N,M)\geq N_{0}\max(\xi^{-(d^{\prime}+2)},1)$ with probability at least $1-\xi$ for all $f_{\theta}$, the following holds:

where $\zeta\in\mathbb{R}$ is constant.

We use Theorem 2 but since we benefit from SWD rather than optimal transport and rely on pseudo-labeling, it is not directly applicable to our algorithm. To extend Theorem 2 to our algorithm, we rely on the following relationship between the optimal transport and SWD [3]:

Building the above two results, we prove Theorem 1.

Considering a minimum confidence threshold of $\tau$ for the pseudo-labeled data points, we can conclude that the probability of the pseudo-label being incorrect is at most $1-\tau$. To quantify the disparity between the error based on the true labels and the pseudo-labels assigned to a specific data point, we can express it as follows:

We use Jensen’s inequality and apply the expectation operator on both sides of Eq. (10):

Now we use Eq. (11) and deduce the following:

Note that Eq. (12) holds for any model with a given parameter $\theta$. So, it holds for the joint optimal parameter $\theta^{*}$ which concludes:

where $e_{C^{\prime}}(\theta)$ is the expected error of the model trained on the combination of the target and pseudo-labeled dataset. Now in Theorem 2, let the pseudo-labeled data points be used as the target domain dataset, applying the triangular inequality on WD term first, then applying Eq. (13) and Eq. (9) on Eq.(8), Theorem 1 follows.

We utilize Theorem 1 to justify how the IMUDA algorithm can enhance model generalization on the target domain. By comparing Equation(7) and Equation (6), we observe that the IMUDA algorithm minimizes an upper bound of the expected error on the target domain. The first three terms in Equation (7) are directly included in the objective function of Equation (6). The first term, which corresponds to the target expected error $e_{\mathcal{T}}$, is minimized through the alignment of the source and target domain distributions in the embedding space. The second term represents the source expected error $e_{\mathcal{S}}$, which is reduced due to the supervised training on the source domain samples. The third term, which captures the discrepancy between the source and target domains, is minimized through the alignment terms in the optimization problem of Equation (6).

The fourth to sixth terms in Equation (7) are constant terms that provide conditions under which our algorithm can work effectively. The term $(1-\tau)$ becomes small when we set the threshold $\tau$ close to 1, implying that the pseudo-dataset is generated with confident labels, as we do in IMUDA. The term $e_{C^{\prime}}(\bm{w}^{*})$ will be small if the two domains are related, such as sharing the same classes, and a model trained jointly on both domains can achieve good performance. This term indicates that the alignment of distributions in the embedding space must be a feasible task for our algorithm to be effective. The last term in Equation (7) is a constant term that, similar to most learning algorithms, becomes negligible when we have large source and target datasets. In conclusion, if the domains are related and the conditions mentioned above hold, the IMUDA algorithm minimizes an upper bound of the expected error on the target domain.

We implement our algorithm and demonstrate that it is effective using standard UDA benchmark datasets. Our implementation is available. We offer comparison results to demonstrate that our method is competitive as well as analytic and ablative experiments to provide a deeper insight about our method.

We introduced a new unsupervised domain adaptation method to mitigate the influence of domain shift on model generalization. Our idea is based on expanding the distances between clusters of different classes within an embedding space. We represent this embedding space as the output-space of a deep neural encoder’s final layer. By increasing the interclass distances, we enhance the model’s ability to generalize in the target domain. We estimate the internal distribution of the source domain data and using the obtained knowledge, push the data representations away from the class boundaries within the embedding space to increase robustness with respect to domain shift. To estimate the internal distribution, we employ a parametric Gaussian mixture model (GMM). Subsequently, we generate confident labeled random samples from the GMM and use them to create larger interclass distances. Our theoretical analysis provides a justification to explain why algorithm can alleviate the impact of domain shift. Our empirical results demonstrate the effectiveness of our approach, showing competitive performance when compared to existing unsupervised domain adaptation methods. Furthermore, we conclude that integrating additional mechanisms is crucial for improving domain alignment in unsupervised domain adaptation, especially given the current performance levels of such algorithms.

Despite strengths, our algorithm has its own limitations. The first major limitation is that we consider that a multimodal distribution is formed in the embedding space, where we know that each class is represented by a single mode. While we observed the validity of this assumption in the datasets and models that we used in our experiments, there is no guarantee that this assumption will be valid in all models and datasets. If this assumption is not valid, then using our approach will not lead to having a good estimate for the internal distribution. A reasonable follow-up work is to address this weakness by proposing a stronger method to estimate the internal representation that relaxes our assumption. A second limitation of our work is that, as we observed in the experiments, our method leads to a better performance when the datasets are balanced. Another direction for future research is addressing the scenario where the source dataset exhibits class imbalance. The third limitation is that the upperbound that we offer in our theoretical is not guaranteed to be tight. A suitable theoretical future work is to offer an estimate about the tightness of the upperbound given the source and the target domain data. Such an estimate can help to check how effective our approach can be given a source and a target domain before attempting using the full pipleine of our method. The fourth limitation of work is that we consider that the source domain samples are readily accessible during UDA but in some applications, the source sample are either unavailable or cannot be shared due to privacy concerns. In those situations, our method is not applicable and a new solution should be proposed. Another limitation that we have is that the pre-adaptation and the post-adaptation steps should be performed sequentially. An improvement for future exploration is to merge these two stages. Finally, we are considering a UDA setting where both domains share exactly the same classes. In practical settings, two domains may share only a subset of classes. Our approach is not directly applicable in those scenarios, i.e., partial domain adaptation, and extending to address partial domain adaptation is another future research possibility for our work.