two stages domain invariant representation learners solve the large co-variate shift in
unsupervised domain adaptation with two dimensional data domains

Sets of data are comprised of $\mathcal{D}_{S}=\{(x_{i}^{S},y_{i}^{S})\}_{i=1}^{N_{S}}$, $\mathcal{D}_{T}=\{x_{i}^{T}\}_{i=1}^{N_{T}}$, $\mathcal{D}_{T^{\prime}}=\{x_{i}^{T^{\prime}}\}_{i=1}^{N_{T^{\prime}}}$ (Source domain, intermediate domain, target domain respectively and we denote $N_{S},N_{T},N_{T^{\prime}}$ as the sample sizes for source, intermediate and target.), in our setting $\mathcal{D}_{T}$ is e.g. from (UserA, Summer) when $\mathcal{D}_{S}$ is from (UserA, Winter) and $\mathcal{D}_{T^{\prime}}$ is from (UserB, Summer). Then let $P_{S}(y|x),P_{S}(x)$ denote the marginal and conditional distribution of source, defined for intermediate and target similarly. They are in the homogeneous domain adaptation assumption, namely they share same sample space but different distribution (Wilson & Cook, 2020). Also this research is in co-variate shift problem, that is generally sharing conditional distribution but different marginal distribution of co-variate (Zhao et al., 2019). The objective is learning $\phi(\cdot)=F\circ C$ to predict ground truth labels for $\mathcal{D}_{T^{\prime}}$ using three sets of data without access to target ground truth labels, $F$ corresponds to feature extractor with arbitrary neural networks parameter $\theta_{f}$ and $C$ task classifier with $\theta_{c}$ to be optimized by gradient descent.

Let’s dive into what is intermediate domain $\mathcal{D}_{T}$, we assume data domain is not one dimensional but is two dimensional e.g. (UserA or UserB, Accelerometer Model A and Accelerometer Model B) in HAR, (UserA or UserB, Winter or Summer) in occupancy detection problem and (Monotone or Color, In the street or not) in image recognition. In this paradigm, $\mathcal{D}_{S},\mathcal{D}_{T},\mathcal{D}_{T^{\prime}}$ may be (UserA, Accelerometer Model A), (UserB, Accelerometer Model A), (UserB, Accelerometer Model B) for instance. These two dimensional factors are influencing data distribution (we call this as two dimensional co-variate shifts), energy consumption data differs between users and also seasons for instance. Basically under this two dimensional co-variate shifts problem correcting difference between domains in UDA is quite difficult but more natural case closer to businesses or real world projects. Additionally we assume gathering unsupervised data $\mathcal{D}_{T}$ is quite easy, so UDA problem with three sets $\mathcal{D}_{S},\mathcal{D}_{T},\mathcal{D}_{T^{\prime}}$ is natural and there is demand for solving this problem. Two dimensional domains assumption is novel in this field, although uses of an intermediate domain to resolve large co-variate shift are explored in (Lin et al., 2021; Zhang et al., 2019; Oshima et al., 2024) as well. Their experiments showed a synthetic intermediate domain can improve UDA (Lin et al., 2021; Zhang et al., 2019), but limited to computer vision tasks technically or experimentally. Our proposition is not limited to a specific data type. We investigated whether or not each dataset follows this assumption in the Appendix E.

To begin with previous domain invariant learning abstraction, their objective functions and optimization are below (we call this as Normal). Domain invariant learning is parallel learning including the feature extractor and task classifier’s learning labelling rule based on $\mathcal{D}_{S}$ by minimizing $L_{task}$ and the feature extractor’s learning domain invariance between $\mathcal{D}_{S},\mathcal{D}_{T^{\prime}}$ by $L_{domain}$ (measurement of distribution gap between source and target, we elaborate later). The constant value $\lambda$ is coefficient of weight to adjust the balance between task classification performance and domain invariant performance. We define $L_{task}=-\sum_{i=1}^{batch\_size}\sum_{j=1}^{num\_class}y_{i,j}^{S}\log(\hat{y}_{i,j}^{S})$ as the cross entropy loss with a predicted probability $\hat{y}_{i,j}^{S}$ for source input.

This will achieve the feature extractor and task classifier’s generalization performance for target data theoretically, since the UDA goal(expectation risk for target discrimination performance) is bounded by marginal distribution difference between source and target (corresponds to $L_{domain}$), empirical risk for source discrimination performance (corresponds to $L_{task}$), conditional distribution gap (under co-variate shift problem this should be small), and so fourth. Let $\mathcal{H}$ be a hypothesis space of VC-dimension $d$, $\hat{\mathcal{D}}_{S},\hat{\mathcal{D}}_{T^{\prime}}$ be empirical sample sets with size $N$ drawn from source joint distribution (features and label) and target marginal distribution (features), $d_{i}$ be the domain label (binary label identifying source or target, $d_{i}\in\{0,1\}$), $\mathbb{E}_{T^{\prime}}(h),\mathbb{E}_{S}(h)$ be expectations for hypothesis $h$ (task classification), $\hat{\mathbb{E}}_{S}(h)$ be the empirical expectation, $\mathbb{I}[\cdot]$ be the function outputting 1 when true inside the square brackets otherwise outputting 0, and then w.p. at least $1-\delta$ ($\delta\in(0,1)$), $\forall h\in\mathcal{H}$, the following inequality holds (Zhao et al., 2019; Ben-David et al., 2006).

Previous studies such as DANNs, CoRALs and DANs have been published as embodiments of equation 1 and 2. The construction of $L_{domain}$ differs across methods. DANNs define it as a loss of domain classification problem, while CoRALs use the distance between covariance matrices of $C$. DANs build it using the multiple kernel variant of maximum mean discrepancy (MK-MMD (Gretton et al., 2012)) calculated on $F$ and $C$, and Deep Joint Distribution Optimal Transportation (DeepJDOT) define it by wasserstein distance based on the optimal transport problem (Damodaran et al., 2018). There are many variants, e.g. Convolutional deep Domain Adaptation model for Time Series data (CoDATS) is signal processing layers and multi sources version of DANNs (Wilson et al., 2020) and CoRALs variants are using different distant metrics for correlation alignment including a euclidean distance (Zhang et al., 2018a), geodesic distance (Zhang et al., 2018b), and log euclidean distance (Wang et al., 2017). For DANNs, optimizing $L_{domain}$ in a adversarial way with the domain discriminator $D$ with arbitrary neural networks’ parameters $\theta_{d}$, we omitted description of $\operatorname*{arg\,max}_{\theta_{d}}L_{domain}$. Please note that $\operatorname*{arg\,min}_{\theta_{c}}L_{domain}$ is only worked for e.g. CoRALs and DANs.

Previous paper (Oshima et al., 2024) is intuitively step-by-step version of this paper (center of Figure 1), this executes DANNs learning between source and intermediate data then second time DANNs between intermediate data (pseudo labeling by that learned task classifier) and target. We call this as Step-by-step. Very limited evaluation was conducted and achieved higher classification performance compared to normal DANNs and without adaptation model using occupancy detection data(Dataset D, explain later). There are five main differences between the method of (Oshima et al., 2024) and this paper, (1) (Oshima et al., 2024) generates noise due to erroneous answers in the pseudo-labelling step ($\{(x_{i}^{T},\hat{y}_{i}^{T})\}_{i=1}^{N_{T}}$ in Figure 1), which may hinder learning convergence, whereas our end-to-end method does not, (2) The two domain invariant representation learnings have partially independent structure, and there is no guarantee that the first learning will necessarily be good for the second learning, but our method can perform the two learnings in end-to-end, which may make it easier to keep the overall balance, (3) They introduced confidence threshold technique which has large impact on learning convergence, but these engineering tricks are not easy to tune in unsupervised settings in practical use cases, (4) Comparative experiments on four sets of data show that our method has a performance advantage on most of the settings (5/8 settings) and (5) The learning algorithm encompasses the entire domain invariant representation learning techniques including CoRALs, DANNs and DANs.

Free parameters(e.g. learning rate for gradient descent optimizer, layer-wise configurations) selection has been regarded as a crucial role because deep learning methods generally are susceptible to, additionally UDA require us to do this in a agnostic way for target ground truth labels at all. We propose free parameter tuning method specialized in this two stages domain invariant learners by extending RV (same as Normal (Ganin et al., 2017)). RV applies pseudo-labeling to the target data and uses the pseudo-labeled data and the source unsupervised data in an inverse relationship (Zhong et al., 2010), authors proved this can measure conditional distribution difference between target data and learned model. We apply the RV idea to two stages domain invariant representation learners, replacing the internal learning steps from regular supervised machine learning with two stages domain invariant representation learners and calculating classification loss between predictions for source data by reverse model and its ground truth labels. We can find better configurations by $\operatorname*{arg\,min}_{\theta}|\phi_{r}(x^{S})-y^{S}|$. Algorithm is denoted in Algorithm 2, and the Theorem 3.1 states that this method can measure the gap in labeling rules between the trained model and the final target data, the proof of the theorem is given in Appendix B.

The evaluation follows the hold-out method. The evaluation score is the percentage of correct labels predicted by the task classifier of the two stages domain invariant learners for the target data i.e. $\mathrm{accuracy}(x^{T^{\prime}})=\frac{1}{n}\sum_{x^{T^{\prime}}}{\color[rgb]{1,.5,0}\mathbb{I}[C(F(x^{T^{\prime}}))=y^{T^{\prime}}]}$ ($n$ as the sample size of target data for testing). The target data is divided into training data and test data in $50\%$,the training data is input to the training as unsupervised data, and the test data is not input to the training but used only when calculating the evaluation scores. In order to take into account the variations in the evaluation scores caused by the initial values of each layer of deep learning, 10 evaluations are carried out for each evaluation pattern and the average value is used as the final evaluation score. In addition, hyperparameters optimisation with 3.2 is performed on the learning rate using the training data. The all codes we used in this paper are available on GitHub ¹¹1please check v1.0.0(release soon) compatible to this paper [Put URL later], also we elaborated experimental configurations in the Appendix D.

We validate our method with 4 datasets and 38 tasks including simulated toy data, image digits recognition, HAR, occupancy detection. We adopt six benchmark models (conventional Train on Target model, Ste-by-step with DANNs or CoRALs, Normal with DANNs or CoRALs, conventional Without Adapt model) as a comparison test to ours in Table 14 (Appendix I). If our hypothesis is true, our method should be close to its Upper bound and better than previous studies and Lower bound models therefore ideal state ”Upper bound $>$ Ours $>$ Max(Step-by-step, Normal, Lower bound)” should be expected.

A. sklearn.datasets.make_moons

Source data is two interleaving half circles with binary class. Intermediate data is rotated $a$ degrees (semi-clockwise from the centre), target data is rotated $2a$ degrees ($a\in\{15,20,25,30,35\}$). These rotations correspond to toy versions of two dimensional co-variate shifts. We set noise=0.1.

B. MNIST, MNIST-M, SVHN(Lecun et al., 1998; Ganin et al., 2017; Netzer et al., 2011)

Source data is modified national institute of standards and technology database (MNIST), intermediate data is MNIST-M which is the randomly colored version of MNIST, target data is the street view house numbers (SVHN). Case with source and target reversed was demonstrated to be coped with by Normal (Ganin et al., 2017). Two dimensional co-variate shifts should be (Monotone, Not on the Street)$\rightarrow$(Color, Not on the Street)$\rightarrow$(Color, On the Street).

C. HHAR(Stisen et al., 2015)

Heterogeneity human activity recognition dataset (HHAR). Signal processing problem to determine human behaviour ($\{bike,sit,stand,walk,stairsup,stairsdown\}$) by accelerometer data (sliding window with size 128, sampling rate is 100-150Hz). Subjects were taking actions in the pre-programmed way, so there are no chances of distribution shift by different or strange movements. The two dimensional co-variate shifts should be (UserA, SensorA)$\rightarrow$(UserB, SeonsorA)$\rightarrow$(UserB, SensorB), same actions but different user and sensor differ in co-variate. We extract 16 patterns randomly from 9 users and 4 models.

D. ECO data set(Beckel et al., 2014)

Electricity consumption & occupancy (ECO) data set. Signal processing problem same as HHAR with different time window and activity classes(only $\{occupied,unoccupied\}$). The two dimensional co-variate shifts should be (HouseA, Winter)$\rightarrow$(HouseB, Winter)$\rightarrow$(HouseB, Summer). Subjects were not pre-programmed in any means so should include distribution shift in multiple ways, though basically the sharing of the rule of being at home when energy consumption is high and absent when it is not is indicated by (Oshima et al., 2024). 16 Patterns as total.

We confirmed the Ours’ obvious performance advantage compared to Step-by-step and Normal in Dataset A with DANNs when larger co-variate shift exists i.e. target data with 60 and 70 degrees rotated (highlighted in red in left of Figure 2). The difference was 0.174 compared to Normal when 60 degrees rotated target, 0.074 compared to Step-by-step and 0.091 compared to Normal when 70 degrees rotated target, and variance was much smaller. In Dataset A with CoRALs, increasing the angle of rotation significantly reduces the evaluation scores of Normal, but Step-by-step and Ours were somewhat able to cope with this (highlighted in red in right).

The Figure 3 shows our methods’ superiority to previous studies in Dataset A-D(with DANNs), D(with CoRALs), namely the ideal state ”Upper bound $>$ Ours $>$ Max(Step-by-step, Normal, Lower bound)” was observed in the UDA experiment. In cases Dataset A and C, a clear difference in accuracy was identified compared to Step-by-step or Normal, the difference in accuracy was 0.074 for Ours and Step-by-step in Dataset C, 0.061 for Ours and Normal, 0.053 for Ours and Normal in Dataset A. In cases other than the above, the degree of deviation from the ideal state is case-by-case. The superiority to Step-by-step i.e. ”Ours $>$ Step-by-step” is confirmed 5 out of 8 settings and this demonstrated effectiveness of proposing method i.e. end-to-end domain invariant learning with three sets of data. The superiority to Normal i.e. ”Ours $>$ Normal” is confirmed 8 out of 8 settings, highlighting the positive impact of two times domain invariant learnings itself.

To begin, we investigate how our method performs learning at each epoch in terms of domain invariance and task classifiability. We simultaneously visualise the learning loss of domain invariance per epoch (i.e. $-L_{domain}=CrossEntropy(\cdot)$), the learning loss of task classifiability and the synchronised evaluation of task classification on test target data. Figure 4 shows that in any case, domain invariance between the source and the intermediate domain and invariance between the intermediate domain and the target are learnt in an adversarial manner and eventually a solution with high invariance is reached. Also we found that the task classifiability to the source is simultaneously optimised and eventually asymptotically approaches zero or small value. Correspondingly to the above three learnings, the evaluation scores are improving and we can recognise that our proposed optimisation algorithm is effective in the point of task classification performance for target data.

To get more insights into our method, we investigated neural networks’ learned representation at both of feature level and classifier level. The Figure 5 includes feature extractor’s learned representation, task classifier’s sigmoid probability for grid space and representation at feature level is colored in two ways i.e. domain labels and task labels. We can qualitatively recognize ours could learn domain invariant feature between source and target and task discriminability for target data while keeping balance between both. Even though we have not accessed any labels for the target data in black in the figure, we can see a probability boundary that discriminates the data approximately perfectly. On the other hand Normal and Step-by-step could not. Normal could not get domain invariance, task variance, and appropriate boundary. Step-by-step could not learn domain invariance and appropriate boundary.

The Figure 6 shows Ours’ success of domain invariant and task variant features acquirement. Other methods are struggling to get task variant features e.g. the space around the yellow-green scatter points is very dense, including the other three classes. It is difficult for them to classify these human’s behaviour classes (this four classes are $\{bike,walk,stairsup,stairsdown\}$ far from $\{stand,sit\}$, Ours could overcome this apparently). The Step-by-step could not learn domain invariant features since we can very easily identify yellow and brown scatter points at a glance.