Delving into the Continuous Domain Adaptation

The main objective of CDA is to learn a generalized model tackling discrepancies among all domains sampled on a continuous attribute. To handle the statistics of unseen domains, we model each distribution parameter $\theta$ as a point in the high-dimension space, as shown in figure 3. Thus, connecting all values of the attribute $a$, we can obtain a trajectory $\theta(a)$ in the space. In this way, infinite domains’ data distribution is represented as a trajectory. Our strategy is to study the geometry of the continuous attribute and use it as an inductive bias in the modeling. Inspired by this principle, we design the alternating direction training strategy, including two steps: Pull and Shrinkage. As shown in figure 3, in Pull Step, we regard the trajectory formed by the two source domains as the inductive bias and pull target domains to the trajectory so that unseen target domains have a higher chance to be close to the trajectory as well. In Shrinkage Step, we progressively shrink and shorten the trajectory. Alternating the two steps in training, the proposed method is proved to generalize to unseen target domains.

As unseen target data is infeasible in training, we approximate it with disturbance on probe target data through which the source-unseen target discrepancy can be estimated by source-probe target discrepancy. When the disturbance is limited to $0$, gradient plenty can be derived as a constraint of the discrepancy. Then, the discrepancy between the unseen target and source domain is minimized by the constraint term. In addition, to obtain global discrepancy by perceiving more source information than a mini-bacth, we decouple the discrepancy from the mini-batch size by utilizing queues.

The overall architecture is shown in figure 2. Specifically, the framework comprises a joint feature encoder $E$, which extracts the feature $e=E(x)$ of input data $x$, and two adapter modules $A_{1}$ and $A_{2}$, each of which consists of two paired classifiers. The reason of adopting two adapter modules rather than one is that, one adapter fails to simultaneously fit two source domains during early training. As a result, it estimates biased cross-domain discrepancy which leads to fragile optimized results. Thus, we design two pairs of classifiers ($F_{1}$,$F_{1}^{\prime}$) and ($F_{2}$,$F_{2}^{\prime}$) to classify source samples and tackle two discrepancy measurements ($\mathcal{D}_{1}$,$\mathcal{D}_{2}$) for two source domains respectively.

To improve model generalization to unseen and infinite target domains, a novel alternating direction strategy is proposed to progressively reduce multiple domains’ discrepancies, as shown in Figure 3. The alternating direction training strategy is composed of two alternating steps. To be specific, in Pull Step, we compute discrepancies between probe target domain $T_{i}$ and two sources domains $S_{1}$, $S_{2}$. Then we reduce the sum of them to pull all target domains close to the trajectory formed by two source domains. In Shrinkage Step, the distance of the trajectory is shortened by reducing the discrepancy between two source domains. Alternating the two steps, the training strategy enables more stable and robust optimization.

The final objective of CDA is to improve the model generalization on the whole data distribution. Nevertheless, the above strategy only reduces discrepancies between source and the seen probe target domains, without considering the unseen target ones. To overcome it, a Continuity Constraint is proposed to regularize the continuous geometry of domain attributes by regularizing the gradient penalty in Pull Step. It is useful because it implicitly constrains that pulling probe target domains to source ones tends to pull the unseen target ones.

As there are infinite target domains and our method has to pull finite target domains to the source ones in the stochastic manner, it is challenging to maintain the global semantic features of the source domain using mini-batch methods. Therefore, a novel implementation is adopted to maintain the global view of two source domains using queues $Q_{1}$ and $Q_{2}$, preserving and updating the complete and up-to-date source domain information. The queue has the following advantages over the previous mini-batch form: 1) the queue size is flexible to set and can be much larger than a mini-batch size; 2) the queue always maintains the newest source domain features.

Multi-Domain Discrepancy. The hypothesis-induced discrepancies (Ben-David et al., 2010; Zhang et al., 2019) require taking supremum over hypothesis space to measure the discrepancy of two domains. In this work, we extend it into multi-domains. Specifically, we propose two pairs of classifiers. Content classifiers $F_{1}$ and $F_{2}$ are used to classify content labels in each source domain, while auxiliary classifiers $F_{1}^{\prime}$ and $F_{2}^{\prime}$ are used to estimate the discrepancies (together with content classifiers) between each source domain and the other target domains for the previously mentioned alternating training strategy. The discrepancy of domain $P$ and $Q$ is formed using the supremum of prediction differences between a pair of classifiers $F_{j},F^{\prime}_{j}(j=1,2)$ as follows:

where $\circ$ denotes function composition, $h_{F}$ is a labeling function: $h_{F}(e)=\arg\max_{y}p(y|x)$, where $p$ is the softmax probabilities predicted by $F$, and $\sigma$ is the softmax function, $\sigma_{j}(z)=\exp(z_{j})/\sum_{i}\exp(z_{i})$.

Advantages of using two adapters and discrepancies. Due to using two source domains, we adopt $F_{i}$ and ${F_{i}}^{{}^{\prime}}$, $i\in\{1,2\}$, to estimate two separate discrepancies between some target and each ($i^{th}$) source domain. It is hard for one classifier to fit two source domains at the beginning of training because they lie in different trajectory positions. Our discrepancy measurement is based on classifiers, it is more accurate to estimate two discrepancies separately.

Pull Step. Feeding samples from $S_{1},S_{2}$ and $T_{i}$ to the network, we train the encoder $E$ to pull the target domains on the trajectory path formed by the two source domains. It is achieved by reducing the sum of $\mathcal{D}_{1}(e_{s_{1}},e_{t_{i}})$ and $\mathcal{D}_{2}(e_{s_{2}},e_{t_{i}})$, which reaches the minimum when $T_{i}$ is on the trajectory. Networks can be trained based on the following objective, $\mathcal{L}_{p}$:

Shrinkage Step. After pulling target domains to the trajectory in the Pull Step, source-only samples are utilized to optimize the encoder $E$ and two classifiers $F_{1}$, $F_{2}$ to shrinkage the trajectory distance by reducing the following discrepancy between two source domains, denoted as $\mathcal{L}_{s}$.

As the source labels are available, the cross-entropy loss $\mathcal{E}(x,y)$ is adopted to train the networks $E,F_{1},F_{2}$ as follow, denoted as $\mathcal{L}_{ce}$:

Remarks. One may argue that we should train two steps together in a unified framework. Although it is practicable, we argue that it will pull both target and source domains together and the geometry of the trajectory will change drastically. As a result, the continuity will not be constrained effectively, which is harmful for generalization on unseen target domains. The comparison result is shown in Sec 5.3.

As we take supremum to define the discrepancy in Eq (4.3), we first show the optimum of $F^{\prime}_{i}$ that reaches the supremum and the formulation of loss function Eq (2) and Eq (3) in two P Step and S Step, respectively. Denoting $P_{s_{j}}$ and $P_{t_{i}}$ as the distributions of two domains’ features $e_{s_{j}}$ and $e_{t_{i}}$ encoded by $E$, we derived the optimal values of $\sigma_{h_{F(e)}}\circ F^{\prime}(e)$ in P and S Steps as follows:

Global optimum. Since the JS-divergence between two distributions is always non-negative and zero only when they are equal. Thus, the solution of P Step in Eq (7) is $p_{s_{1}}=p_{t_{i}}$ and $p_{s_{2}}=p_{t_{i}}$, while that of S Step is $p_{s_{1}}=p_{s_{2}}$. Two steps run alternatively and reach global minimum when $p_{s_{1}}=p_{s_{2}}=p_{t_{i}}$ i.e., the domains are aligned perfectly and the trajectory becomes a point as shown in the right part of Figure 3.

The above strategy is not guaranteed to pull the unseen target domains close to the trajectory. To address it, we propose to add the continuity constraint in the optimization. Given the probe and the neighborhood unseen target domain with attribute $a$ and $a+\Delta a$ respectively, their encoder features can be represented as $e_{a}$ and $e_{a+\Delta a}$. Though estimating the discrepancy $\mathcal{D}_{i}(e_{s_{i}},e_{t(a+\Delta a)})$ accurately is intractable, we can find a constant $\eta$ that:

As the domain variations are mainly caused by the attribute variations, when $\Delta a\to 0$, $e_{t}{(a+\Delta a)}=e_{t}(a)+\beta\Delta a$, where $\beta$ is another constant. Therefore, Eq.(8) can be reformulated as:

Empirically we set ${\eta}/{\left\|\beta\right\|}=1$ and implement it as the gradient penalty in Pull Step, denoted as $\mathcal{L}_{gp}$:

In this way, when we minimize the discrepancy of between the source and probe target domains, $\mathcal{D}_{i}(e_{s_{i}},e_{t})$, the discrepancy of between the source and unseen target domains , $\mathcal{D}_{i}(e_{s_{i}},e_{t(a+\Delta a)})$, is minimized implicitly.

The fundamental problem of the alternating direction strategy and the CDA task is to robustly estimate the domain shifts in the stochastic manner. Existing discrepancy computation performance heavily relies on mini-batch size, especially in CDA task where there are infinite target domains under test and the discrepancy is biased. To decouple the discrepancy from the mini-batch size, we introduce queue to obtain more global and complete cross-domain discrepancy. Concretely, in Shrinkage Step, the samples in two source domains are progressively replaced in two queues, denoted as $Q_{1}$ and $Q_{2}$. In Pull Step, we compute the discrepancy between the target domain and two source domains.

The overall optimization is as follows:

Rotating EMNIST(Cohen et al., 2017) is an extension of MNIST including six different splits including 131,600 characters of 47 classes. Images are rotated by $0^{\circ}$ to $180^{\circ}$. We adopt this dataset to study continuous domain variations caused by rotation angles of 2D images.

SmallNorb(LeCun et al., 2004) contains 48,600 images of 50 toys from 5 generic categories: four-legged animals, human figures, airplanes, trucks, and cars. The toys were imaged under 9 elevations (30 to 70 degrees every 5 degrees). We adopt this dataset to study continuous domain variations caused by elevation camera views.

Face Age Synthesis. Age is a common and continuous attribute that causes humans’ appearance variations, without existing dataset to support, we propose to use Lifespan Age Transformation Synthesis (Or-El et al., 2020) to synthesize a full span of 0–90 ages with the interval of 10. 294,330 images of 1000 identity are selected from CelebA(Liu et al., 2015) under synthesis based on the top 1,000 identities FID (Xu et al., 2018).We adopt this dataset to study continuous domain variations caused by age of human identities.

Benchmark Analysis: We conduct an in-depth analysis of the cross-domain statistics in CONDA benchmark in Figure 5. Three results are reported including the 1) target domain performances using different source-only models; 2) average target domain performances of source-only models; 3) MMD (Tolstikhin et al., 2016) between source and target domain features using pre-trained source-only model.

From the Figure 5 (a)(d)(g), we made two observations: first, the performance of each source domain model (a plot using a specific color) reaches the peak at the source attribute but successively drops when the attribute keeps moving far away from the source one. This suggests that the domain shift increases with the increasing attribute differences. Second, SmallNorb and Face Age Synthesis exhibit the similar smooth performance patterns among all target domains, however, the performance curve in Rotating EMNIST has a very steep peak point on the source attribute and performances drop very quickly for other attributes/domains. This suggests that the current deep network architecture is not suitable to tackle rotation variations and the domain gaps in this dataset seem the largest. It confirms the fact that deep architectures cannot tackle the domain shift in CDA.

In the Figure 5 (b)(e)(h), SmallNorb and Face Age Synthesis exhibit the similar average performances patterns among all target domains, where the performance first rises and then falls. Instead, the performance curve in EMNIST is stable among different attributes. It is reasonable because deep networks are not able to handle rotation variations.

Finally, it is observed in Figure 5 (c)(f)(i) that the closer to the diagonal, the darker the point color is. The phenomenon is strongly correlated to the accuracy curve. It provides the insight that the choice of samples to annotate has an impact on the performance when the annotation budget is limited.

Splits. To perform comprehensive evaluation, we design 13 splits, with the attempt to cover all combinations of choices of source, probe target and target test domains in the evaluation. The source domain locations in the attribute space are denoted as P1-P4, while four settings of the segment to randomly sample the probe target denotes as S1-S4, as shown in Figure 6.

Implementation. We perform extensive evaluations on three datasets. For Rotating EMNIST and smallNorb, a four-layer CNN encoder $E$ and a two-layer MLP for each classifier are adopted. For Face Age Synthesis, we use pre-trained ResNet-50 from ImageNet as the encoder. We adopt Adam with learning rate $1\times 10^{-4}$. See the appendix for more details. The average classification accuracy of all target domains is used in the evaluation. The results of 13 splits on Rotating EMNIST, SmallNorb and Face Age Synthesis are shown in Table 2.

Comparisons. In each split, performances of our method and six existing comparison methods are reported. As it is the first time to address the problem of CDA, we reproduce existing methods in different tasks. As To summarize, these comparison methods comprise source only(SO) model, Continuously Indexed DA(CIDA(Wang et al., 2020a)), Single Source DA(BCDM (Li et al., 2021b), ATDOC(Liang et al., 2021)), Multi Source/Target DA(MCC(Jin et al., 2020)) and Domain Generalization(StableNet(Zhang et al., 2021)).

To summarize the main observations: (1) our method outperforms all other comparison methods in all splits and datasets. It demonstrates that our methods can effectively handle the continuous domain shifts using two source domains. (2) It is worth noting that some methods (i.e., CIDA, ATDOC) perform worse than SO performance.

In this section, we perform ablation studies on all splits to investigate the effectiveness of the proposed three main contributions: discrepancy measure, two-stage strategy and feature queues. Results are shown in Table 3.

We compare the following variant methods and analyze their results. V1: We adopt three binary domain discriminators to encourage domain confusions between ($S_{1},S_{2}$), ($S_{2},T$) and ($S_{1},T$). V2: We merge P and S Steps into a single step and minimize three pairs of discrepancies at the same time, without fixing $S_{1}$ and $S_{2}$ in P Step.V3: We merge P and S Steps into a single step without fixing $S_{1}$ in P Step.V4: We merge P and S Steps into a single step without fixing $S_{2}$ in P Step.V5: We remove the S Step but keeping other components in our method. V6: We remove the queue implementation. V7: We remove the continuity constraint.

Compared with our method, the adaptation performances of all variants drastically decrease, and Variant 1 achieves the worst performance. Variant 1 is equivalent to reducing three JS divergence losses. The proposed method outperforms this variant, demonstrating that directly minimizing pair-wise discrepancies fails to generalize to unseen target domain in the continuous domain space. In addition, disjoint domains in continuous domain space leads to unreliable training of the encoder as discussed in (Arjovsky and Bottou, 2017). From comparison with variants 2-5, merging stages and unfixing source domains in P Step will cause ambiguous optimization direction which is harmful for adaptation. For example, if we unfix the source domain, the geodesic path is dynamic and the target domains are hard to converge. Comparing with variant 6, it seems that designed queues can further improve the model performances. It confirms that queues may be able to minimize the error term between the expected and the empirical ones. Comparing with variant 7, it seems that continuity constraint is effective.

Why are some methods worse than SO? It demonstrates that CDA is a challenging problem, and most existing DA methods can not be applied directly. The reason is twofold: 1) It may be because the adversarial-based methods only align the observed probe target domain and source domain but do not consider the unseen target domain. As a result, the model overfits on source and probe target domains. 2)These methods propose one classifier for all domains. Therefore, it is hard to fit all domains with large discrepancies. Meanwhile, it shows the advantage of proposing two classifiers.

The choice of two source domains its effects We set up this novel benchmark to analyze this point. If two domains are similar, the adaptation performances degrade. It can be seen in Tables 2 that P4 achieves the worst performance.

Queue size. We evaluate our methods with different queue sizes, and the results are visualized in Figure 7(a). It can be observed that the accuracy curve raises at the beginning, and then falls. It is because that the queue enlarges the number of samples that compute the discrepancy, thus it promotes the adaptation. But large size makes the queues have more features extracted by the model before several iterations, which lead to imprecise discrepancy computation.

Number of probe target domains. We evaluate the task using different number of probe target domains in the experiment in Figure 7(b). It can be observed that with the increasing number of probe target domains, the overall accuracy increases for all methods. Our method always achieves the best-performing method for different setups.

We perform t-SNE plot of the two source domain features and target features in Figure 8. The more evenly the points of three colors are mixed, the better the domains are aligned. We highlighted some areas using red dashed circles in the subfigure of CIDA, BCDM and ATDOC. In these areas, grey dots (target) dominate, with very few blue (source 1) and orange (source 2) dots overlapping with grey ones. However, this pattern does not appear in the t-SNE plot of our method, where dots of three colors are mixed more evenly. It seems that our method achieves the best domain alignment on unseen target domains as the source features spread more in the target zones.

In Figure 7(c), we plot the loss $\mathcal{L}_{s}$ with respect to the number of iterations in the training procedure of smallNorb. The loss implies the distance between all target domains and the trajectory, that is – the aim of this is to verify the effectiveness of our Pull Step. It can be observed that our method achieves comparable convergence and relatively small loss, which suggests that target domains are able to be pulled close to the trajectory.

In Figure 7(d), we visualize the $l2$ distance between two classifiers’ predictions of the target sample. The aim is to demonstrate the distance between two source domains in the trajectory, that is, verifying the effectiveness of our Shrinkage Step. First, due to discrepancies between two source domains and the high impact of cross-entropy loss in the beginning, two classifiers learn to discriminate in their individual domain. Therefore, the $l2$ distance starts increasing. Next, due to our proposed discrepancy measurement and reduction losses dominating the training, domains are aligned together and the two classifiers are pulled closely (distances are small). The convergence implies the shrinkage of the trajectory formed by all domains.

Note that we can observe the regularly alternating patterns of the loss functions in Figure 7(c-d). This is because the training procedure is in the alternating manner and the two steps are adversarial. The above two convergences jointly demonstrate that our method is able to reduce the discrepancy progressively.