ToAlign: Task-oriented Alignment for Unsupervised Domain Adaptation

Domain adversarial learning based UDAs typically train a domain discriminator $D$ to distinguish which domain (i.e., source or target) a sample belongs to, and adversarially train a feature extractor $G$ to fool the discriminator $D$ in order to learn domain-invariant feature representations. The network is also trained under the supervision of image classification on the labeled source samples. Particularly, $D$ is optimized to minimize the domain classification loss $\mathcal{L}_{D}$ (i.e., binary cross entropy loss). Meanwhile, $G$ is optimized to maximize the domain classification loss $\mathcal{L}_{D}$ and minimize the image classification loss $\mathcal{L}_{cls}$ (i.e., cross entropy loss):

$$\begin{split}&\operatorname*{argmin}_{D}\mathcal{L}_{D},\\ &\operatorname*{argmin}_{G}\mathcal{L}_{cls}-\mathcal{L}_{D},\\ \end{split}$$

(1)

To achieve adversarial training, usually, gradient reversal layer (GRL) [18, 19] which connects $G$ and $D$ is used via multiplying the gradient from $D$ by a negative constant during the back-propagation to $G$. $\mathcal{L}_{D}$ is typically defined as [19, 41, 15]:

$$\begin{split}\mathcal{L}_{D}(\mathbf{X}_{s},\mathbf{X}_{t})=-\mathbb{E}_{\mathbf{x}_{s}\sim\mathbf{X}_{s}}\left[\log(D(G(\mathbf{x}_{s})))\right]-\mathbb{E}_{\mathbf{x}_{t}\sim\mathbf{X}_{t}}\left[\log(1-D(G(\mathbf{x}_{t})))\right],\end{split}$$

(2)

In adversarial learning based UDAs, a feature ingested by $D$ as a holistic feature from a source or target sample, in general contains both task/classification-discriminative information and task-irrelevant information. Intuitively, aligning the task-irrelevant features would not effectively reduce the domain gap of the task-discriminative features and thus brings no obvious benefit for the classification task. Mistakenly aligning the target features with the source task-irrelevant features would hurt the discrimination power of the target features. We also experimentally confirm that in Figure 3, i.e., aligning with task-irrelevant features (TiAlign, line in purple) drastically reduces the classification accuracy on the target domain. Therefore, we propose to decompose a holistic feature of each source sample into a task-discriminative feature and a task-irrelevant feature to enable the task-oriented alignment with the target features.

Particularly, we softly select/re-weight (based on Grad-CAM [55]) the feature vector $\mathbf{f}^{s}$ of a source sample to obtain task-discriminative feature $\mathbf{f}_{p}^{s}$ that is discriminative for identifying the groundtruth class, which we refer to as positive feature. Correspondingly, the task-irrelevant feature $\mathbf{f}_{n}^{s}$ can be obtained simultaneously, which we refer to as negative feature.

As illustrated in Figure 1, we obtain a feature map $F\in\mathbb{R}_{+}^{H\times W\times M}$ (i.e., a tensor of non-negative real numbers, with height $H$, width $W$, and $M$ channels) from the final convolutional block (with ReLU layer) of the feature extractor. After spatial-wise global average pooling (GAP), we have a feature vector ${\mathbf{f}}=pool(F)\in\mathbb{R}^{M}$. The logits for all classes are predicted via the classifier $C(\cdot)$. Based on the response $C({\mathbf{f}})$, we can derive the gradient $\mathbf{w}_{cls}\in\mathbb{R}^{M}$ of $y^{k}$ w.r.t. $\mathbf{f}$:

$$\mathbf{w}^{cls}=\frac{\partial{y^{k}}}{\partial\mathbf{f}},$$

(3)

where $y^{k}$ is the predicted score corresponding to the ground-truth class $k$. As analyzed in [55, 9, 56], the gradient $\mathbf{w}^{cls}$ conveys the channel-wise importance information of feature $\mathbf{f}$ for classifying the sample into its groudtruth class $k$. We draw inspiration from Grad-CAM which uses $\mathbf{w}^{cls}$ to modulate the feature map in channel-wise to find the classification-discriminative features. Similarly, modulated with $\mathbf{w}^{cls}$, we can obtain the task-discriminative (i.e., positive) feature as:

$$\mathbf{f}_{p}=\mathbf{w}^{cls}_{p}\odot\mathbf{f}=s\mathbf{w}^{cls}\odot\mathbf{f},$$

(4)

where $\odot$ represents the Hadamard product, the attention weight vector $\mathbf{w}^{cls}_{p}=s\mathbf{w}^{cls}$, where $s\in\mathbb{R}_{+}$ is an adaptive non-negative parameter to modulate the energy $\mathcal{E}(\mathbf{f}_{p})=||\mathbf{f}_{p}||_{2}^{2}$ of $\mathbf{f}_{p}$ such that $\mathcal{E}(\mathbf{f}_{p})=\mathcal{E}(\mathbf{f})$:

$$s=\sqrt{\frac{||\mathbf{f}||_{2}^{2}}{||\mathbf{w}^{cls}\odot\mathbf{f}||_{2}^{2}}}=\sqrt{\frac{\sum_{m=1}^{M}f_{m}^{2}}{\sum_{m=1}^{M}(w^{cls}_{m}f_{m})^{2}}},$$

(5)

Motivated by the counterfactual analysis in [55], the task-irrelevant (i.e., negative) feature can be represented as $\mathbf{f}_{n}=-\mathbf{w}^{cls}_{p}\odot\mathbf{f}$, where $-\mathbf{w}^{cls}_{p}$ delights the task-discriminative channels since the task-discriminative channels (with larger values in $\mathbf{w}^{cls}_{p}$) correspond to ones with smaller values in -$\mathbf{w}^{cls}_{p}$.

To better understand and validate the discriminativeness of the positive and negative features, we visualize the spatial maps $F$ with channels modulated by $\mathbf{w}^{cls}$ and $-\mathbf{w}^{cls}$ following [55, 78]. As shown in Figure 4, the positive information is more related to the foreground objects that provide the discriminative information for the classification task, while the negative one is more in connection with the non-discriminative background regions.

Understanding from the Meta-knowledge Perspective. To enable a better understanding of ToAlign on why it works well, here, we analyse ToAlign from the perspective of meta-learning with meta-knowledge.

In an adversarial UDA framework, the image classification task and domain alignment task can be considered to be a meta-train task $\mathcal{T}^{tr}$ and a meta-test task $\mathcal{T}^{te}$, respectively. ToAlign actually introduces knowledge communication from $\mathcal{T}^{tr}$ to $\mathcal{T}^{te}$. In the meta-training stage, we can obtain the prior/meta-knowledge $\phi^{tr}$ of $\mathcal{T}^{tr}$. Without effective communication between $\mathcal{T}^{tr}$ and $\mathcal{T}^{te}$, the optimization of $\mathcal{T}^{te}$ may contradict that of $\mathcal{T}^{tr}$, considering that they have different optimization goals. To improve the knowledge communication from $\mathcal{T}^{tr}$ to $\mathcal{T}^{te}$, certain meaningful prior/meta-knowledge $\phi^{tr}$ is helpful for a more effective $\mathcal{T}^{te}|_{\phi^{tr}}$. A typical implementation of passing meta-knowledge from $\mathcal{T}^{tr}$ to $\mathcal{T}^{te}$ is based on gradients [40, 17, 35, 68, 34], i.e., $\nabla\mathcal{T}^{tr}$, which provides knowledge of $\mathcal{T}^{tr}$. Other mechanisms e.g., leveraging the parameters regularizer in a way of weight decay, are also exploited [1, 76]. In our ToAlign, instead of encoding the meta-knowledge $\phi^{tr}$ into the gradients w.r.t. the parameters, we use $\mathcal{T}^{tr}$ to learn/derive attention weights for identifying $\mathcal{T}^{tr}$-related sub-features in the feature space and then pass such prior/meta-knowledge $\phi^{tr}$ to $\mathcal{T}^{te}$ to make meta-test task $\mathcal{T}^{te}_{\phi^{tr}}$ adapt its optimization based on $\phi^{tr}$.

In this work, actually, we are motivated by the reliable human prior knowledge on what should be aligned across domains to better assist classification task for UDA (i.e., task/classification-discriminative features), while excluding the interference from task-irrelevant ones. Accordingly, in our design, we obtain the prior/meta-knowledge for identifying task-discriminative features from the classification task (meta-train) and apply it to the domain alignment task (meta-test) to achieve task-oriented alignment.

Datasets. We use two commonly used benchmark datasets (i.e., Office-Home [63] and VisDA-2017 [49]) for SUDA and a large-scale dataset DomainNet [46] for MUDA and SSDA. 1)

Office-Home [63] consists of images from four different domains: Art (Ar), Clipart (Cl), Product (Pr), and Real-World (Rw).

Each domain contains 65 object categories in office and home environments. Following the typical settings [15, 13, 68, 41], we evaluate methods on one-source to one-target domain adaptation, resulting in 12 adaptation cases in total. 2) VisDA-2017 [49] is a synthetic-to-real dataset for domain adaptation with over 280,000 images across 12 categories, where the source images are synthetic and the target images are real collected from MS COCO dataset [38]. 3) DomainNet [46] is a large-scale dataset containing about 600,000 images across 345 categories, which span 6 domains with large domain gap: Clipart (C), Infograph (I), Painting (P), Quickdraw (Q), Real (R), and Sketch (S). For MUDA, following

the settings in [46, 70, 13, 34, 62], we evaluate methods on five-sources to one-target domain adaptation, resulting in 6 MUDA cases in total. For SSDA, we take the typical protocal in [23, 51, 13], where there are 7 SSDA cases conducted on the 4 sub-domains (i.e., C, R, P and S) with 126 sub-categories selected from DomainNet. All methods are evaluated under the one-shot/three-shot setting respectively, where besides unlabeled samples, one/three sample(s) per class in the target domain are available during training.

Implementation Details. We apply our ToAlign on top of two different baseline schemes: DANNP [15, 68] and HDA [13]. DANNP is an improved variant of the classical adversarial learning based adaptation method DANN [18, 19], where the domain discrimination $D$ is conditioned on the predicted class probabilities. HDA is a state-of-the-art adversarial training based method which leverages the domain-specific representations as heuristics to obtain domain-invariant representations.

We use the ResNet-50 [22] pre-trained on ImageNet [31] as the backbone for SUDA, while using ResNet-101 and ResNet-34 for MUDA and SSDA respectively. Following  [68, 41, 13], the image classifier $C$ is composed of one fully connected layer. The discriminator $D$ consists of three fully connected layers with inserted dropout and ReLU layers. We follow [74] to take an annealing strategy to set the learning rate $\eta$, i.e., $\eta_{t}=\frac{\eta_{0}}{(1+\gamma p)^{\tau}}$, where $p$ indicates the progress of training that increases linearly from 0 to 1, $\gamma=10$, and $\tau=0.75$. The initial learning rate $\eta_{0}$ is set to $1e-3,3e-4,3e-4$, and $1e-3$ for SUDA on Office-Home, SUDA on VisDA-2017, MSDA on DomainNet, and SSDA on DomainNet, respectively. All reported experimental results are the average of three runs with different seeds.

Effectiveness of ToAlign on Different Baselines. Our proposed ToAlign is generic and applicable to different domain adversarial training based baselines, where we focus on what features to align instead of the alignment methods. The last four rows in Table 1 show the ablation comparisons on Office-Home. Our ToAlign improves the accuracy of baseline DANNP and HDA by 2.0% and 1.1% respectively. As can be seen from the results in Table 1, Table 2, Table 5 and Table 6, our ToAlign can consistently bring significant improvement over the baseline schemes under different domain adaptation settings, i.e., SUDA, MUDA and SSDA. ToAlign enables the domain alignment task to proactively serve the classification task, resulting in more effective feature alignment for image classification.

Effectiveness of Different Ways to Obtain Positive Features. As mentioned in Sec. 3.2, we use $\mathbf{w}^{cls}_{p}=s\mathbf{w}^{cls}$ as the attention weight (which conveys the classification prior/meta-knowledge) to derive positive feature $\mathbf{f}_{p}$, where $s$ is a parameter to modulate the energy of $\mathbf{f}_{p}$. We study the influence of $s$ under the setting of Rw$\rightarrow$Cl on Office-Home for our scheme DANNP+ToAlign and illustrate the results in Table 3. As discussed around Eq. (5), we can use an adaptively calculated $s$, which achieves 2% improvement over the baseline on target test data. Moreover, we can treat $s$ as a preset hyper-parameter. We found that the performance drops drastically if $s$ is too small (e.g., $s=1$). That is because the energy of the source positive feature will get too weak when $s$ gets too small (e.g., the source feature $\mathbf{f}$’s average energy $\mathcal{E}(\mathbf{f})$ is about 800; if $s=1$, the source positive feature’s average energy $\mathcal{E}(\mathbf{f}_{p})$ is about 2). Then, it would be ineffective to align the target with the source positive features. When $s$ is larger than 16, the performance significantly outperforms the baseline and approaches the result of using adaptive $s$. As an optional design choice, we could transform the weight $\mathbf{w}^{cls}$ with certain activation function $\sigma(\cdot)$ such as Sigmoid or Softmax followed by a best selected scaling factor $s$, i.e., $\mathbf{w}^{cls}_{p}=s\sigma(\mathbf{w}^{cls})$. We found the results (i.e., 69.6/69.7 for Sigmoid/Softmax) are close to that without activation function. We reckon that what is more important is the relative importance among the elements in $\mathbf{w}^{cls}$. For simplicity, we finally take the adaptive $s$ (cf. Eq. 5) for all experiments.

Single Source Unsupervised Domain Adaptation (SUDA). We incorporate our ToAlign into the recent state-of-the-art UDA method HDA [13], denoted as HDA+ToAlign. Table 1 shows the comparisons with the previous state-of-the-art methods on Office-Home. HDA+ToAlign outperforms all the previous methods and achieves the state-of-the-art performance. It is noteworthy that HDA+ToAlign achieves the best adaptation results on almost all the one-source to one-target adaptation cases thanks to the effective feature alignment for classification. The results on VisDA-2017 could be found in Appendix, where HDA+ToAlign outperforms HDA by 0.9%.

Multi-source Unsupervised Domain Adaptation (MUDA). Table 2 shows the results on DomainNet, where all the methods take ResNet-101 as the feature extractor. We build our Baseline based on HDA [13]. For simplicity, we replace the multi-class domain discriminator in the original HDA by a two-class one as in [61, 18, 70]. Note that CMSS [70] selects suitable source samples for alignment while our ToAlign selects task-discriminative sub-feature for each sample for task-oriented alignment. Compared with Baseline, ToAlign brings about 0.9% improvement and helps to achieve the best performance on this more challenging dataset.

Semi-supervised Domain Adaptation (SSDA). Table 5 and Table 6 show the results on one-shot and three-shot SSDA respectively, where all the methods use ResNet-34 as backbone. To compare with previous methods, we apply ToAlign on top of HDA. HDA+ToAlign outperforms HDA by 0.6%/1.2% for one-/three-shot settings, and surpasses all previous SSDA methods.

In Table 4, we compare the training complexity and performance of ToAlign with baseline DANNP, and DANNP+MetaAlign [68] which incorporates meta-learning to coordinate the optimization of domain alignment and image classification. In contrast, inspired by the prior knowledge of what feature should be aligned to serve classification task, we distill such meta-knowledge from classification task and explicitly pass it to alignment task for classification-oriented alignment, eschewing complex optimization. Compared with baseline, ToAlign introduces negligible additional computational cost (only 7%) and occupies almost the same GPU memory as the baseline, which is much smaller than that of DANNP+MetaAlign, which almost doubles the computational cost due to its complex meta-optimization. Thanks to our explicit design which makes domain alignment effectively serve the classification task, our ToAlign achieves superior performance to MetaAlign.

We visualize the target feature response maps $F$ (which will be pooled to be the input of the image classifier) of the Baseline (DANNP) and ToAlign in Figure 5. Baseline sometimes focuses on the background features which are useless to the image classification task, since it aligns the holistic features without considering the discriminativeness of different channels/sub-features. Thanks to our task-oriented alignment, in ToAlign, the features with higher responses are in general related to task-discriminative features, which is more consistent with human perception. More results can be found in the Appendix.