Towards Stable and Comprehensive Domain Alignment:
Max-Margin Domain-Adversarial Training

In unsupervised domain adaptation, we assume that a model works with a labeled dataset $\textbf{X}_{S}$ and an unlabeled dataset $\textbf{X}_{T}$. Let $\textbf{X}_{S}=\{(\textbf{x}^{s}_{i},y^{s}_{i})\}_{i\in[{N_{s}}]}$ denote the labeled dataset of $N_{s}$ samples from the source domain, and the certain label $y^{s}_{i}$ belongs to the label space $Y$ that is a finite set ($Y={1,2,...,K}$). The other dataset $\textbf{X}_{T}=\{\textbf{x}^{t}_{i}\}_{i\in[{N_{t}}]}$ has $N_{t}$ samples from the target domain but has no labels. We further assume that two domains have different distributions, i.e. $\textbf{x}^{s}_{i}\sim\mathcal{D}_{S}$ and $\textbf{x}^{t}_{i}\sim\mathcal{D}_{T}$. In other words, there exist some domain shift (Ben-David et al., 2010) between $\mathcal{D}_{S}$ and $\mathcal{D}_{T}$. The ultimate goal is to learn a model that can predict the label $y^{t}_{i}$ given the target input $\textbf{x}^{t}_{i}$.

To achieve domain alignment, Domain-Adversarial Training (DAT) is a minimax game between a shared feature extractor $F$ for two domains and a domain classifier $D$. The domain classifier is trained to determine whether the input sample belongs to the source or the target domain while the feature extractor learns to deceive the domain classifier, which is formulated as:

We usually utilize Convolutional Neural Network (CNN) as the feature extractor and fully connected layers (FC) as the domain classifier. Theoretically, DAT reduces the cross-domain discrepancy and helps learn domain-invariant representations (Ganin et al., 2016). However, the training of DAT is rather unstable. Without sophisticated tuning of the hyper-parameters, DAT cannot often reach the convergence. Through empirical experiments, we observe that such instability is due to the imbalanced adversarial game between $D$ and $F$. The binary domain discriminator $D$ can easily achieve convergence with very high accuracy at an early training epoch, while it is much harder for the feature extractor $F$ to fool the domain discriminator and to simultaneously perform well on the source domain. In this sense, there is a strong likelihood that the domain classifier overwhelms DAT, and the only solution is to palliate the training of $D$ by tuning the hyper-parameters according to different tasks. In our method, we restrict the capacity of the domain classifier so as to form a minimax game in a harmonious manner. Inspired by the max-margin loss in Support Vector Machine (SVM) (Cristianini et al., 2000) (i.e. hinge loss), if we push the source domain and the target domain away from a margin rather than as far as possible, then the training task of $F$ to fool $D$ becomes much easier. For a binary domain classifier, we define the margin loss as

where $y$ is the predicted domain label, $[\cdot]^{+}$ is $max(0,\cdot)$, $m$ is a positive margin and $t$ is the ground truth label for two domains (assuming $t=-1$ for the source domain and $t=1$ for the target domain). Then we introduce our MDAT scheme based on an innovative network architecture.

Besides the training instability issue, DAT also suffers from restrictive feature-level alignment – lack of pixel-level alignment. To realize stable and comprehensive domain alignment together, we first propose an Adversarial Reconstruction Network (ARN) and then elaborate MDAT.

As depicted in Figure 1, our model consists of three parts including a shared feature extractor $G_{e}$ for both domains, a label predictor $G_{y}$ and a reconstruction network $G_{r}$. Let the feature extractor $G_{e}(\textbf{x};\theta_{e})$ be a function parameterized by $\theta_{e}$ which maps an input sample x to a deep embedding z. Let the label predictor $G_{y}(\textbf{z};\theta_{y})$ be a task-specific function parameterized by $\theta_{y}$ which maps an embedding z to a task-specific prediction $\hat{y}$. The reconstruction network $G_{r}(\textbf{z};\theta_{r})$ is a decoding function parameterized by $\theta_{r}$ that maps an embedding z to its corresponding reconstruction $\hat{\textbf{x}}$.

The first learning objective for the feature extractor $G_{e}$ and the label predictor $G_{y}$ is to perform well in the source domain. For a supervised $K$-way classification problem, it is simply achieved by minimizing the negative log-likelihood of the ground truth class for each sample:

where $\textbf{y}^{s}_{i}$ is the one-hot encoding of the class label $y^{s}_{i}$ and the logarithm operation is conducted on the softmax predictions of the model.

The second objective is to render feature learning to be domain-invariant. This is motivated by the covariate shift assumption (Shimodaira, 2000) that indicates if the feature distributions $S(\textbf{z})=\{G_{e}(\textbf{x};\theta_{e})|\textbf{x}\sim\mathcal{D}_{S}\}$ and $T(\textbf{z})=\{G_{e}(\textbf{x};\theta_{e})|\textbf{x}\sim\mathcal{D}_{T}\}$ are similar, the source label predictor $G_{y}$ can achieve a similar accuracy in the target domain. To this end, we first design a decoder network $G_{r}$ that serves as a domain classifier. In MDAT, we train the decoder network $G_{r}$ to only reconstruct the features in the source domain and to push the features in the target domain away from a margin. In this way, the decoder has the functionality of distinguishing the source domain from the target domain. The objective of training $G_{r}$ is formulated as

where $m$ is a positive margin and $\mathcal{L}_{r}(\cdot)$ is the mean squared error (MSE) term for the reconstruction loss defined as

where $||\cdot||^{2}_{2}$ denotes the squared $L_{2}$-norm. Compared the normal binary domain classifier (e.g. fully connected layers), the decoder network is tailored as a smoothing domain discriminator by separating two domains from a specific margin rather than as far as possible.

Oppositely, to form an adversarial game, the feature extractor $G_{e}$ learns to deceive $G_{r}$ such that the learned target features are indistinguishable to the source ones, which is formulated by:

Then the whole learning procedure of ARN with MDAT can be formulated by:

where $\mathcal{L}_{y}$ denotes the negative log-likelihood of the ground truth class for labeled sample $(\textbf{x}_{i}^{s},\textbf{y}_{i}^{s})$ and $\alpha$ controls the interaction of the loss terms. In the following section, we derive an optimal solution of MDAT and provide theoretical justifications on how MDAT reduces the distribution discrepancy for UDA.

Considering the adversarial game between a reconstruction network $R$ and a feature extractor $E$ (i.e. $G_{r}$ and $G_{e}$ in our network, respectively), we prove that if the feature extractor $E$ maps both source domain $x^{s}\sim P_{S}(x^{s})$ and target domain $x^{t}\sim P_{T}(x^{t})$ to a common feature space $\mathcal{Z}=\{z=E(x)|z\sim P_{z}(z)\}$, the MDAT system reaches a Nash Equilibrium. This theoretically explains how MDAT enables the feature extractor to learn domain-invariant features. Similar to EBGAN (Zhao et al., 2017), we assume $E$ and $R$ have infinite capacity. Denote $R$ as the MSE of the reconstruction network. We first define two objectives:

In MDAT, we train the feature extractor $E$ to minimize the quantity $V(E,R)$ and train the reconstruction network $R$ to minimize the quantity $U(E,R)$. A Nash equilibrium of our system is a pair $(E^{*},R^{*})$ that satisfies:

Proof. We first prove Eq.11:

As we know $f(x)=x+[m-x]^{+}$ is monotonically increasing on $[0,+\infty)$, $V(E^{*},R)$ reaches its minimum when $R^{*}(z)=0$:

When $R^{*}(z)=0$, we expand $U(E,R^{*})$ in Eq.12:

As $U(E,R^{*})\geq 0$, we get $U(E^{*},R^{*})\leq U(E,R^{*})$.

It can be easily observed that the optimal solution of MDAT is a Nash equilibrium when the feature extractor maps two domains into a common feature space, i.e. aligning the distributions in the feature space.

We further investigate how the proposed method connects the learning theory of domain adaptation. The rationale behind domain alignment is motivated from the learning theory of non-conservative domain adaptation problem by Ben-David et al. (Ben-David et al., 2010):

Theoretically, when we minimize the $\mathcal{H}\triangle\mathcal{H}$-distance, the upper bound of the expected error for the target domain is reduced accordingly. As derived in DAT (Ganin & Lempitsky, 2015), assuming a family of domain classifiers $\mathcal{H}_{d}$ to be rich enough to contain the symmetric difference hypothesis set of $\mathcal{H}_{p}$, such that $\mathcal{H}_{p}\triangle\mathcal{H}_{p}=\{h|h=h_{1}\oplus h_{2},\,h_{1},h_{2}\in\mathcal{H}_{p}\}$ where $\oplus$ is XOR-function, the empirical $\mathcal{H}_{p}\triangle\mathcal{H}_{p}$-distance has an upper bound w.r.t. the optimal domain classifier $h$:

where $\hat{\mathcal{D}}_{S}$ and $\hat{\mathcal{D}}_{T}$ denote the distributions of the source and target feature space $\mathcal{Z}_{S}$ and $\mathcal{Z}_{T}$, respectively. Note that the MSE of $G_{r}$ plus a ceiling function is a form of domain classifier $h(\textbf{z})$, i.e. $\lceil[m-\mathcal{L}_{r}(\cdot)]^{+}-0.5\rceil$ for $m=1$. It maps source samples to $0$ and target samples to $1$ which is exactly the upper bound in Eq.19. Hence, our reconstruction network $G_{r}$ maximizes the domain discrepancy with a margin and the feature extractor learns to minimize it adversarially.

Compared with the conventional DAT-based methods that are usually based on a binary logistic network (Ganin & Lempitsky, 2015), the proposed ARN with MDAT is more attractive and incorporates new merits conceptually and theoretically:

(1) Effective gradients and balanced adversarial training. Using the decoder as domain classifier with a margin loss to restrain its overwhelming capacity in adversarial training, the adversarial game can continuously provide effective gradients for training the feature extractor, leading to better alignment and balanced adversarial training. Moreover, through the experiments in Section 4, we discover that our method shows more stable training procedure and strong robustness to the hyper-parameters, i.e. $\alpha$ and $m$, greatly alleviating the parameters tuning for model selection.

(2) Richer information for comprehensive domain alignment. Rather than typical DAT that uses a bit of domain information, MDAT utilizes the reconstruction network as the domain classifier that captures more domain-specific and pixel-level features during the unsupervised reconstruction (Bousmalis et al., 2016). Therefore, MDAT further helps address pixel-level domain shift apart from the feature-level shift, leading to comprehensive domain alignment in a straightforward manner.

(3) Feature interpretability for method validation. MDAT allows us to visualize the features by directly reconstructing target features to images by the decoder network. It is crucial to understand to what extent the features are aligned since this helps to reveal the underlying mechanism of adversarial domain adaptation. We interpret these adapted features in Section 4.3.

Digits (Ganin et al., 2016). We utilize four digit datasets including MNIST, USPS, SVHN and Synthetic Digits (SYN) that form four transfer tasks: MNIST$\to$USPS, USPS$\to$MNIST, SVHN$\to$MNIST and SYN$\to$SVHN.

Office-Home (Venkateswara et al., 2017) is a challenging UDA dataset including 15,500 images from 65 categories. It comprises four extremely distinct domains: Artistic images (Ar), ClipArt (Cl), Product images (Pr), and Real-World images (Rw). We evaluate on all twelve transfer tasks.

WiFi Gesture Recognition (Zou et al., 2019) consists of six gestures recorded by Channel State Information (CSI) (Xie et al., 2018). Each CSI sample is a 2D matrix that depicts the gesture with the surrounding layout environment. Thus, the CSI data collected in two environments forms two domains, which formulates a spatial adaptation problem.

We compare with state-of-the-art UDA methods that perform three ways of domain alignment. Specifically, MMD regularization (Long et al., 2015) and CORAL (Sun & Saenko, 2016) are based on statistical distribution matching. DRCN (Ghifary et al., 2016) and DSN (Bousmalis et al., 2016) use the reconstruction network for UDA, while more prevailing UDA methods adopt domain-adversarial training including DANN (Ganin & Lempitsky, 2015), ADDA (Tzeng et al., 2017), CyCADA (Hoffman et al., 2018), CDAN (Long et al., 2018), MCD (Saito et al., 2018), CADA (Zou et al., 2019), TransNorm (Wang et al., 2019a) and BSP (Chen et al., 2019). The baseline results are reported from their original papers where available.

We used Pytorch to implement our model. For Digits dataset, we follow the same protocol in (Hoffman et al., 2018) and the same network architecture of (Ganin & Lempitsky, 2015). For Office-Home, we adopt ResNet-50 pretrained on ImageNet as our backbone. According to the standard protocols in (Long et al., 2015), we employ all the labeled source samples and unlabeled target samples for training. For WiFi Gesture Recognition data, we employ the modified LeNet and the standard protocol in (Zou et al., 2019). The designs of $G_{r}$ are the inverse of $G_{e}$ with pooling operation replaced by upsampling. We fix $\alpha=0.02$ and $m=5$ in all the experiments, which are obtained on SVHN$\to$MNIST by Baysian optimization (Malkomes et al., 2016). We adopt mini-batch SGD optimizer with momentum of 0.9 and the progressive training strategy in DANN (Ganin & Lempitsky, 2015).

The classification accuracies on Digits are shown in Table 1. Our method outperforms all other methods on four transfer tasks. Specifically, for SVHN$\to$MNIST where severe pixel-level domain shift exists, our method significantly improves DANN by 22.7%, which justifies the efficacy of ARN for addressing pixel-level shift. Our method also performs well when the target domain are quite small, achieving 98.6% accuracy on MNIST$\to$USPS. In Table 2, our method improves the source-only model by 32.9% on WiFi spatial adaptation problem, which indicates that MDAT is also helpful for non-visual domain adaptation problem. Table 3 shows the performance on large-scale dataset and MDAT yields better performance against other domain alignment approaches.

Ablation study. To verify the contribution of the reconstruction network $G_{r}$ and MDAT, we discard the term $\mathcal{L}_{r}(\textbf{x}^{t})$ in Eq.4, and evaluate the method, denoted as ARN w.o. MDAT in Table 1. Comparing it with source-only model, we can infer the improvement of reconstructing target samples. ARN w.o. MDAT improves tasks with low-level domain shift such as MNIST$\leftrightarrow$USPS, which conforms with our discussion that unsupervised reconstruction is instrumental in learning pixel-level features. Comparing ARN w.o. MDAT with the original ARN, we can infer the contribution of MDAT. Table 1 shows that the MDAT achieves an impressive margin-of-improvement. For USPS$\to$MNIST and SVHN$\to$MNIST, the MDAT improves ARN w.o. MDAT by around 30%. It demonstrates that MDAT that helps learn domain-invariant features is the main reason for the tremendous improvement.

Toy dataset. We study the behavior of MDAT on a variant of inter-twinning moons 2D problem, where the target samples are rotated $30^{\circ}$ from the source samples. 300 samples are generated for each domain using scikit-learn (Pedregosa et al., 2011). The adaptation ability is investigated by comparing MDAT with DANN and source-only model. As shown in Figure 2, we visualize the changing boundaries during 10 epochs training. In Figure 2(a), the model is overfitting the source domain, and the decision boundary does not change. In Figure 2(b) and 2(c), both DANN and MDAT adapt the boundaries to the target samples, but MDAT shows faster and better adaptation during 10 epochs. Integrating the training procedure of SVHN$\to$MNIST in Figure 3(a), we justify that more effective gradients are provided by MDAT for better adaptation performance.

Gradients and stability analysis. We further study the training procedure of MDAT on SVHN$\to$MNIST w.r.t. loss and target accuracy in Figure 3(a) and 3(b), respectively. In Figure 3(a), ARN has steadily decreasing loss ($\mathcal{L}_{r}$) for all $\alpha$, but the domain loss in DAT ($\mathcal{L}_{d}$) becomes extremely small at the beginning. These observations conform with our intuition: the domain classifier in DAT is too strong to impede the adversarial training, while MDAT provides more effective gradients for training feature extractor by restricting the capacity of domain classifier. With effective gradients, the adversarial game is more balanced, which is validated in Figure 3(b) where the test accuracy of ARN is more stable than that of DAT across training epochs.

Parameter sensitivity. We investigate the sensitivity of $\alpha$ and $m$ on SVHN$\to$MNIST. In Table 4, the results show that ARN achieves good performance as $\alpha\in[0.01,0.1]$. Even with larger $\alpha$, ARN is able to achieve convergence. In comparison, denoting $\alpha$ as the weight of adversarial domain loss ($\mathcal{L}_{d}$), the DANN cannot converge when $\alpha>0.2$ due to the imbalanced adversarial game between the overwhelming domain classifier and the feature extractor. For the sensitivity of $m$, the accuracy of ARN exceeds 96.0% as $m\geq 1$. In Section 3.5, as $m\geq 1$, the decoder serves as a domain classifier. These analyses validate that ARN is more insensitive to the parameters than that of DANN. Even in the worst cases, ARN can always achieve convergence.

T-SNE embeddings. We analyze the performance of domain alignment for DANN (DAT) (Ganin & Lempitsky, 2015) and ARN (MDAT) by plotting T-SNE embeddings of the features z on the task SVHN$\to$MNIST. In Figure 4(a), the source-only model obtains diverse embeddings for each category but the domains are not aligned. In Figure 4(b), the DANN aligns two domains but the decision boundaries of the classifier are vague. In Figure 4(c), the proposed ARN effectively aligns two domains for all categories and the classifier boundaries are much clearer.

Interpreting adapted features via reconstruction. One of the key advantages of ARN is that by visualizing the reconstructed target images we can infer how the features are domain-invariant. We reconstruct the MDAT features of the target domain and visualize them in Table 5. It is observed that the target features are reconstructed to source-like images by the decoder $G_{r}$. As discussed before, intuitively, MDAT forces the target feature to mimic the source feature distribution, which conforms with the visualization. Similar to image-to-image translation, this indicates that our method conducts implicit feature-to-feature translation that transfers the target features to source-like features, and hence the features are domain-invariant.

We have presented all the results of the sensitivity study in Section 4.3, and now we show their detailed training procedures in Figure 5(a) and 5(b). It is observed that the accuracy increases when $\alpha$ drops or the margin $m$ increases. The reason is very simple: (1) when $\alpha$ is too large, it affects the effect of supervised training on source domain; (2) when the margin $m$ is small, the divergence between source and target domain (i.e. $\mathcal{H}\triangle\mathcal{H}$-distance) cannot be measured well.

Here we provide more visualization of the reconstructed images of target samples. In Figure 6, the target samples are shown in the left column while their corresponding reconstructed samples are shown in the right. We can see that for low-level domain shift such as MNIST$\leftrightarrow$USPS, the reconstructed target samples are very source-like while preserving their original shapes and skeletons. However, for larger domain shift in Figure 6(c) and 6(d), they are reconstructed to source-like same digits but simultaneously some noises are removed. Specifically, in Figure 6(d), we can see that one target sample (SVHN) may contain more than one digits that are noises for recognition. After reconstruction, only the right digits are reconstructed. Some target samples may suffer from terrible illumination conditions but their reconstructed digits are very clear, which is amazing.