Partial Domain Adaptation Using Graph Convolutional Networks

In partial domain adaptation, adopting standard domain adaptation algorithms, which learn all source and target samples with the same weight, cause performance degeneration by outliers, i.e., $x_{i}^{s}$ corresponding to $y_{i}^{s}\notin\left|C_{t}\right|$. Therefore, we introduce a graph partial domain adaptation (GPDA) network, which aims to reduce the learning impact of the outliers during training time. As illustrated in Fig. 2, we exploit data structure, forcing to align the distributions of the same category in a weight framework. $E$ is a feature extractor, and $F_{s}$ and $F_{t}$ are classifiers for source samples in the source label space and the common label space, respectively. Task-specific features are learned by classifiers in supervised learning as followed:

	$\displaystyle L_{S}(\theta_{{F}_{s}},\theta_{E})=\frac{1}{n_{s}}\sum_{x_{i}\in D_{s}}L_{y}(F_{s}(E(x_{i})),y_{i}),$		(1)
	$\displaystyle L_{T}(\theta_{{F}_{t}},\theta_{E})=\frac{1}{n_{s}}\sum_{x_{i}\in D_{s}}\gamma_{{y_{i}}}L_{y}(F_{t}(E(x_{i})),y_{i}),$		(2)

where $L_{y}$ is the cross entropy loss function, and $\gamma_{{y_{i}}}$ is a probability of a source label $y_{i}$ belonging to the target label space. Here, $\boldsymbol{\gamma}\in R^{\left|C_{s}\right|}$, which indicates the contribution of each source class, and $\boldsymbol{\gamma}$ can be calculated as follows:

$$\boldsymbol{\gamma}=\frac{1}{n_{t}}\sum_{x_{i}\in D_{t}}F_{s}(E(x_{i})).$$

(3)

For domain invariant features, we combine Graph Convolutional Networks $G$ with a domain classifier $D$. $D$ is trained to distinguish the source domain from the target domain, and simultaneously $G$ and $E$ are trained to confuse $D$. This loss function can be expressed as follows:

$$\begin{split}L_{D}(\theta_{E},\theta_{G},\theta_{D})=&-\frac{1}{n_{s}}\sum_{x_{i}\in D_{s}}\gamma_{{y_{i}}}L_{bce}(D(G(E(x_{i}),A),d_{i}))\\ &-\frac{1}{n_{t}}\sum_{x_{i}\in D_{t}}L_{bce}(D(G(E(x_{i}),A),d_{i})),\end{split}$$

(4)

where $L_{bce}$ is the binary cross entropy loss and $d_{i}\in\{0,1\}$ indicates the domain label. Moreover, a label relational graph $A$ and moving average centroid separation $L_{CS}$ lead to aligning the distributions of same category in the source and target domains, which are described in section 2.2 and section 2.3, respectively. The total objective function is as follows:

$\displaystyle L_{GPDA}=L_{S}+L_{T}+\lambda_{1}L_{D}+\lambda_{2}L_{CS},$

(5)

where $\lambda_{1}$ and $\lambda_{2}$ are the trade-off parameters, and we set trade-off parameters to $\lambda_{1}=1.0$ and $\lambda_{2}=1.0$ for all experiments. Finally, the proposed GPDA network can be solved by a minimax optimization problem as follows:

$\displaystyle\begin{aligned} (\hat{\theta}_{E},\hat{\theta}_{G},\hat{\theta}_{{F}_{s}},\hat{\theta}_{{F}_{t}})&=\underset{\theta_{E},\theta_{G},\theta_{{F}_{s}},\theta_{{F}_{t}}}{\operatorname{arg}\,\operatorname{min}}\;L_{GPDA},\\ \hat{\theta}_{D}&=\underset{\theta_{D}}{\operatorname{arg}\,\operatorname{max}}\;L_{GPDA}.\end{aligned}$

(6)

Graph Convolutional Networks (GCN)[18] are motivated by a first-order approximation of localized spectral filters on graphs  [19]. Each GCN layer with $N$ nodes is described as followed:

$$Z={\tilde{D}}^{-\frac{1}{2}}\tilde{A}{\tilde{D}}^{-\frac{1}{2}}X\Theta,$$

(7)

where $X\in{R}^{N\times F}$ is a $F$-dimensional node signal matrix, and $\Theta\in{R}^{F\times F^{\prime}}$ is a learnable filter changing a node signal to $F^{\prime}$ dimension with $\tilde{A}=A+I_{N}$ and $\tilde{D}_{ii}=\sum_{j}\tilde{A}_{ij}$.

GCN outperforms in tasks on datasets with defined node-to-node relationships [20], and has recently been used in computer vision. Unlike datasets such as Citeseer, Cora, and Pubmed [20], which are datasets with predefined nodes and edges, it is important to determine nodes and edges appropriately for tasks in computer vision. To adopt Graph Convolutional Networks for computer vision tasks, Chen et al. [21] use the graph by setting nodes to word embedding vectors and edges to class co-occurrence patterns within the dataset for multi-label image classification. In addition, for group action recognition, Wu et al. [22] set nodes to feature maps of people and edges to appearance relation. However, since there is no graph for partial domain adaptation, we construct a label relational graph to use Graph Convolutional Networks in partial domain adaptation.

In the label relational graph, each node feature represents the feature map of a sample, then the ith node feature of the graph $X_{i}$ is obtained by:

$$X_{i}=E(x_{i}),$$

(8)

where $E$ and $x_{i}$ indicate the feature extractor and the ith input image, respectively. An adjacency matrix $A$ contains relationships of the nodes, i.e., edges, and each component of $A$ is obtained as follows:

$${A}_{ij}=\sum_{c=0}^{C-1}{y_{i,c}y_{j,c}},$$

(9)

where $C$ is the number of class, and $y_{i,c}$ and $y_{j,c}$ are labels of the ith and jth image, respectively. In the case of the source images, there are corresponding ground truth labels, which are one-hot vectors, but in the case of the target images, labels are not given in training. We exploit pseudo-labels [23], which are well known for semi-supervised learning techniques. Finally, the label relational graph has a large value between images that are likely to have the same class and has a low value for unrelated images. Layerwise propagation of the GCN layer with the label relational graph smooths features of images with the same class, which leads to aligning the distributions of the same class.

Graph convolution may lead to smoothing the features of different classes because of incorrect pseudo labels. To alleviate this smoothing effect, we introduce moving average centroid separation, which follows the idea in [25]. Specifically, we use features of the labeled source samples and pseudo-labeled target samples. In [25], they design the class centroid alignment module to map features in the same class nearby, while we introduce moving average centroid separation to map features in the different classes separately. The moving average centroid separation objective function is as follows:

$$L_{CS}(\theta_{E},\theta_{G})=-\sum_{k=0}^{C-1}\parallel c^{s}_{k}-c^{t}_{(k+i)\,mod\,C}\parallel^{2},$$

(10)

where $c^{s}_{k}$ and $c^{t}_{k}$ are centroids of feature maps of class $k$ in the source and target domains, respectively, and $i$ is a random integer number within [1, $C-1$], updated in each iteration. Through the objective function, false signals in pseudo-labeled target samples are suppressed, and features in the different classes are explicitly separated from each other.

We evaluate our architecture to compare with state-of-the-art networks for partial domain adaptation on two benchmark datasets: Digit [15, 16] and Office-31 [17].

Digit. We utilize MNIST and USPS for two domain adaptation tasks (i.e., MNIST $\rightarrow$ USPS and USPS $\rightarrow$ MNIST). MNIST and USPS consist of 10 images containing numbers from 0 to 9, but domains are different. MNIST is collected from students, whereas USPS is a dataset for US portal service. In the PDA setting, we use all images and labels as the source dataset, and adopt images corresponding to the first five classes as the target dataset as conducted in [11] (i.e., $\left|C_{s}\right|=10$, $\left|C_{t}\right|=5$).

Office-31. Office-31 is a standard benchmark for domain adaptation. It consists of 4,652 images and 31 categories collected from three different domains. Amazon (A) contains images from amazon.com. DSLR (D) and Webcam (W) are taken by a DSLR camera and a web camera, respectively. We utilize the Office-31 dataset for six domain adaptation tasks. We use all images and labels as the source dataset, and adopt images corresponding to the ten classes as the target dataset as conducted in [8] (i.e., $\left|C_{s}\right|=31$, $\left|C_{t}\right|=10$).

We implement our GPDA network based on Pytorch, and exploit CNN architectures for the Digit dataset, as the same protocol in DANN [1]. For the office-31 dataset, we finetune ResNet-50 [24] pre-trained on ImageNet [26]. For a fair comparison, we use the same base network for previous methods. We add two GCN layers with 256 and 1024 channels on Digit and Office-31, respectively, since increasing the number of layers or channels for GCN layers do not improve performance. New layers are trained from scratch with 10 times the learning rate of the pre-trained layer. We use SGD with the momentum of 0.9 and the learning rate decay strategy implemented in DANN [1], and the learning rates of all new layers are increased gradually from 0 to 1 as DANN [1] also.

We show that our GPDA network outperforms previous methods on the Digit dataset in Table 1. Source only and DANN are the methods without the domain adaptation algorithm and with the standard domain adaptation algorithm, respectively. In the situation where the label spaces of the source and target are different, the performance of DANN is lower than the way without using the domain adaptation method, i.e., source only. This is the result of the outliers interfering with distribution alignment. Our method achieves state-of-the-art performance compared to previous partial domain adaptation methods [9, 11].

In Table 2, the proposed method achieves state-of-the-art performance with average gain of 1% beyond ETN [10], which adds the novel domain classifier. Specifically, our network has the same algorithm for reducing the learning impact of outliers as PADA [6], but shows a performance improvement of 4.6% on average, exploiting the newly introduced a label relational graph and moving average centroid separation. It shows that using data structure and considering the distribution of each category are valid for partial domain adaptation. Moreover, we experiment with ablation studies to examine the effect of each module. Baseline indicates the network without a label relational graph and moving average centroid separation. It outperforms PADA by exploiting the additional classifier expected to learn images in the common label space. We experiment our GPDA network without a label relational graph and moving average centroid separation, respectively. Ours w/o $L_{CS}$ only leads to align the distributions of the same category, whereas ours w/o graph separate the distributions of different categories. They provide higher performances than baseline, meaning that each module is effective for PDA, and we can see that two modules work complement each other to obtain better overall results.