Spatial Attention Pyramid Network for Unsupervised Domain Adaptation

Over the past few years, deep neural network (DNN) significantly pushed forward the state-of-the-art in several tasks in computer vision field, such as object detection [31, 44], instance segmentation [11, 10], and semantic segmentation [27, 2]. Notably, the DNN-based methods rely on large-scale annotated training data, which is difficult to cover diverse application domains. That is, the feature distributions (e.g., local texture, object appearance and global structure) between source domain and target domain are dissimilar or even completely different. To avoid expensive and time-consuming human annotation, unsupervised domain adaptation is proposed to learn discriminative cross-domain representation in such domain shift circumstance [30].

Most of previous methods [16, 26, 15] attempt to globally align the entire distributions between the source and target domains. However, it is challenging to generate a unified adaptation function for various scene layouts and appearance variation of different objects. Recent methods focus on transferring texture and color statistics within object instances or local patches. To deal with domain adaptation in the object detection and instance segmentation tasks, the basic idea in [4, 45] is to exploit discriminative features in bounding boxes of objects and attempt to align them across both source and target domains. However, the context information around the objects is not fully exploited, causing inferior results in some scenarios. Meanwhile, some domain adaptation methods for semantic segmentation [40, 28] enforce the semantic consistency between the pixels or local patches of the two domains, leading to deficiencies of critical information from object-level patterns. To that end, recent methods [34, 36] concatenate global context feature and instance-level feature for distribution alignment, and optimize the model based on several loss terms for global-level and local-level features with pre-set weights. However, this method fails to exploit the context information of objects, which is not optimal in challenging scenarios.

In this paper, we design the spatial attention pyramid network (SAPNet) to solve unsupervised domain adaptation for object detection, instance segmentation, and semantic segmentation. Inspired by spatial pyramid pooling [12], we construct the spatial pyramid representation with multi-scale feature maps, which integrates full of holistic (image-level) and local (regions of interest) semantic information. Meanwhile, we design a task-specific guided spatial attention mechanism to capture multi-scale context information. In this way, discriminative semantic regions are attended in a soft manner to extract features for adversarial learning. Extensive experiments are conducted on various challenging domain-shift datasets, such as Cityscapes [5] to FoggyCityscapes [35], PASCAL VOC [7] to Clipart [18], and GTA5 [32] to Cityscapes [5]. It is worth mentioning that the proposed method surpasses the state-of-the-art methods on various tasks, i.e., object detection, instance segmentation, and semantic segmentation. For example, our SAPNet improves from Cityscapes [5] to FoggyCityscapes [35] by $3\%$ mAP in terms of object detection, and achieves comparable accuracy from GTA5 [32] to Cityscapes [5] in terms of semantic segmentation.

Contributions

. 1) We propose a new spatial attention pyramid network to solve the unsupervised domain adaptation task for object detection, instance segmentation, and semantic segmentation. 2) We develop a task-specific guided spatial attention pyramid learning strategy to merge multi-level semantic information in feature maps of the pyramid. 3) Extensive experiments conducted on various challenging domain-shift datasets for object detection, instance segmentation, and semantic segmentation, demonstrate the effectiveness of the proposed method, surpassing the state-of-the-art methods.

We design a spatial attention pyramid network (SAPNet) to solve various computer vision tasks, such as object detection, instance segmentation, and semantic segmentation. First of all, we define the labeled source domain $\mathcal{D}_{s}$ and the unlabeled target domain $\mathcal{D}_{t}$, which are subject to the complex and unknown distributions in the source and target domains. Our goal is to find discriminative representation for the distributions of both source and target domains to capture various semantic information and local patterns in different tasks. The architecture of SAPNet is presented in Fig. 1.

Spatial pyramid representation.

According to [12, 22], spatial pyramid pooling can maintain spatial information by pooling in local spatial bins. To better adapt source and target domains, we develop a spatial pyramid representation to exploit the underlying distribution within an image.

Specifically, as shown in Fig. 1, the feature map $\hat{f}\in\mathbb{R}^{\hat{C}\times H\times W}$ is extracted from the backbone $G$ of the network, where $\hat{C}$, $H$ and $W$ are the channel dimension, height, width of feature maps respectively. To improve efficiency, we first reduce the number of channels in $\hat{f}$ to $\bar{f}\in\mathbb{R}^{C\times H\times W}$ gradually by using three $1\times 1$ convolutional layers, i.e., we set $C=256$ in all our experiments. Second, we use multiple average pooling layers with different sizes to operate upon the feature map $\bar{f}$ separately. The sizes of the pooling operation are $\{k^{n}\}_{n=1}^{N}$, where $N$ is the number of pooling layers. That is, the rectangular pooling region with the size $k^{n}$ at each location of $\bar{f}$ is down-sampled to the average value of each region, resulting in the pyramid of $N$ pooled feature maps $\{f^{1},\dots,f^{N}\}$. In this way, every pooled feature map $f^{n}\in\mathbb{R}^{C\times H_{n}\times W_{n}}$ in the pyramid can encode different semantic information of objects or layouts within the image.

It is worth mentioning that the proposed spatial pyramid representation is related with spatial pyramid pooling (SPP) for visual recognition [12]. While they share the pooling concept, we would like to highlight two important differences. First, we use average pooling instead of max pooling to construct the spatial pyramid representation. It can better capture the overall strength of local patterns such as edges and textures, which is demonstrated in the ablation study. Second, SPP pools the features with just a few window sizes and concatenates them to generate fixed-length representation; while SAP is designed to capture multi-scale context information of all levels in the pyramid. Thus, it is difficult to use a large number of windows with different sizes for SPP due to computational complexity.

Attention mechanism.

Moreover, we integrate the spatial attention pyramid strategy to enforce the network to focus on the most discriminative semantic regions and feature maps. There are mainly two advantages of introducing the attention mechanism in the spatial pyramid architecture. First, there exist different local patterns in each spatial location of feature maps. Second, different feature maps in the pyramid have different contributions to the semantic representation. The detailed learning method is described in two aspects as follows.

To facilitate highlighting the most discriminative semantic regions, the spatial attention masks for the pyramid $\{f^{1},\dots,f^{N}\}$ are learned based on the guided information from the task-specific heads (i.e., object detection, instance segmentation and semantic segmentation). For object detection and instance segmentation, the guided information is the output map with the size of $A\times H\times W$ from the classification head of region proposal network (RPN). It can predict object’s confidence in terms of the locations in feature maps for all the number of anchors $A$. That is, it encodes the distribution of objects which is suitable for object detection and instance segmentation problem. For semantic segmentation, the guided information is the output map with the size of $C_{sem}\times H\times W$ from the segmentation head, where $C_{sem}$ is the number of semantic categories. We denote the guided map as $\mathcal{\hat{P}}$.

Then, we concatenate the guided map $\mathcal{\hat{P}}$ and feature map $\hat{f}$ to generate guided feature map $\mathcal{\bar{P}}$ followed by $3$ convolutional layers. The guided feature map $\mathcal{\bar{P}}$ is shared for all $N$ scales. To adjust each scale of feature map $f^{n}$ in the spatial pyramid, we resize $\mathcal{\bar{P}}$ to the size $C\times H_{n}\times W_{n}$ at each level. The spatial attention mask $\mathcal{P}^{n}\in\mathbb{R}^{H_{n}\times W_{n}}$ can be predicted by the followed $3$ convolutional layers. Finally, $\mathcal{P}^{n}$ is normalized using the softmax function to compute the spatial attention, i.e.,

where $\omega^{n}(i,j)$ indicates the value of the attention mask $\omega^{n}$ at $(i,j)$. Thus, we have $\sum_{i=1}^{H_{n}}\sum_{j=1}^{W_{n}}\omega^{n}(i,j)=1$. As shown in Fig. 2, we provide some examples with normalized attention masks $\omega^{n}$ for different feature maps in the spatial pyramid when $N=7$. We can conclude that the feature map with different scale $k^{n}$ focuses on different semantic regions. For example, in the forth row, the feature map with smaller pooling size ($k=3$) pays more attention on the seagull, while the feature map with larger pooling size ($k=21$) focuses on the sail boat and the neighbouring context. Based on different guided information, $\omega^{n}$ recalibrates spatial responses in feature map $f^{n}$ adaptively.

On the other hand, it can be seen that not all the attention masks correspond to meaningful regions (see the attention mask with pooling size $k=37$ in the forth row). Inspired by [23], we develop a dynamic weight selection mechanism to adjust the channel-wise weight of feature maps in the pyramid adaptively. To consider feature maps with different size, we normalize $f^{n}$ to an attention vector $V^{n}\in\mathbb{R}^{C\times 1}$ using the corresponding spatial attention weight $\omega^{n}$ as:

where $i$ and $j$ enumerate all spatial positions of weighted feature map $f^{n}\cdot\omega^{n}$. Thus the attention vectors have the same size for all the feature maps in the pyramid. Given attention vectors $\{V^{1},\dots,V^{N}\}$, we first fuse these vectors via an element-wise addition, i.e., $v=\sum_{n=1}^{N}V^{n}$. Then, a compact feature ${\bf z}\in\mathbb{R}^{d\times 1}$ is created to enable the guidance for adaptive selections by the batch normalization layer, where $d$ is the dimension of the compact feature ${\bf z}$, and we set it to $\frac{C}{2}$ in all experiments. After that, for each attention vector $V^{n}$, we compute the channel-wise attention weight $\phi^{n}\in\mathbb{R}^{C\times 1}$ as

where $\{a_{i},\dots,a_{N}\}$ are learnable parameters of fully connected layers for each scale. We have $\sum_{c=1}^{N}{\phi^{n}(c)}=1$, where $\phi^{n}(c)$ is the $c$-th element of $\phi^{n}$. In Fig. 2, we show the corresponding weights of each feature map in the spatial pyramid. Specifically, we compute the mean of channel-wise attention weight $\phi^{n}$ for each scale in each image. Finally, the fused semantic vector ${\cal V}\in\mathbb{R}^{C\times 1}$ is obtained through the channel-wise attention weight as ${\cal V}=\sum_{n=1}^{N}{V^{n}\cdot\phi^{n}}$, where ${\cal V}$ is a highly embedded vector in the latent space that encodes the semantic information of different spatial locations, different channels and the relations among them.

Optimization.

The whole network is trained by minimizing two loss terms, i.e., adversarial loss and task-specific loss. The adversarial loss is used to determine whether the sample comes from the source domain or target domain. Specifically, we calculate the probability $x_{i}$ of the sample belonging to the target domain based on the fused semantic vector ${\cal V}$ using a simple fully-connected layer. The proposed SAPNet is denoted as $D$. Then, the adversarial loss is computed as

where $y_{i}$ is the domain label ($0$ for source domain and $1$ for target domain) and $\mathcal{H}$ is the cross-entropy loss function. On the other hand, the task loss $\mathcal{L}_{\text{task}}$ is determined by the specific task, i.e., object detection, instance segmentation, and semantic segmentation. The loss is computed as

where $G$ and $R$ are the backbone and task-specific components of the network, respectively. $y_{i}^{s}$ is the ground-truth label of sample $i$ in the source domain. We have $\mathcal{L}_{\text{task-specific}}=\{\mathcal{L}_{\text{det}},\mathcal{L}_{\text{ins}},\mathcal{L}_{\text{seg}}\}$. Taking object detection as an example, we denote the objective of Faster R-CNN as $\mathcal{L}_{\text{det}}$, which contains classification loss of object categories and regression loss of object bounding boxes. In summary, the overall objective is formulated as

where $\lambda$ controls the trade-off between task-specific loss and adversarial training loss. Following [4, 34], we use gradient reverse layer (GRL) [8] to enable adversarial training where the gradient is reversed before back-propagating to $G$ from $D$. We first train the networks with only source domain to avoid initially noisy predictions. Then we train the whole model with Adam optimizer and the initial learning rate is set to $10^{-5}$, then divided by $10$ at $70,000$, $80,000$ iterations. The total number of training iterations is $90,000$.

We implement our SAPNet method with PyTorch [29], which is evaluated in three domain adaptation tasks, including object detection, instance segmentation, and semantic segmentation. For fair comparison, we set the shorter side of the image to $600$ following the implementation of [34, 36] with RoIAlign [11] in object detection; for instance segmentation and semantic segmentation, we use the same settings as previous methods. To consider the trade-off between accuracy and complexity, the number of pyramid levels is set to $N=13$ for object detection and instance segmentation, i.e., we have the spatial pooling size set $K=\{3,6,9,12,15,18,21,24,27,30,33,35,37\}$. Note that we start from the initial pooling size $3\times 3$ with the step of $3$, and the last two pooling sizes are reduced from $\{38,41\}$ to $\{35,37\}$ because of the width limit of feature map. For semantic segmentation, the number of pyramid levels is set to $N=9$ since semantic segmentation involves feature maps with higher resolution, i.e., $K=\{3,9,15,21,27,33,39,45,51\}$. The hyper-parameter $\lambda$ is used to control the adaptation between source and target domains. Thus, we use different $\lambda$ in different tasks. Empirically, we set a larger $\lambda$ for adaptation between similar domains (e.g., Cityscapes$\to$FoggyCityscapes), and set a smaller $\lambda$ for adaptation between dissimilar domains (e.g., PASCAL VOC$\to$WaterColor). We choose $\lambda$ based on the performance on the validation set.

In this work, we propose a general unsupervised domain adaptation framework for various computer vision tasks including object detection, instance segmentation and semantic segmentation. Given target-specific guided information, our method can make full use of feature maps in the spatial attention pyramid, which enforces the network to focus on the most discriminative semantic regions for domain adaptation. Extensive experiments conducted on various challenging domain adaptation datasets demonstrate the effectiveness of the proposed, which performs favorably against the state-of-the-art methods.

This work was supported by the Key Research Program of Frontier Sciences, CAS, Grant No. ZDBS-LY-JSC038, the National Natural Science Foundation of China, Grant No. 61807033 and National Key Research and Development Program of China (2017YFB0801900). Libo Zhang was supported by Youth Innovation Promotion Association, CAS (2020111), and Outstanding Youth Scientist Project of ISCAS.