Domain Adaptive Object Detection via Feature Separation and Alignment

As is known to all, domain adversarial loss can reduce the inconsistency between the features distribution of the two domains. However, there will still be some distractive information that cannot be reduced, \eg, the background distractive information of the source/target domain. So we propose a GSFS module based on a dual-stream network, and employ it to separate the distractive/shared information which is useless/useful for object detection with the high-level features through the private encoder $E_{s}$ and $E_{t}$, the shared decoder $S_{D}$ and the features extraction module $F_{\tau}$, as shown in Figure 1. Firstly, $F_{3}$ extracts high-level features related to object detection via the detection loss. Then domain adversarial loss of LGFA module makes the features $f_{\tau}$ of the source/target domain aligned. As a result, $f_{3}$ is the shared high-level features useful for detection. For the GSFS module, we define the different loss to limit the similarity of $\left\{d^{s},f_{3}^{s}\right\}$ and $\left\{d^{t},f_{3}^{t}\right\}$:

where $\lVert\cdot\rVert_{F}^{2}$ is the squared Frobenius norm , $g(\cdot)$ is the global pooling layer, $d_{i}^{s}$, $d_{i}^{t}$, $f_{3,i}^{s}$, $f_{3,i}^{t}$ denote distractive information output by private encoder $E_{s}$/$E_{t}$ and the shared information $f_{3}$ of source/target domain for $i$th image, respectively.

In addition, considering that simple restrictions will make $d^{s}$ and $d^{t}$ meaningless, and the information related to objects cannot include all the information of the input image, we use the fusion features $f_{\mathit{fus}}^{s}\!=\![d^{s},f_{3}^{s}]$, $f_{\mathit{fus}}^{t}\!=\![d^{t},f_{3}^{t}]$ as the input of the shared decoder $S_{D}$ to reconstruct the input image, where $[\cdot,\cdot]$ denotes the concatenation of two feature maps in the channel dimension. Through the restriction of image reconstruction, the separated features can possess almost all the information of source/target images. So that the private encoder extracts the distractive information irrelevant to detection in the image. Notably, the private encoder inputs two-domain images in gray-scale, and extract the distractive information which is irrelevant to the color of the two-domain images. If the input is an RGB image, it will not only increase the difficulty of reconstruction, but also make the private encoder disturb by the color difference of the image, while ignoring other distractive information on the two data domains.

Finally, the reconstruction loss of the gray-scale images is defined as:

where $\lVert\cdot\rVert_{1}^{1}$ is the $\ell_{1}$-norm, ${x}_{g,i}^{s}$, ${x}_{g,i}^{t}$, $\hat{x}_{g,i}^{s}$ and $\hat{x}_{g,i}^{t}$ are the $i$th gray-scale input images and reconstructed images by the shared decoder $S_{D}$ in source/target domain.

The RILA module consists of a grouping component and a context-aware RILA component. Please refer to supplemental material to see illustration and more details.
Grouping. Some alignment module [5, 16, 4] implement the instance-level feature alignment of all region proposals. However, the absence of labels on the target domain leads to the training of RPN mainly guided by the labeled data on the source domain [31]. Therefore, we cannot ensure that the region proposals on the target domain contain the objects, where may contain a lot of background noise. Meanwhile, objects are generally detected by multiple region proposals, which means that some region proposals are redundant. We should perform feature alignments for the instance-level features in the region proposals where the object is most likely to exist, instead of aligning all features. As we all know, the locations of region proposals where contain the same object at multiple scales should be similar if they are predicted by a reliable RPN. A natural idea is to cluster the similar region proposal coordinates to reduce redundancy for ROIs and eliminate the background noises by excluding outliers.

In particular, we use RPN to get the predicted bounding box $\{b_{i,x},b_{i,y},w_{i},h_{i}\}_{i=1}^{N}$ of the region proposals, where $(b_{i,x},b_{i,y})$ is the center coordinate of the $i$th predicted bounding box, $w_{i}$ is its width, and $h_{i}$ is its height. Then we plan to exploit the scale-space filtering (SSF) [37] algorithm to cluster the center coordinates to adaptively obtain $K$ cluster centers, which means that ROIs can be divided into $K$ categories, and thereby, the instance-level features can be indirectly divided into $K$ categories. As a result, we only need to align the instance-level features in each category.
Adaptive candidate region searching. SSF is a clustering method that adaptive to the category numbers without a manual setting. It is very suitable for the grouping module due to the unknown object numbers in the target domain. Therefore, inspired by [37], we decided to use SSF to improve the traditional K-means algorithm to accomplish the clustering of bounding boxes. By regarding each sample as a “light point”, the SSF algorithm formulates the clustering issue as finding the “lighting blob center” under different blur scales, \ie, different scale-space maps. The specific workflow can be divided into the following steps:
(1) Clustering center iteration. For a given dataset $\mathcal{X}=\left\{x_{i}\in\mathbb{R}^{2}:i=1,\ldots,N\right\}$ (in this paper, this is a set of center coordinates of predicted bounding boxes by RPN), we define $P(x,\sigma_{j})$ as the scale-space map for $\mathcal{X}$ under the blur scale $\sigma_{j}$. It can be calculated by $P(x,\sigma_{j})=p(x)*\phi(x,\sigma_{j})$, where $p(x)$ means the scatter image of data samples in $\mathcal{X}$, $\phi(\cdot,\sigma_{j})$ is the Gaussian blur kernel with the blur scale $\sigma_{j}$, and $*$ represents the convolution operator. So the set of clustering centers $\mathcal{C}(\sigma_{j})$ under different scales $\sigma_{j}$ is obtained by $\nabla_{x}P(x,\sigma_{j})=0$. In iteration steps, $\sigma_{j}$ is firstly updated, then after $\mathcal{C}(\sigma_{j})$ has converged, a larger $\sigma_{j+1}$ is obtained by $\sigma_{j+1}\!=\!k\sigma_{j}$ ($k$ is a hyperparameter and $k\!>\!1$), and $\mathcal{C}(\sigma_{j+1})$ are iterated again until all samples belong to one category. More details can be seen in the supplemental material.

(2) Adaptive category number selection. With the completion of the above iterations, we have obtained a serious of clustering center, \ie, $\{\mathcal{C}(\sigma_{1}),\cdots,\mathcal{C}(\sigma_{j}),\cdots,\mathcal{C}(\sigma_{J})\}$. We hope to select the clustering model that best fit for the data distribution. So the “lifetime” [37] is used as a measure of the stability for category number, which is calculated by

where $\varepsilon=0.01,c=1/\log(1.05)$. Notably, the interval $\mathbf{\Sigma}=\left\{\sigma_{J_{inf}},\sigma_{J_{inf}+1},\dots,\sigma_{J_{sup}}\right\}$ with the longest “lifetime” represent the relative appropriate scales. Finally, the most appropriate clustering model $\{\mathcal{C}(\sigma^{*}),\sigma^{*}\}$ equals to median of $\mathbf{\Sigma}$, and the adaptive category number $K$ equals to the element number of $\mathcal{C}(\sigma^{*})$.

(3) Remove outliers. After obtaining the optimal $\{\mathcal{C}(\sigma^{*}),\sigma^{*}\}$, we can use the model to group the centers $\{(b_{i,x},b_{i,y})\}_{i=1}^{N}$ of the predicted bounding box of all region proposals. For each bounding box center $b_{i}=(b_{i,x},b_{i,y})$, if $\lVert b_{i}-c_{k}^{*}\rVert_{2}>\sigma^{*}$, we consider $b_{i}$ to be an outlier, and its corresponding region proposal probably contains the background noise, which should be excluded from grouping. Through this method, we can restrict certain outlier region proposals from participating in instance-level alignment, and find the instance features that need to be aligned in the image more robustly.
Context-Aware Region-Instance-Level Alignment. After clustering the proposals, we need to refine the instance-level features of each category for instance alignment. For simplicity, we reshape the features of instances belonging to the same category into the same size and then concatenate them into a feature map as the input of the domain classifier. But the effect of redundant or noisy region proposals is not fundamentally solved. Therefore, we use the global pooling layer to refine the instance-level features in the same category. Let $\Theta_{k}\in\mathbb{R}^{m_{k}\times d}$ denote the instance-level features of the $k$th category obtained by SSF, where $m_{k}$ is the number of the $k$th category , $d$ is the dimension of the instance-level features. Through the global pooling layer, we can get the feature of the $k$th category $\hat{\Theta}_{k}\in\mathbb{R}^{1\times d}$, which can fully reflect the instance-level features of the $k$ category and thereby reduce the influence of redundancy and noise.

Then, we use the context fusion instance-level features [32] as the input of bounding box regression and classification prediction in the instance alignment and object detection module. We acquire three different levels of context vectors $f_{l},f_{m},f_{g}$ from the domain classifier $D_{l},D_{m},D_{g}$, and get the instance-level features $f_{r}$ of each region proposals based on ROI-Pooling [13]. Through feature concatenation, the fused context instance-level features $f_{ins}=[f_{l},f_{m},f_{g},f_{r}]$ is input into the global pooling layer and obtain refined feature of each category $f_{ins_{e}}$, as shown in Figure 1.

Subsequently, we input the global pooling instance-level features into the domain classifier $G_{ri}$, and output the domain labels to achieve regional instance alignment. For the loss function of this module, we use Focal Loss [23, 32] as follows:

where $K_{i}^{s}$ is the number of adaptive categories for $i$th source sample, $f_{\mathit{ins_{e},k}}^{s}$ is the instance-level features of the $k$th region of the source domain after global pooling, and $\gamma$ is the parameter of Focal Loss.

In short, this module can solve the redundancy of the region proposals and the influence of the background noise, so that the instance alignment is focused on the features where the object is existing.

The loss of object detection includes the classification loss $\mathcal{L}_{c}$ on the source domain and the regression loss $\mathcal{L}_{r}$ of the bounding boxes [31]. For the LGFA module, we use the Local Feature Masks and IWAT-I frameworks mentioned in [4], and the corresponding adversarial loss is $\mathcal{L}_{lg}\!=\!\mathcal{L}_{adv_{1}}\!+\!\mathcal{L}_{adv_{2}}\!+\!\mathcal{L}_{adv_{3}}$ in Figure 1. By combining all modules mentioned above, the overall objective function of the model is

where $D,F,E,S$ denote the domain classifier, object detection framework, private encoder and shared decoder, respectively. $\lambda$ and $\beta$ are turning parameters. The sign of gradients for the last item is flipped by a gradient reversal layer proposed by [9].

In all experiments, the object detection model follows the settings of [32, 40, 4], using Faster R-CNN [31] with ROI-alignment [13], where the shorter side of the images equal to 600 and the hyper-parameters in the object detection are set based on the setting of [31]. For different dataset, the backbone network utilize VGG-16 [34] or ResNet-101 [15] with the parameter initialization weights following the pre-trained on ImageNet [7]. A stochastic gradient descent training model with a momentum of 0.9 is used, and the learning rate for the first 50K iterations is set to 0.001 and then decays to 0.0001. In each iteration, we input a pair of source and target domain images. After 70K iterations, we calculate the mean average precision (mAP) where the IoU threshold is 0.5. We set $\gamma\!=\!5$ in Eq. (4) and $\beta\!=\!0.1$ in Eq. (5) in all experiments. $\lambda$ is set to 1 for PASCAL $\to$ Clipart1k and Cityscapes $\to$ Foggy-Cityscapes while set to 0.1 for Sim10k $\to$ Cityscapes. Our experiments are implemented by the Pytorch [27].

We compare our FASNet with various SOTA DAOD methods, including Domain adaptive Faster-RCNN (DA-Faster) [5], Selective Cross-Domain Alignment (SCDA ) [40], Strong-Weak Distribution Alignment (SWDA) [32], Domain Diversification and Multi-domain-invariant Representation Learning (DDMRL) [20], Hierarchical Transferability Calibration Network (HTCN) [4], Prior-based Domain Adaptive Object Detection (PBDA [35]) and Asymmetric Tri-way Faster-RCNN (ATF) [17]. We cite the quantitative results in their original papers for comparison.

In this section, some ablation experiments are established to show the rationality for different components in FSANet. Furthermore, we contrast the effectiveness between RILA and traditional instance-level alignment module and explain the advantage of SSF in RILA compared with other different clustering methods.