Channel-wise Alignment for Adaptive Object Detection

Object detection is one of the most important topic in the field of computer vision. In recent years, most detection networks can be gathered into two categories, that is one-stage and two-stage.

One-stage detectors do not need Region Proposal Network (RPN) to generate a series of candidate regions. They solve the object classification and box regression problem with only just one single network, such as YOLO (Redmon et al. 2016; Redmon and Farhadi 2017, 2018), SSD (Liu et al. 2016) and its variants (Zhu et al. 2020; Wang, Zhu, and Lin 2020).

Two-stage detectors generate a number of proposals using RPN in the first stage, and then fine-tune these proposals using another network in the second stage. Though one-stage detectors reach higher speed than two-stage detectors, many state-of-the-art frameworks still using two-stage detectors, which have higher performance than one-stage detectors, e.g. R-CNN (Girshick et al. 2014), Fast R-CNN (Girshick 2015), Faster R-CNN (Ren et al. 2015), and Mask R-CNN (He et al. 2017), etc.

However, these state-of-the-art object detectors still face the same awkward challenge, which is relying heavily on the annotated datasets in specific scenes. For example, a state-of-the-art detector trained on public dataset like KITTI or Cityscapes, whose performance may be severely degraded when working in real world environment like autonomous driving system due to domain shift. We can solve this problem by collecting better labeled data from target domain, and then fine-tune the detector on these data, yet it is time-consuming and expensive, even the labeled data can’t be collected in some cases.

Domain adaptation (Ben-David et al. 2010; Mansour, Mohri, and Rostamizadeh 2009) has been well-focused in recent years, which is trying to adapt a model trained on one domain to another new domain. At present, a number of domain adaptation works focus on image classification (Duan, Tsang, and Xu 2012; Duan et al. 2011; Fernando et al. 2013; Gong et al. 2012; Gopalan, Li, and Chellappa 2011; Kulis, Saenko, and Darrell 2011) and semantic segmentation (Sun et al. 2019; Chen et al. 2019; Li, Yuan, and Vasconcelos 2019; Luo et al. 2019; Vu et al. 2019; Zou et al. 2018; Chen et al. 2020). In image classification, some representative methods are effective, including domain transfer multiple kernel learning (Duan, Tsang, and Xu 2012; Duan et al. 2011), asymmetric metric learning (Gong et al. 2012), subspace interpolation (Fernando et al. 2013), geodesic flow kernel (Gopalan, Li, and Chellappa 2011) and subspace alignment (Kulis, Saenko, and Darrell 2011). In semantic segmentation, Sun et al. (Sun et al. 2019) propose a cascade-weighted network to measure the similarity between synthetic pixels and real pixels, (Chen et al. 2019; Zhu et al. 2018) conduct an adversarial adaptation method based on pixel-wise level or design a loss function to align the domain. (Song et al. 2020) proposes three alignment modules to achieve the domain adaptation on stereo matching.

There have a number of state-of-the-art domain adaptation achievements, yet most of these methods are focus on image classification or semantic segmentation, thus application in the field of object detection is still in early stages (Shen et al. 2019). Chen et al. (Chen et al. 2019) propose to adapt domain gap with both image-level and instance-level feature alignment, furthermore, integrate gradient reverse layer (GRL) (Ganin et al. 2016) into Faster-RCNN framework to reduce the domain gap. Saito et al. (Saito et al. 2019) pay attention to low-level and high-level features, and tries to align them. Zhu et al. (Zhu et al. 2019) focus on clustering high relationship regions and aligning these regions.

It is obvious that previous works do not discover the high relationship between channels in feature map. In our work, we design channel-wise alignment method to reduce the gap of source domain and target domain which include two parts, i.e. self channel alignment and cross channel alignment. Furthermore, we add an an adversarial learning part on RPN sub-network, which aims at generating more robust region proposals. Note that our work can be trained in end-to-end fashion, and do not need any target domain annotations.

In the field of domain adaptation for detection, our target is to train a detector in one domain that can be adapted to another domain directly. To be more precise, we would like to learn the domain-invariant features by using the unsupervised method so that the learned model can perform well in both source and target domains. Inspired by the semantic coherence between channels and different objects in the image (as aforementioned), we propose two channel-wise alignment schemes and a new domain classifier on RPN that can drop into the Faster RCNN (Ren et al. 2015) smoothly. These three additional modules can be regarded as the alignment losses to help optimize the model to a better status during end-to-end training. When inference, they will be removed so there is no additional computational cost for testing.

As illustrated in Fig. 1, we observe that each channel is usually attentive to the particular semantic object (i.e., in each channel, the same class of objects will be prominent together), and different channels tend to focus on distinct objects, showing the diversity and informativeness across all channels. These phenomena drive us to make a reasonable assumption that each channel on the feature map has its own regions of interest. Motivated by this, we explore the inner information within channels and the mutual relationship cross channels by proposing the self channel alignment and cross channel alignment modules. Furthermore, we also notice that the coarse regions generated by RPN contribute significantly in the Faster R-CNN (Ren et al. 2015) detector. Thus, we further draw attention on the RPN to obtain a domain-invariant RPN module (by applying a domain classifier after RPN module) that can generate more robust coarse proposals.

The channel-wise alignment contains two parts: self channel-wise alignment module and cross channel-wise alignment module. The former one is used to align the domain gap within channel interior and the latter can reduce or alleviate the domain gap between different channels.

We observe that it can achieve better integration if we apply the proposed module to all the stages of a network, as shown in Fig. 2. Thus, the total loss of this module can be formulated as:

$$\mathcal{L}_{s}=\sum_{r}\mathcal{L}_{s}^{r}.$$

(2)

Here we would like to simultaneously optimize the parameters of the domain classifier to minimize the above domain classification loss, and also optimize the parameters of the base network to maximize this loss. Thanks to the gradient reverse layer (Ganin et al. 2016), we can train this module and the detection framework together through an end-to-end manner by inserting GRL to the beginning of each side branch.

During the training, we can obtain two decay gram matrix, i.e. $G_{dg}^{s}$ from the source domain and $G_{dg}^{t}$ from the target domain. We finally calculated the mean square error (MSE) loss between this two decay gram matrix, which can be formulated by  5:

$$\mathcal{L}_{c}=\frac{1}{|N|}\sum_{i}\sum_{j}\|G_{dg_{(i,j)}}^{s}-G_{dg_{(i,j)}}^{t}\|^{2},$$

(5)

where $N$ denotes the total number of elements of decay gram matrix $G_{dg}^{s}$ (or $G_{dg}^{t}$). The mechanism of cross channel-wise alignment module has been shown in Figure  2 and Figure  3.

Previous works of domain adaptation on object detection try to obtain a domain-invariant backbone network mainly through aligning domain gap in image-level and instance-level (Chen et al. 2018) or local-strong and global-weak level (Saito et al. 2019), or mining the relation of highly relative proposals (Zhu et al. 2019). Yet none of them pay attention to the robustness of RPN. We assume that if RPN module can be trained as much robust as possible, the proposals generated by RPN will be more domain-invariant, which can be fine-tuned later by Fast R-CNN (Ren et al. 2015) part. We also note that, RPN module has only two convolution layers, i.e. classification convolution layer and regression convolution layer. These two layers can also play the role of sliding window as mentioned by (Ren et al. 2015). Based on these findings, we do not use channel-wise alignment in RPN module, due to the function of RPN module does not obtain highly semantic feature, just generating proposals through sliding window mechanism instead. Thus, in this paper, we are trying to reduce the domain gap for the each active location on feature map that is fed into RPN. Besides, RPN generates the proposals exactly through every active location on feature map, so if we can reduce the domain gap of these active locations, we could improve the robustness of the whole model.

Motivated by above, we first integrate gradient reverse layer (Ganin et al. 2016) into RPN module. Then we add three convolution layers to reduce the output channels by the factor 2 for the purpose of reducing the computational cost of the later FC layer. Finally, we get a one-dimensional vector through FC layer. We call these three parts as RPN domain classifier. We use binary cross entropy loss as our optimize objective. The loss of RPN domain classifier denoted as equation 6:

$$\mathcal{L}_{r}=-\sum_{i,u,v}\left[D_{i}\log{p_{(u,v)}^{i}}+\left(1-D_{i}\right)\log{\left(1-p_{(u,v)}^{i}\right)}\right].$$

(6)

In equation 6, $i$ denotes the $i$-th image during training, $D_{i}$ denotes the domain label as mentioned above, $(u,v)$ denotes the active location on feature map which will be fed into domain classifier of RPN.

As mentioned above, we have proposed three domain adaptive parts which are integrated into the Faster R-CNN, thus we divided the whole architecture into two parts: detection part i.e. Faster R-CNN and domain adaptation part i.e. self channel-wise alignment module, cross channel-wise alignment module and RPN domain classifier.

Our total objective optimization $\mathcal{L}_{total}$ can be computed as the equation 7:

$$\mathcal{L}_{total}=\mathcal{L}_{det}+\lambda_{s}\mathcal{L}_{s}+\lambda_{c}\mathcal{L}_{c}+\lambda_{r}\mathcal{L}_{r},$$

(7)

where $\mathcal{L}_{det}$ is the Faster R-CNN loss function followed by (Ren et al. 2015), as is shown in equation 8:

$$\mathcal{L}_{det}=\mathcal{L}_{cls}+\mathcal{L}_{loc},$$

(8)

where $\mathcal{L}_{cls}$ is the cross-entropy loss function and $\mathcal{L}_{loc}$ denotes the smooth L1 loss function. Note that $\mathcal{L}_{det}$ only works for source domain because of there is no label for target domain. $\lambda_{s}$, $\lambda_{c}$ and $\lambda_{c}$ are trade-off parameter, which are all set to be 1 in our experiments.

During training, we resize the short size of 600 pixels to fit in GPU memory, set the batch size to 1 (i.e. one source image and one target image at every iteration). Note that, for KITTI (Geiger et al. 2013) dataset, we just remain the short size to 375 pixels, due to the image shape of KITTI (Geiger et al. 2013) is 375$\times$1250. We fine-tune the whole model with learning rate of 0.001 for 15 epochs and 0.0001 for later 10 epochs and weight decay of 5$\times$1e-4. Note that, for fair comparison, we report mean average precision (mAP) with a threshold of 0.5 for evaluation (i.e. AP50) follows (Chen et al. 2018).

We evaluate our proposed method on four datasets, including Cityscapes, Foggy-Cityscapes, KITTI and SIM-10k. We use these four datasets conduct four experiments. For normal-to-foggy domain adaptation, we experiment on Cityscapes and Foggy-Cityscapes. For cross-camera domain adaptation, we experiment on Cityscapes and KITTI. For synthetic-to-real domain adaptation, we experiment on Cityscapes and SIM-10k. For domain adaptation on instance segmentation task, we experiment on Cityscapes and Foggy-Cityscapes.

The detection framework we used in our experiment is Faster-RCNN. We compare our method with current state-of-the-art works (Zhu et al. 2019; Chen et al. 2018; Saito et al. 2019). We observe that some other methods are either under weakly supervised or class-specific settings, which cannot be directly compared here.

The results are illustrated in the Table 1. From this table, we can see that our results remarkably outperform the existing state-of-the-art method (Saito et al. 2019) with a large margin, i.e., about 5.5% improvement (39.8% v.s. 34.3%). This result demonstrates that our proposed method is more robust for object detection under different weather conditions.

In our experiment, we use SIM-10k as synthetic data and Cityscapes as real data. We only train the model on the Car category because of the limitation of provided bounding boxes. We first train our model on the labeled SIM-10k and label-free Cityscapes . Then we test on car class on Cityscapes dataset. The results are illustrated on Table 2. From the table, we can see that our result is better than SCDA (Zhu et al. 2019).

To give an in-depth analysis of our model, we conduct extensive ablation studies on normal-to-foggy setting. We first analyse the improvement of our all proposed modules. Then we analyse the improvement of different combination of our self channel-wise alignment module.

We first verify SCA, CCA and RDC independently, and find SCA has the highest improvement than the other two. It yields an improvement of 13.5% over the Source-only method. It indicates that our proposed self channel-wise method has the highest contribution. The CCA module also can improve 0.6% than the Source-only method. The RDC can improve 2.7%. Then we compose these three modules one with another and find SCA with CCA can reach to 39.3%, which is highest than SCA with RDC and CCA with RDC. It indicates that our sef channel-wise method and cross channel-wise method can work complementarily. It proves our assumption that the channel information can improve the total accuracy largely. Finally, we compose all the three modules and obtain the highest results. Besides, we also conduct the extensive experiments to demonstrate the effectiveness of our decay matrix, as mentioned in Section Channel-wise Alignment. The results show that our decay matrix indeed improve performance on all tasks, as is shown in Tabel 6.

In this section, we introduce our further research on domain adaptation for instance segmentation. Instance segmentation task not only has the annotated bounding boxes, but also has the labelled mask, it can be used for verifying the scalability of our proposed method.

We mainly use our SCA and CCA in Mask R-CNN for this experiment. The network’s backbone is also VGG16. The datasets we use for this experiment are Cityscapes and Foggy-Cityscapes . The reason why we use this two datasets for this experiment is that they both have annotated mask and bounding boxes. The domain adaptation is from Cityscapes to Foggy-Cityscapes . We train our model on the labelled Cityscapes and label-free Foggy-Cityscapes. Then we evaluate the model on labelled Foggy-Cityscapes. The detection threshold that we set for calculating mAP is 0.7.

The results are reported in Table 7. First, comparing Faster R-CNN with Source-only, we can observe that it improves about 0.6% by adding the mask branch. Comparing our results with Source-only, we can see our proposed method yields significant improvement, where the improvements are 3.8% on mAP (Box) and 12.1% on mAP (Mask). In addition, we also compare our results with current state-of-the-art method SCDA, it also improves about 1% on mAP (Box) and comparable mAP (Mask). Furthermore, we visualize the domain adaptation results and compare them with Source-only results in the supplementary materials.