Domain Adaptive Object Detection for Autonomous Driving under Foggy Weather

Recent advancement in deep learning has brought outstanding progress in autonomous driving [33, 6, 25, 53], and object detection has been one of the most active topic under this field [41, 59, 8, 45]. Regarding the network architecture, current object detection algorithms can be roughly split into two categories: two-stage methods and single-stage methods. Two-stage object detection algorithms typically compose of two processes: 1) region proposal, 2) object classification and localization refinement. R-CNN [14] is the first work for this kind of methods, it applies selective search for regional proposals and independent CNNs for each object prediction. Fast R-CNN [13] improves R-CNN by obtaining object features from the shared feature map learned by one CNN. Faster R-CNN [37] further enhances the framework by proposing Region Proposal Network (RPN) to replace the selective search stage. Single-stage object detection algorithms predict object bounding boxes and classes simultaneously in one same stage. These methods usually leverage pre-defined anchors to classify objects and regress bounding boxes, they are less time-consuming but less accurate compared to two-stage algorithms. Milestones for this category include SSD-series [29], YOLO-series [36] and RetinaNet [28]. Despite their success in clear-weather visual scenes, these object detection methods might not be employed in autonomous driving directly due to the complex real-world weather conditions.

In order to address the diverse weather conditions encountered in autonomous driving, many datasets have been generated [40, 31, 32, 34] and many methods have been proposed [22, 17, 35, 2, 42, 44, 18] in recent years. For example, Foggy Cityscape [40] is a synthetic dataset that applies fog simulation to Cityscape for scene understanding in foggy weather. TJU-DHD [32] is a diverse dataset for object detection in real-world scenarios which contains variances in terms of illumination, scene, weather and season. In this paper, we focus on the object detection problem in foggy weather. Huang et al. [22] propose a DSNet (Dual-Subnet Network) that involves a detection subnet and a restoration subnet. This network can be trained with multi-task learning by combining visibility enhancement task and object detection task, thus outperforms pure object detectors. Hahner et al. [17] develop a fog simulation approach to enhance existing real lidar dataset, and show this approach can be leveraged to improve current object detection methods in foggy weather. Qian et al. [35] propose a MVDNet (Multimodal Vehicle Detection Network) that takes advantage of lidar and radar signals to obtain proposals. Then the region-wise features from these two sensors are fused together to get final detection results. Bijelic et al. [2] develop a network that takes the data from four sensors as input: lidar, RGB camera, gated camera, and radar. This architecture uses entropy-steered adaptive deep fusion to get fused feature maps for prediction. These methods typically rely on input data from other sensors rather than RGB camera itself, which is not the general case for many autonomous driving cars. Thus we aim to develop an object detection architecture that only takes RGB camera data as input in this work.

Domain adaptation reduces the discrepancy between different domains, thus allows the model trained on source domain to be applicable on unlabeled target domain. Previous domain adaptation works mainly focus on the task of image classification [46, 47, 56, 48], while more and more methods have been proposed to solve domain adaptation for object detection in recent years [5, 24, 39, 60, 50, 55, 58, 49, 15]. Domain adaptive detectors could be obtained if the features from different domains are aligned [5, 18, 39, 49, 15, 52]. From this perspective, Chen et al. [5] introduce a Domain Adaptive Faster R-CNN framework to reduce domain gap from image level and instance level, and the image-and-instance consistency is subsequently employed to improve cross-domain robustness. He et al. [18] propose a MAF (multi-adversarial Faster R-CNN) framework to minimize the domain distribution disparity by aligning domain features and proposal features hierarchically. On the other hand, some works try to solve domain adaptation through image style transfer methods [41, 24, 21]. Shan et al. [41] first convert images from source domain to target domain with image translation module, then train the object detector with adversarial training on target domain. Hsu et al. [21] choose to translate images progressively, and add a weighted task loss during adversarial training stage for tackling the problem of image quality difference. Many previous methods [62, 27, 38, 4] design complex architectures. [62] used multi-scale backbone Feature Pyramid Networks and considered pixel-level and category-level adaptation. [27] used the complex Graph Convolution Network and graph matching algorithms. [38] used the similarity-based clustering and grouping. [4] uses the uncertainty-guided self-training mechanism (Probabilistic Teacher and Focal Loss) to capture the uncertainty of unlabeled target data from a gradually evolving teacher and guides student learning. Differently, our method does not bring extra learnable parameters to original Faster R-CNN model because our AdvGRL is based on adversarial training (gradient reversal) and Domain-level Metric Regularization is based on triplet loss. Previous domain adaptation methods usually treat training samples at the same challenging level, while we employ advGRL for adversarial hard example mining to improve transfer learning. Moreover, we generate an auxiliary domain and apply domain-level metric regularization to avoid overfitting.

As illustrated in Fig. 2, our proposed model adopts the pipeline in Faster R-CNN for object detection. The Convolutional Neural Network (CNN) backbone extracts the image-level features from the RGB images and send them to Region Proposal Network (RPN) to generate object proposals. Afterwards, the ROI pooling accepts both image-level features and object proposals as the input to retrieve the object-level features. Eventually, a detection head is applied on the object-level features to produce the final predictions. Based on the Faster R-CNN framework, we integrate two more components: image-level domain adaptation module, and object-level domain adaptation module. For both modules, we deploy a new Adversarial Gradient Reversal Layer (AdvGRL) together with the domain classifier to extract domain-invariant features and perform adversarial hard example mining. Moreover, we involve an auxiliary domain to impose a new domain-level metric regularization to enforce the feature metric distance between different domains. All three domains, i.e., source, target, and auxiliary domains, will be employed simultaneously during the training.

The image-level domain representation is obtained from the backbone feature extraction and contains rich global information such as style, scale and illumination, which can potentially pose significant impacts on the detection task [5]. Therefore, a domain classifier is introduced to classify the domains of the upcoming image-level features to enhance the image-level global alignment. The domain classifier is just a simple CNN with two convolutional layers and it will output a prediction to identify the feature domain. We use the binary cross entropy loss for the domain classifier as follows:

where $i\in\{1,...,N\}$ represents the $N$ training images, $G_{i}\in\{1,0\}$ is the ground truth of the domain label in the $i$-th training image ($1$ and $0$ stand for source and target domains respectively), and $P_{i}$ is the prediction of the domain classifier.

Besides the image-level global difference in different domains, the objects in different domains might be also dissimilar in the appearance, size, color, etc. In this paper, we define each region proposal after the ROI Pooling layer in Faster R-CNN as a potential object. Similar with image-level adaptation module, after retrieving the object-level domain representation by ROI pooling, we implement a object-level domain classifier to identify the feature derivation from local information. A well-trained object-level classifier, a neural network with 3 fully-connected layers, will help align the object-level feature distribution. We also use the binary cross entropy loss for this domain classifier:

where $j\in\{1,...,M\}$ is the $j$-th detected object (region proposal) in the $i$-th image, $P_{i,j}$ is the prediction of the object-level domain classifier for the $j$-th region proposal in the $i$-th image, and $G_{i,j}$ is the corresponding binary ground-truth label for source and target domains respectively.

In this section, we first review the original Gradient Reversal Layer (GRL) [10], then we make a detailed description of the proposed Adversarial Gradient Reversal Layer (AdvGRL) for our domain adaptive object detection framework.

The original GRL is used for unsupervised domain adaptation of the image classification task [10]. Specifically, it leaves the input unchanged during forward propagation and reverses the gradient by multiplying it by a negative scalar when back-propagating to the base network ahead during training. A domain classifier is trained to maximize the probability of identifying the domain while the base network ahead is optimized to confuse the domain classifier. In this way, the domain-invariant features are obtained to realize the domain adaptation. The forward propagation of GRL is defined as:

where ${\mathbf{v}}$ is an input feature vector, and $R_{\lambda}$ denotes the forwarding function that GRL performs, and the back-propagation of GRL is defined as:

where $\mathbf{I}$ is an identity matrix and $-\lambda$ is a negative scalar.

The original GRL sets either a constant or a changing $-\lambda$ based on the training iterations [10]. However, this setting ignores an insight that different training samples might have different challenging levels during the transfer learning. Therefore, this paper proposes a novel AdvGRL to perform adversarial mining for the hard examples together with the domain adaptation to further enhance the model’s transfer learning capabilities under challenging examples. This can be done by simply replacing $\lambda$ by a new $\lambda_{adv}$ in Eq. (4) of GRL, which forms the proposed AdvGRL. Particularly, $\lambda_{adv}$ is calculated as:

where $L_{c}$ is the loss of the domain classifier, $\alpha$ is a hardness threshold to judge whether the training sample is challenging, $\beta$ is the overflow threshold to avoid generating excessive gradients in the back-propagation, and $\lambda_{0}=1$ is set as a fixed parameter in our experiment. In other words, if the domain classifier’s loss $L_{c}$ is smaller, the domain of the training sample can be more easily identified, whose feature is not the desired domain-invariant feature, so this kind of training sample is a harder example for domain adaptation. The relation of $\lambda_{adv}$ and $L_{c}$ is shown in Fig. 3.

On summary, the proposed AdvGRL has two effects: 1) AdvGRL could use negative gradients during back-propagation to confuse the domain classifier so as to generate domain-invariant features; 2) AdvGRL could perform adversarial mining for the hard examples to further enhance the model generalization under challenging examples. The proposed AdvGRL is applied to both image-level and object-level domain adaptation in our domain adaptive object detection framework, as shown in Fig. 2.

Previous existing domain adaptation methods mainly focus on the transfer learning from source domain $S$ to target domain $T$, which neglects the potential benefits of the third related domain can bring. To address this and thus additionally involve the feature metric constraint between different domains, we introduce an auxiliary domain for a domain-level metric regularization during the transfer learning.

Based on the source domain $S$, we can apply some advanced data augmentation methods to generate an auxiliary domain $A$. For the autonomous driving scenario, the training data in different weather conditions can be synthesized from the clear-weather data, then the three input images of our architecture (as shown in Fig. 2) could be aligned images. For example, we generate an auxiliary domain with the advanced data augmentation method RainMix [16, 20]. Specifically, we randomly sample a rain map from the public dataset of real rain streaks [11], then perform random transformations using the RainMix technique on the rain map, where these random transformations (i.e., rotate, zoom, translate, shear) are sampled and combined. Finally, these transformed rain maps can be blended with the source domain images, which can simulate the diverse rain patterns in the real world. The example of generating auxiliary domain is shown in Fig. 4. Different with other methods including data augmentation to the source/target domain, by generating an auxiliary domain with data augmentation, the domain-level metric constraint between source, auxiliary, and target domains is ensured.

Let us define the $i$-th training image’s global image-level features of $S$, $A$ and $T$ as $F_{i}^{S}$, $F_{i}^{A}$, and $F_{i}^{T}$ respectively. We expect to ensure the feature metric distance between the $F_{i}^{S}$ and $F_{i}^{T}$ closer than the feature metric distance between $F_{i}^{S}$ and $F_{i}^{A}$ after reducing the domain gap between $S$ and $T$, which is defined as:

where $d(,)$ denotes the metric distance of the corresponding features. This constraint can be implemented by a triplet structure, where the $F_{i}^{S}$, $F_{i}^{T}$, $F_{i}^{A}$ can be treated as anchor, positive and negative in the triplet structure. Therefore, as the domain-level metric regularization on image features, the above image-level constraint in Eq. (6) is equivalent to minimize the following image-level triplet loss:

where the parameter $\delta$ is used as a margin constraint and we set $\delta=1.0$ in our experiments.

Similarly, let us define the $i$-th training image’s $j$-th object-level features of $S$, $A$ and $T$ as $f_{i,j}^{S}$, $f_{i,j}^{A}$, and $f_{i,j}^{T}$ respectively. As the domain-level metric regularization on object features, we will also minimize the following object-level triplet loss:

The final training loss of the proposed network is a summation of each individual part, which can be written as:

where $L_{cls}$ and $L_{reg}$ are the loss of classification and the loss of regression in the original Faster R-CNN respectively, and $w$ is a weight to balance the Faster R-CNN loss and the domain adaptation loss for training. We set $w=0.1$ in our experiments. In the training, the proposed domain adaptive object detection framework can be trained in an end-to-end manner using a standard Stochastic Gradient Descent algorithm. During the testing, the original Faster R-CNN architecture with trained adapted weights can be used for object detection, after removing the domain adaptation components.

Our model has the capability to be adapted to general domain adaptive object detection. For the scenarios that the images from target domain are synthesized from the source domain with pixel-to-pixel correspondence (e.g., Cityscapes$\longrightarrow$Foggy Cityscapes), our method can be directly applied without modification. For the scenarios where target and source domains do not have strict correspondence (e.g., Cityscapes$\longrightarrow$KITTI), our method can be applied by simply removing the $L^{R}_{obj}$ loss to eliminate the dependence on the object alignment in the model training.

Our experiments are based on the public object detection benchmarks Cityscapes [7] and Foggy Cityscapes [40] for autonomous driving. Cityscapes [7] is a widely used autonomous driving dataset, which is a collection of images with city street scenarios in clear weather conditions from 27 cities. In Cityscapes dataset, there are 2,975 training images and 500 validation images with instance segmentation annotations which can be transformed into bounding-box annotations with $8$ categories. All images are 3-channel RGB images and captured by a car-mounted video camera with the same resolution of $1024\times 2048$. Foggy Cityscapes [40] is established by simulating the fog of different intensity levels on the Cityscapes images, which generates the simulated three levels of fog based on the depth map and a physical model [40]. Its image number, resolution, training/validation split, and annotations are same as those of Cityscapes. Following the previous methods [5, 49, 15], the images with the fog of highest intensity level are utilized as the target domain for transfer learning in our experiments.

The results of weather adaptation from clear weather to foggy weather are represented in Table 1. Compared with other domain adaptation methods, we can see that our proposed method achieves the best detection performance with a mAP of $42.3\%$, which is higher than the second best method UaDAN [15] with a mAP improvement of $1.2\%$. For each category, we can see that the proposed method is able to alleviate the domain gap over most of the categories in Foggy Cityscapes, e.g., bus got $51.2\%$, bicycle got $39.1\%$, person got $36.5\%$, rider got $46.7\%$, and train got $48.7\%$ as the best performance in AP, which is highlight in Table 1. The proposed method can reach the $48.7\%$ AP for the train detection in Foggy Cityscapes, compared to the $45.4\%$ AP by the second best method DA-Faster, where the proposed method is $3.3\%$ better than DA-Faster. While PDA got $54.4\%$ in car, GPA got $32.4\%$ in motorcycle, DA-faster got $30.9\%$ in truck as the best performance in some categories, the proposed method is comparable across these three categories with a minor difference. Obviously, compared to these recent domain adaptation methods, the proposed method achieves the best performance in overall mAP performance and more than half categories of Foggy Cityscapes.

To fully evaluate the proposed method, we conduct an experiment to perform the cross-camera adaptation between real-world autonomous driving datasets with different camera settings. To apply our method to the unaligned datasets in the real-world, we simply remove $L^{R}_{obj}$ (Eq. 8) to apply our method from Cityscapes (source) to KITTI [12] (target) datasets for cross-camera adaptation. Same as [5], we use KITTI training set (7,481 images of resolution $1250\times 375$) as target domain in both adaptation and evaluation, and AP of Car on target domain is evaluated. The result is in Table 3, where the proposed method achieved outstanding performance compared with recent SOTA methods.

The effect of each individual proposed component for the domain adaptation detection method is investigated in this section. All experiments are conducted with the same RestNet-50 backbone on the Cityscapes$\longrightarrow$Foggy Cityscapes experiment. The results are presented in Table 2. In the first row, ‘img’ and ‘obj’ stand for the image-level adaptation module and object-level adaptation module respectively, while ‘AdvGRL’ and ‘Reg’ denote the proposed Adversarial Gradient Reversal Layer and domain-level metric Regularization respectively. ‘img+obj+GRL’ stands for the baseline model in our experiment. We denote that ‘img+obj+AdvGRL’ (Ours w/o Auxiliary Domain) and ‘img+obj +AdvGRL+Reg’ use the AdvGRL to replace the original GRL. The ‘Source only’ indicates the Faster R-CNN model without domain adaptation trained only with labeled source domain images. The ablation study in Table 2 clearly justifies the positive effect of each proposed component of the domain adaptive object detection.

The study on different hyper-parameters of Eq. 9 and Eq. 5 are conducted. We use the Cityscapes$\longrightarrow$Foggy Cityscapes as the study case. First, the loss balance weight $w$ in Eq. 9 is set as $0.1$, $0.01$, $0.001$ separately for training, and the corresponding detection mAP(s) are 42.34, 41.30, 41.19, respectively. Second, in the AdvGRL (Eq. 5), the overflow threshold $\beta$ and hardness threshold $\alpha$ are set as (1) $\beta=30,\alpha=0.63$, (2) $\beta=10,\alpha=0.63$, (3) $\beta=30,\alpha=0.54$, and (4) $\beta=10,\alpha=0.54$, where $\alpha=0.54$ is computed by averaging the values of Eq. 1 when $P_{i}=0.9,G_{i}=1$ and $P_{i}=0.1,G_{i}=0$. The detection mAP(s) of these settings are (1) 42.34, (2) 38.83, (3) 39.38, (4) 40.47, respectively.

Using $\lambda_{adv}$ of the proposed AdvGRL, we could find some hard examples, as shown in Fig. 5. We compute the $L_{1}$ distance of features $F^{S}_{i}$ and $F^{T}_{i}$ after the CNN backbone of Fig. 2 as the approximated hardness $ah$, where smaller $ah$ means harder for transfer learning. Intuitively, if the fog covers more objects as shown in bounding-box regions of Fig. 5, it will be more difficult.