Adversarial Alignment for Source Free Object Detection

Aligning source and target domain is a widely-used method for conventional domain adaptation tasks (Kang et al. 2019; Zhu et al. 2019; Wang et al. 2019). Given the source and target data, we can intuitively achieve the alignment in the data space (Chen et al. 2020) or feature space (Saito et al. 2019). However, the lack of source data poses a new challenge for domain alignment.

Although we have no access to source data, the source-pretrained model does convey rich information about the source domain. Accordingly, we propose to self-divide the target data into two subsets by the pre-trained model to explicitly simulate the source and target domain. To achieve this goal, we design a detection variance-based division criterion, where the detection variance is calculated based on predictions yielded by the source-pretrained model on target data. It is motivated by a finding that larger detection variances denote higher similarity to the source data. Specifically, the pre-trained model tends to yield more predictions of hard samples for source-similar images while directly ignore these hard samples for source-dissimilar images. These predictions of hard samples usually have high uncertainty, and hence have larger variances during adaptation.

We formulate the calculation of the detection variance as:

where $F_{\theta_{s}}(X_{i})$ represents the predictions of image $X_{i}$ via the source-pretrained model. As such calculation is intractable in practice, we instead approximate this calculation with Monte-Carlo sampling. Inspired by (Gal and Ghahramani 2016), we formulate the sampling function with dropout, which is a widely-used stochastic regularization tool in deep learning (Blundell et al. 2015). This approximation is easy to perform via $M$ stochastic forward passes without changing the detection model.

As the corresponding outputs $F_{\theta_{s}}(X_{i})=(\bm{b_{i}},\bm{c_{i}})$ are composed of localization coordinates and classification scores, we formulate the detection variance as the product of two terms, box localization variance $v_{bi}$ and classification score variance $v_{ci}$. Given a prediction with $N_{i}$ boxes and $K$ classes, we have $\{\bm{b_{ij}}=(x_{ij}^{1},y_{ij}^{1},x_{ij}^{2},y_{ij}^{2})\}^{N_{i}}_{j=1}$ and $\{\bm{c_{ij}}=(c_{ij}^{1},c_{ij}^{2},...,c_{ij}^{K})\}^{N_{i}}_{j=1}$. We can formulate $v_{bi}$ and $v_{ci}$ as follows:

where $\bm{b_{ij}^{m}}$, $\bm{c_{ij}^{m}}$ denote the localization coordinates and classification scores of the $m$-th forward pass of the $j$-th bounding box in $X_{i}$ respectively, and $\overline{\bm{b_{ij}}}$, $\overline{\bm{c_{ij}}}$ denote the corresponding average value of total $M$ forward passes. Then we have the detection variance of $X_{i}$ as $v_{i}=v_{bi}v_{ci}$. We order these images according to their variances from small to large, and use $r_{i}$ to denote the ranking of $X_{i}$. We define the variance level of the $i$-th image as $vl_{i}=\frac{r_{i}}{N}$. We consider $X_{i}$ as source-similar if $vl_{i}\geq\sigma$ and source-dissimilar otherwise, where $\sigma\in(0,1)$ is a pre-defined threshold. In this way, we divide the target domain data into source-similar and source-dissimilar subsets for the preparation of adversarial alignment.

In this subsection, we introduce how to achieve domain alignment with the divided source similar and source dissimilar subsets. As shown in Stage 3 in Figure 2, we incorporate an adversarial training procedure into a mean-teacher architecture to achieve domain alignment in the feature space. Specifically, we build a teacher model $F_{tea}$ and a student model $F_{stu}$, which apply the same network architecture with pretrained model $F$ and are initialized with parameters $\theta_{s}$. The parameters of the student model are quickly updated for domain alignment while the parameters of the teacher model are slowly updated. There are two loss functions in our adversarial alignment process to learn the student model: mean teacher loss for model adaptation and adversarial loss for domain alignment.

The goal of the mean teacher loss is to use the pseudo labels generated with the pretrained teacher model to supervise the training of the student model. First, we feed the teacher model with source similar data $X_{i}^{s}$ and feed the student model with the data augmentation version $Aug(X_{i}^{s})$. Then the mean teacher loss is formulated with the outputs of both teacher and student models as:

where $L_{reg}$ is the location regression loss which calculates the L1-smooth distance for predicted and supervised bounding box and $L_{cls}$ is the cross-entropy loss for classification supervision. Both $L_{reg}$ and $L_{cls}$ are widely-used losses in the field of object detection, such as Faster-RCNN (Ren et al. 2015).

To align the feature spaces of source similar and source dissimilar subsets, we further conduct adversarial learning with the outputs of the student model. We take the student model as the “generator” and build extra “discriminators” to play a min-max adversarial game, where the “generator” tries to fool the “discriminators” by generating features that can’t be distinguished as source similar or source dissimilar. To capture both local and global information, we follow SW-Faster (Saito et al. 2019) to build a local discriminator $D_{l}$ and a global discriminator $D_{g}$, where $D_{l}$ focus on the foreground objects and $D_{g}$ focus on the background.

The adversarial losses with global and local discriminators can be formulated as:

where $X_{i}^{s}$, $X_{j}^{d}$ denote the $i$-th source similar image and $j$-th source dissimilar image; $F_{l}$, $F_{g}$ denote different layers on the backbone that capture the local feature (feature maps) and global feature (a feature vector) respectively; $W,H$ denote width and height of local feature map on the backbone; $N_{s}$, $N_{d}$ denote the total number of source similar images and source dissimilar images, and $\gamma$ is a Focal loss (Lin et al. 2017) parameter which controls the model to focus on hard-to-classify examples but not the easy ones. Compared with the local adversarial loss, the global one applies the Focal loss to focus on the hard examples and ignore easy-to-classify examples to achieve a weak alignment, without hurting the performance of the local model (Saito et al. 2019).

Finally, we update the student model by fusing the mean teacher loss and adversarial losses as

where $L_{adv}=L_{local}+L_{global}$. With this loss function, these embeddings are encouraged to have similar distributions for both source-similar and source-dissimilar images (achieved by adversarial loss $L_{adv}$) and have a great discriminative ability for the target domain detection (achieved by mean teacher loss $L_{mt}$).

After adversarial alignment, we can get pseudo labels of high quality over the whole target data. Hence, we finetune the detector by using both source-similar and source-dissimilar data to make full use of the information of the whole target domain. Both teacher and student models are initialized with the parameters of the student model learned in stage 3. As the detection error mainly comes from false negative objects (Li et al. 2021a), we employ mosaic augmentation (VS, Oza, and Patel 2022; Wang et al. 2021) to simulate the false negatives to better detect small-scale and obscured objects.

As shown in Figure 2 Stage 4, we feed the teacher model with four independent target images $\{X_{i1}^{t},X_{i2}^{t},X_{i3}^{t},X_{i4}^{t}\}$ and generate independent predictions $\{Y_{i1}^{t},Y_{i2}^{t},Y_{i3}^{t},Y_{i4}^{t}\}$. Then we resize and mosaic these four images with data augmentation into a combined image $X_{im}$, with the same size as the original input $X_{i1}^{t}$. Likewise, we obtain the mosaic pseudo label $Y_{im}$. We formulate the loss during the fine-tuning stage as follows:

where $L_{reg}$ and $L_{cls}$ are basic location regression and classification losses respectively, which are the same as the ones in Equ. (4).

We evaluated our method on five popular object detection datasets. The detailed information of these datasets is summarized in the following: (1)Cityscapes (Cordts et al. 2016) collects different scenes from various cities on the street, which contains 2,975 training images and 500 validation images. We utilized the rectangle of the instance mask to obtain bounding boxes following previous work. (2)Foggy-Cityscapes (Sakaridis et al. 2018) is constructed from Cityscapes by simulating three levels of foggy weather. It contains the same amount of images as the Cityscapes. We inherited the annotations of Cityscapes. (3)KITTI (Geiger, Lenz, and Urtasun 2012) is a dataset containing 7,481 training images for autonomous driving different from Cityscapes. (4)Sim10k (Johnson-Roberson et al. 2017) is a simulation dataset generated from a popular computer game Grand Theft Auto V. It contains 10,000 images of the synthetic driving scene with 58,071 bounding boxes of the car. (5)BDD100k (Yu et al. 2018) is a large dataset including 100k images with six types of weather, six different scenes, and three categories for the time of day. Following (Xu et al. 2020a), we applied the daytime subset of the dataset in our experiment, where 36,728 images were used for training and the other 5,258 images for validation.

We followed the setting in (Chen et al. 2018) that adopted Faster-RCNN (Ren et al. 2015) with VGG-16 pretrained on ImageNet (Russakovsky et al. 2015) for our detection model. In all experiments, the shorter side of each input image was resized to 600. The detector was trained with Stochastic Gradient Descent (SGD) with a learning rate of 0.001. To stabilize the training of Mean Teacher (Tarvainen and Valpola 2017), we only updated the teacher model every 2500 iterations using exponential moving average (EMA) weights of the student model. In the pseudo label generation process, we filtered out the bounding boxes whose classification scores were lower than 0.7 to control the quality of pseudo labels. In the testing phase, we evaluated the adaptation performance by reporting mean average precision (mAP) with an IoU threshold of 0.5. Following (Saito et al. 2019), we set $\lambda$ = 0.1 for Sim10k $\rightarrow$ Cityscapes in Equ. (7) and $\lambda$ = 1 for other tasks. We set the threshold parameter $\sigma=0.7$ as it is empirically found to result in the best performance. All experiments were implemented with PyTorch 1.7.1.

In this subsection, we evaluated the transferability of our method in 4 aspects, including from a normal environment to foggy weather, from training dataset to unseen new scenes, from synthetic to real images, and from a data-limited source to a large-scale target. For a fair comparison, we strictly followed the experiment setting of SFOD-Mosaic (Li et al. 2021a) and applied the similar source-only model, even if a more complex source-only model results in better performance. Specifically, we mainly compared our method $\text{A}^{2}$SFOD with multiple recent methods such as DA-Faster (Chen et al. 2018), SW-Faster (Saito et al. 2019), DA-Detection (Hsu et al. 2020), MAF (He and Zhang 2019), AT-Faster (He and Zhang 2020), and Noise Labeling (Khodabandeh et al. 2019); the baseline method SFOD-Mosaic (Li et al. 2021a); the “Source only” method trained with only source training data as the lower bound; and the “Oracle” method trained using labeled target data as the upper bound. Generally speaking, we achieved significant performance improvement over other methods in all settings.

In this subsection, we conducted ablation studies to investigate the effectiveness of each component and gave more analysis. We are to answer the following questions.

Q: Why the detection variance of the pretrained model can be regarded as a criterion for target division. A: It is because of a finding that the images with larger detection variances are more similar to the data in the source domain. Target division aims to divide the target data into a source-similar set and a source-dissimilar set for aligning the target data and untouched source data. In this paper, we proposed to use detection variance as the criterion for target division since we found the larger detection variance denotes higher recall and more similar to the source data. The pretrained model tends to give more predictions on the source-similar images. It causes a higher recall since more predictions mean less missed detection. On the other hand, more predictions indicate that some predictions are uncertain, which have a large variance with different dropout samplings. We designed two experiments to verify our findings.

To verify the relation between the detection variance and the similarity to source data, we designed a qualitative experiment to show the source images and target images at different variance levels. While it is difficult to quantitatively identify this relation due to the lack of the similarity metric, we can clearly observe that the images with larger variance are more similar to the source data from Figure 4. As we set the threshold parameter $\sigma=0.7$, (a), (b) and (c) are considered to be source-dissimilar images and (d), (e) represent source-similar images.

To verify whether the detection variance and recall are positively correlated, we provided a quantitative evaluation. Specifically, given a pre-trained model, we first sorted the testing data by the variance value and split them into several groups. Then we tested the model on the groups with different variance levels and calculated the recall score. As shown in Figure 3, we conducted experiments on four settings including $Cityscapes\rightarrow Foggy-Cityscapes$, $KITTI\rightarrow Cityscapes-Car$, $Sim10k\rightarrow Cityscapes-Car$, and $Cityscapes\rightarrow BDD100k$, where $A\rightarrow B$ denotes that we pretrained the model on A and tested the model on B. In all settings, we found the variance level and recall score have highly positive correlations, with correlation coefficients $R^{2}=0.962,0.933,0.848,0.927$ respectively.

Besides, we also provided the mAP performance in the blue curves of Figure 3, which shows that we can obtain better detection results in images with larger variances. Generally speaking, we can obtain better detection results with the source pretrained model on the source similar images, which also demonstrates the positive relation between large variance and source similarity.

Q: Can we directly apply the recall as the criterion? A: No. Recall is a metric that evaluates the detection of a set of images. However, the criterion is a metric for each image to divide the source-similar and source-dissimilar sets. When considering recall in a single image, the number of objects is variant and may mislead the division.

Q: What are the effects of each component in the method? A: Both target self-division (TSD) based alignment and Mean Teacher (MT) based fine-tuning are critical for our method. The main differences between $\text{A}^{2}$SFOD and other methods are the target self-division (TSD) based alignment and Mean Teacher (MT) based fine-tuning. To explore the effectiveness of each component, we implemented an ablation study experiment on Cityscapes $\rightarrow$ Foggy-Cityscapes. The experiment results are summarized in Table 5. We remove the Mean-Teacher based fine-tuning and target self-division from $\text{A}^{2}$SFOD as our baseline. When adding the MT to the baseline, we can achieve +2.7$\%$ mAP improvement. When applying TSD-based alignment, we can further achieve the 35.4$\%$ mAP score with +2.8$\%$ improvement. These significant improvements demonstrate that both components are critical for our method.