Quantifying Context Bias in Domain Adaptation for Object Detection

Background influence [20, 21, 22, 23, 24, 25, 26, 27] and context bias [28, 29, 30, 31] aim at improving performance in tasks such as classification, object recognition, and object localization. Xiao et al. [20] and Zhu et al. [24] studied background effect on accuracy of classification by modifying images with different combinations of foreground and background. A paper proposed a graphical model [30] which modeled foreground-background associations in conditional probability which serves as a methodological inspiration for us. Various studies have addressed context bias using several techniques, such as data augmentation to generate out-of-distributions samples into the background, combination of naturally unmatched background and foreground (e.g., an elephant in room), and applying background removal during training. Torralba [32] demonstrates background effect can be factorized into object priming, focus of attention, and scale selection by modeling the foreground-background associations in a probabilistic model. Liang et al. [21] studied background influence using fashion dataset [33, 34]. These studies [35, 36] can localize foreground objects better than CAM-based algorithms without using bounding box information and with only classification labels. These prior works focus on context bias in the same domain and uses datasets with smaller variations like centered objects or single objects. It lacks to provide strong insight for DAOD.

Different variations of DAOD methods have been proposed using feature alignment, synthetic images, and self-training or self-distillation. Feature alignment is to find transformations between source and target domain to reduce distribution shift with adversarial training [37, 1, 38, 39]. It can be helpful to extract common latent features from different domains. Progressive Domain Adaptation for Object Detection [40] synthesized new dataset by using cycleGAN [41] which enables to bridge domain gaps and Self-Adversarial Disentangling for Specific Domain Adaptation [42] achieved 45.2 mAP on Cityscapes to Cityscapes foggy dataset using synthetic images. Gong et al. [43] utilized transformers to focus on aligning features across backbone and decoder networks. However, combining multiple sources into a single dataset and performing single-source domain adaptation for feature alignment does not guarantee better performance compared to using the best individual source domain [44].

Self-training uses a teacher model to predict pseudo labels on target domains to gradually understand domain shiftiness [45, 46, 47, 2, 48]. MIC [4] employed masked images on teacher-student model and MRT [49] suggested modified masked based retraining approach on the teacher-student model. [50] performed an alignment and distillation to enforce invariance across domains to reduce discrepancy of features.

Finding common features from multiple domains is critical for DAOD. They have summarily demonstrated that the foreground features in latent space can be aligned using dimension reduction methods such as UMAP [51] and t-SNE [52]. These studies do not address how to manage context bias when adapting across different domains. Instead, they propose and validate their methods within DAOD framework using accuracy metrics. Thus, we focus on analyzing the root causes of domain discrepancy in object detection both qualitatively and quantitatively.

Prior studies [30, 32] researched context bias for object classification. The studies pointed out that relying only on local features (foreground features in our case) has limitations, including degraded quality due to noise and ambiguity in the target search space. They extended the likelihood to incorporate context information surrounding the foreground, which enhances object classification by providing a stronger conditional probability. The conditional probability of the object ($O$) given the features ($f$) was given as:

where $F$ and $B$ are the foreground and background features The modeling can also be interpreted as:

where the object is represented by scale ($\sigma$), location ($Y$), and category ($c$).

The challenge with using a convolutional neural network (CNN) to estimate likelihood is the inability to explicitly teach the model to learn each factor in a specific order. In other words, it means that parameters of CNN can be different depending on how it can be trained. In CNNs, likelihood estimation is a process to find the mean of a distribution, which necessitates more samples to accurately estimate the true mean. This aligns with the principle that a more extensive and refined dataset, achieved through data augmentation, is crucial for better performance [53].

In the context of a graphical casual model (F→Y ←B), it represents the joint distribution P(Y, F, B), which can be decomposed as either $P(Y|F,B)P(F|B)P(B)$ or $P(Y|F,B)P(B|F)P(F)$. In the SUN 09 training set used by [30], the association between roads and cars is strong, and the number of road pixels surpasses object pixels (similar to Cityscapes: Fig. 1). Consequently, the CNN is more likely to learn $P(F=car|B=road)$ than $P(B=road|F=car)$, leading to the induction of the equation $P(Y|F,B)P(F|B)P(B)$ during training. Also, the factor $P(B=road)$ can be factorized as $P(Color|X)*P(shape|X)$ and can be learned more easily and earlier than $P(F=car)=P(Color|X)*P(shape|X)$. It can cause why the object detection model fails to detect on different domain especially when the road is not given and changes.

Another reason is related to the receptive field of CNN architecture. As layers in a CNN go deeper, the receptive field increases, causing pooled information after convolution operations mixing foreground and background features [54]. This mixing can create ambiguity between foreground and background features.

Given these considerations, we now delve into how we empirically provide results of context bias on different datasets.

We utilized Detectron2 with three different datasets which semantic labels are available:

• •

  Cityscapes - we used the training dataset of 2,975 images to extract foreground and background features and validation set of 500 images.

• •

  Cityscapes foggy beta 0.02 - it is used only for experiments. It contains synthetic foggy dataset with different levels of foggy controlled by a hyper parameter.

• •

  CARLA - It has 19 different driving scenarios, sampling same number of images as Cityscapes training set.

We focused on three labels: Car, Truck, and Bus.

Detectron2 model consists of a backbone network which extracts feature maps from the input image at five different scales, region proposal networks extracting object regions from multi-scale features and box heads which aggregates the region proposals. To train the Detection2 model, we employed a batch size of 8, a stepLR scheduler with iterations of {10,000, 20,000, 30,000, 35,000, and 37,500} (about 100 epochs). The initial learning rate was set to 0.015, and the image resolution was $1024\times 2048$, with horizontal flip and brightness augmentation. We used the Adam optimizer, regression loss for bounding boxes, and cross-entropy loss for label classifications. After training, the model achieved an AP50 of 53.72 on the Cityscapes validation set and 88.623 on the CARLA validation set. While the result on CARLA indicates overfitting, it provides useful insights into feature alignment between the two different domains.

To separate foreground and background related features per target object, we utilized a combination of activation hooks and semantic pixel labels. To get the overall activation region of the foreground, we utilized Smooth GradCAM++[55] which extracts spatial information from attention and semantic maps, distinguishing foreground and background features. Following this separation, we saved feature maps as feature vectors per object across various layers. While the entire area of the activation is provided by Smooth GradCAM++, we manipulated what percentage of the background activation region is required for the foreground object detection. Therefore, we expanded the area thresholds in terms of bins (where bin 1 represents almost no background and bin 9 represents “all” activation). In other words, the region of interest from background is gradually extended as bin parameter increases. Figure 5 shows visualized example of foreground and background extracted in from different layers and threshold on image from the Cityscapes and CARLA. The algorithm 1 shows the process of extracting features.

MMD was proposed by [56] as a kernel-based statistical test to find the distance between two given distributions. MMD between two distributions $X$ and $Y$ can be mathematically defined as:

where $x_{i}\in X$, $y_{i}\in Y$. Alternatively, a variance based discrepancy has also been proposed by [57] as:

where $b$ represents the unit ball of kernel Hillbert space $\mathcal{H}_{k}$. We compute MMD and MVD between foreground and background feature vectors from different datasets with separated scaled features. Since the MMD values are too small to comprehend, we scaled the values by 1000. UMAP clustering depends on the hyper-parameters such as minimum distance to visualize and number of neighbors to compute weights among samples. To avoid bias, we kept all hyperparameters constant for all experiments.

We start our analysis by plotting the foreground and background feature distributions of different domains using UMAP. Figures 6 and 7 present the visualization of the foreground and background features from different domains in 2D and 3D. In Figure 6, we use the features at the $4^{th}$ ResNet layer from three different data distributions - (a) Cityscapes training dataset as source and validation dataset as target; (b) Cityscapes train dataset as source and Cityscapes foggy dataset as target; and (c) Cityscapes train dataset as source and CARLA as target. The interesting finding is the differences of background alignment between the three comparisons. It is immediately apparent that as the target domain shifts away from the source domain, the background becomes more separable than the foreground. For the Cityscapes training and validation dataset, the foreground and background are separable from each other but are mingled up between source and target. For the second panel, the foreground features are together while the background features are separable but overlapping. We can see an extreme case in the third panel where the foreground features between Cityscape and CARLA are next to each other but are non-overlapping while the background features are very distant from each other. The same patterns of Cityscapes-CARLA are demonstrated in Figure 7 when it uses 3d UMAP embedding. We visualized corresponding image patches for qualitative analysis. Noise and ambiguous foreground features are aligned with the background. Beside, background features contains foreground features because of objects adjacent each other. It can lead to alignment with foreground features.

Given the qualitative analysis, we derive quantitative estimates of difference in foreground and background feature distributions using MMD and MVD. Figures 8 and 9 show the MMD and MVD in violin plots across different layers and bins respectively for different domains. We first split it into small objects in image frame (400 to 6000 sq. pixels) and medium sized objects (6000 to 52000 sq. pixels). In case of Cityscapes-train-val, it shows the highest amount of discrepancy between foreground-to-background (FB) and background-to-foreground (BF), while having less discrepancy between foreground-to-foreground (FF) and background-to-background (BB). Significant differences or discernible patterns were not observed in the MVD.

Overall, in case of Cityscapes-CARLA domains, it shows the bin 1 does not show significant difference and bin 9 has significant differences across FF, FB, BF, and BB. As we go deeper ($5^{th}$ layer), the MMD of BB has the higher value compared to that observed in the shallower layer ($2^{nd}$ layer). The variability in the overall metrics is higher in the second figure is larger as compared to the first as expected.

The Detectron2 model trained on the Cityscapes dataset achieves 53.72 mAP, with performance dropping to 41.06 mAP on the CARLA validation set and 37.77 mAP on the Cityscapes foggy beta 0.02 set. Despite the CARLA dataset being relatively easier, there is a significant drop of 47.56 mAP from 88.62 mAP, which implies that even when foreground features are well-aligned, the difference in background associations contributes to the significant drop. This indicates that using a trained model without proper alignment of background context can lead to severe degradation in performance. The observation of MMD in different layers implies that CNNs typically learn finer, low-level features at shallower layers and captures progressively representing more abstract, higher-level features at deeper layers. The minimal variance observed among features post-embedding suggests that the CNN model captures backgrounds across various domains. The challenge of DAOD may stem from insufficient understanding of the association between foreground and background elements.

Even though we demonstrate how the background and foreground features are aligned and separable, extracting the features separately and using them is tremendously computationally expensive. Therefore, using any contextual cue in the object detection model needs to be computationally efficient to make it useful. While CAM is a useful tool for obtaining attention maps, its effectiveness in analyzing activation maps for detection tasks on entire images is questionable. To address this concern, we used only true positive object detection cases and measured correspondences between attention maps and the foreground region of objects using instance segmentation labels. With models trained on Cityscapes and Carla, this approach showed a complete match. Another limitation is that we need to check different class labels to find association between foreground and background like pedestrian, bicyclists. We also use only a single model and therefore, the results can differ with respect to models. For example, vision transformer is a dominant model achieving better accuracy in multiple fields and it can show different results. Finally, we leave a method to mitigate or leverage the context bias during training as future work because it is beyond the current scope.