Predicting Class Distribution Shift for Reliable Domain Adaptive Object Detection

Initially proposed for semi-supervised learning [6], self-training methods have recently produced SOTA results on UDA-OD benchmarks [4, 5]. Several augmentations have been made to the standard Mean Teacher framework [6] to improve pseudo-label reliability under domain shift. Methods have been proposed for merging patches from labelled and unlabelled images [9], selecting optimal pseudo-labels to balance true positive and false negative detections [11] and performing style transfer between the source and target domains [5]. Recognising that pseudo-labels are inevitably unreliable, Chen et al.[4] propose a probabilistic self-training framework that does not require confidence thresholds [14]. Instead, they implement an entropy focal loss that encourages the student to pay more attention to high certainty detections. While promising, these methods fail to leverage the rich contextual information available during robotic deployment and focus on shifts in the general appearance of images. Consequently, the potential for contextual and class distribution shift is largely ignored. Motivated by this, Xu et al.[15] enforce consistency between the object detector and an image-level multilabel classifier, which is more robust to arbitrary changes in image background. However, this method was proposed for use in domain alignment instead of self-training. Cai et al.[10] model the relationships between objects in a scene using a graph structure, and enforces consistency between student and teacher graphs. While incorporating contextual information improves robustness, the pseudo-labels and relational graphs generated using arbitrary thresholds remain unreliable. Furthermore, these methods do not consider the impact of class distribution shift on self-training. In response, we model image context to predict the class distribution of unlabelled data in novel deployment environments. This prior can be incorporated into self-training via ACT, which generates pseudo-labels to match the predicted class distribution [13]. This method has shown promising results in SSOD, and we propose a series of changes to optimise it for domain adaptation.

It is well established that class imbalance leads to severe confirmation bias during self-training, as dominant classes are predicted with high confidence and subsequently reinforced [8]. Solutions to this problem have been proposed for SSOD, such as the weighting of low probability classes [8, 14] or ACT to align the distribution of pseudo-labels with a known class distribution [13]. However, existing methods fail to address the class imbalance problem in the presence of domain shift.

Existing UDA-OD benchmarks often focus on appearance changes due to factors such as weather, lighting conditions, or image corruptions [1], but do not adequately assess realistic class distribution shifts. Ignoring class distribution shift is a critical limitation, as they are standard in open-world robotic deployment and can significantly impact the performance of object detection algorithms [16, 2, 17]. In particular, the Cityscapes [18] to Foggy Cityscapes [19] adaptation scenario contains no class distribution shift as the target domain is an augmented version of the source images [1]. Similarly, the cross camera adaptation and simulation to real scenarios only contain a single class [1]. The final benchmark studies adaptation from a small to large-scale dataset [20], reflecting the challenge of deploying a robot in a new environment with a broad array of conditions. We find that the shift in location produces significant class distribution shift, making it useful for this work. However, the impact of class distribution shift on UDA-OD is poorly evaluated using existing benchmarks. In turn, the benefits of explicitly addressing this problem during self-training are largely unexplored.

The abundance of captioned images has allowed the learning of transferable image representations using natural language supervision. One notable example is Contrastive Language-Image Pretraining (CLIP), which trains a model to predict the images and text that occur together using 400 million image-text pairs [12]. Because natural language descriptions are highly generalisable, CLIP can be easily transferred to new tasks and exhibits superior resistance to domain shift compared to supervised pre-training methods [21, 22, 12, 23]. Language descriptions also contain a wealth of contextual information [24, 25, 26], making them useful for modelling the deployment environment of a robot. Recent work proposes using the similarity between an image and a series of text prompts produced by CLIP as a representation of image context [23]. This representation is a strong predictor of object occurrence in an image, helping to improve semantic segmentation performance in open-world settings. In this work, we leverage the domain invariance and contextual detail of these image-text similarity scores to predict the class distribution of unlabelled data.

In UDA-OD, a set of $n_{l}$ labelled images $D_{l}=\left\{X_{l},Y_{l}\right\}$ containing $n_{c}$ classes from the the source domain and $n_{u}$ unlabelled images $D_{u}=\left\{X_{u}\right\}$ from the target domain are provided. The goal of UDA-OD is to use both the labelled source and unlabelled target data to optimise performance of the object detector in the target domain.

The Mean Teacher framework [6] encourages consistent predictions by a student and teacher model on augmented versions of the unlabelled data. This process leads the student and teacher to learn a solution from the source domain that is transferable to the target domain. An overview of this method is provided in Figure 1. The student and teacher models share the same network architecture, but different weights $\theta^{s}$ and $\theta^{t}$ respectively. For a given unlabelled sample $x_{u}$, a weakly augmented version $x_{u}^{t}$ is created for input into the teacher model by applying horizontal flipping and multi-scaling [13]. After non-maximum suppression (NMS), a confidence threshold is applied to the teacher’s predictions $f(x_{u}^{t},\theta^{t})$ to generate pseudo-labels $y_{u}$. Strong augmentations including color jittering, grayscale, gaussian blurring and cutout patches [13] are applied to create $x_{u}^{s}$, which is used to generate the student’s predictions $f(x_{u}^{s},\theta^{s})$. In turn, an unsupervised loss can be calculated as:

where $\mathcal{L}_{cls}$ is the classification loss of the object detector, and $\mathcal{L}_{reg}$ is the box regression loss. A standard supervised loss of the same form is calculated using a labelled sample $(x_{l},y_{l})$ from the source domain:

The overall loss for a batch containing both labelled and unlabelled samples is then calculated as:

where $\lambda$ is a hyperparameter to weight the unsupervised loss. This loss is used to update the student via gradient descent, and the exponential moving average (EMA) of the student’s weights are used to update the teacher model. That is, after each training iteration t the weights of the teacher model are calculated as:

where $\alpha$ controls the stability of the teacher model.

ACT aims to select confidence thresholds for Mean Teacher to overcome the class imbalance problem. It does so by aligning the class distribution of the pseudo-labels with that of the labelled data [13]. The method firstly calculates the class ratio of the labelled data $\bm{r^{l}}=[r^{l}_{1},...,r^{l}_{n_{c}}]$ and the number of objects per image $n^{l}_{o}$. Using this class distribution prior, the expected number of objects in the unlabelled data for each class $c$ is calculated as:

The confidence threshold for class $c$ is then selected such that $n_{c}^{u}$ pseudo-labels are generated. That is:

where $\bm{p_{c}^{sort}}$ is a list of the confidence scores predicted for class $c$ by the teacher, sorted in descending order. ACT can be used with any self-training approach based on Mean Teacher, but specific changes to the standard implementation are recommended by the authors [13]. A proportion of the pseudo-labels are defined as reliable and used to calculate the hard classification and box regression losses defined in (1). To deal with the noise present in the unreliable pseudo-labels, a soft classification loss is calculated using the cross-entropy between student and teacher classification predictions [13]. This results in a reformulation of (1) as:

where $y_{ur}$ and $y_{uu}$ refer to the reliable and unreliable pseudo-labels, and $\widehat{\mathcal{L}}_{cls}$ denotes the soft classification loss.

Given a batch of $n$ images $X_{i}$ and associated texts $X_{t}$, CLIP is trained to predict which possible image-text pairs $X_{i}\times X_{t}$ actually occurred. Images and texts are input to an image encoder $I$ and text encoder $T$ to extract their respective embeddings. These embeddings are then projected into a shared, multi-modal representation space with learnt projection matrices $W_{i}$ and $W_{t}$. L2-normalisation is then applied to extract the final multi-modal embeddings $Z_{i}\in\mathbb{R}^{n\times d}$ and $Z_{t}\in\mathbb{R}^{n\times d}$:

The cosine similarity of the shared embeddings is then calculated, and multiplied by a learnt temperature parameter $t$ to calculate similarity scores $S\in\mathbb{R}^{n\times n}$ for each text and image pair in the batch:

During training, a cross-entropy loss is applied to the similarity scores to encourage similarity between true image-text pairs and discourage similarity between false pairs. Our work uses the pretrained model to generate similarity scores between images and image classification labels.

We aim to predict the class ratio of the unlabelled data $\bm{r^{u}}\in\mathbb{R}^{n_{c}}$, such that it can be used as prior for performing ACT. An overview of this approach can be found in Figure 2. We start by using CLIP to generate a representation of the semantic context of each image. To do so, we calculate the similarity between the labelled images $X_{l}$ and a series of natural language prompts $L$. The performance of vision-language models on downstream tasks is sensitive to the format of natural language labels, leading to the development of prompt optimisation techniques [25, 27]. We implement a simple approach to generate $L$ where each class label $c$ is converted to a prompt of the form “a photo of $c$” [12]. For each labelled image, this results in a vector of similarity scores $\bm{s_{l}}\in\mathbb{R}^{n_{c}}$ that characterise how the image relates to the text prompts [23]. As the natural language descriptions used to generate the similarity vectors are resistant to domain shift [21, 22], this results in a contextual representation that is consistent across the source and target data.

We investigate two linear regression models, g and k, for predicting the class ratio given the similarity vector $\bm{s}$ as a domain invariant input. The absolute model $g(\bm{s},\beta_{g})$ is optimised using the labelled data to predict the absolute number of instances of each class in an image. By normalising the output of $g(\bm{s},\beta_{g})$ to sum to 1, this model can be used to predict the class ratio. The relative model $k(\bm{s},\beta_{k})$ is optimised using the labelled data to directly predict the ratio of each class in an image. The relative model does not consider how many objects occur in an image, instead learning how likely a class is to occur relative to other classes.

To make a prediction for the class distribution of the unlabelled dataset, we use CLIP to extract the similarity vector $s_{u}$ for each unlabelled image. We then calculate the mean similarity vector across the unlabelled dataset $\bm{\overline{s_{u}}}\in\mathbb{R}^{n_{c}}$ and input it to the linear regression models:

The relative and absolute models produce distinct predictions $\bm{r^{u}_{a}}\in\mathbb{R}^{n_{c}}$ and $\bm{r^{u}_{r}}\in\mathbb{R}^{n_{c}}$ for the class ratio, either of which can be used in our framework. However, neither prediction is consistently more accurate than the other, and the optimal prediction to use for a specific scenario cannot be determined without labelled data. To ensure consistent performance across all scenarios, we therefore average the output of the two models. To do so, we propose using the geometric mean, as it is more appropriate than the arithmetic mean for summarising the central tendency of ratios. We term the element-wise geometric mean of the class ratio predictions $\bm{r^{u}_{merge}}\in\mathbb{R}^{n_{c}}$ as the merged prediction:

Secondly, we found that the merged prediction persistently underestimates the shift from the labelled class ratio $\bm{r^{l}}$ to the unlabelled class ratio $\bm{r^{u}}$. A heuristic solution to this underestimation problem is to square the relative change in the class ratio predicted by $\bm{r^{u}_{merge}}$. Formally, we express $\bm{r^{u}_{merge}}$ relative to $\bm{r^{l}}$ and calculate the squared prediction $\bm{r^{u}_{square}}$ for the class ratio as:

Finally, we apply L1 normalisation to $\bm{r^{u}_{square}}$ so the distribution sums to 1.

Even with a perfectly accurate class distribution, we find that under significant domain shift the pseudo-labels generated by the teacher remain unreliable early in self-training. Thus, instead of estimating the average number of objects per image in the unlabelled data, we set this value to target a mean confidence score $\tau$ across the generated pseudo-labels. This accounts for the initial poor performance of the teacher in the target domain, and allows the number of pseudo-labels per image to increase as the model becomes more confident. We use Algorithm 1 to calculate the number of objects per image required to deliver a mean pseudo-label confidence score of $\tau$. In this algorithm, the average number of objects per image $n_{o}\in\mathbb{R}$ is iteratively increased by $\Delta n_{o}$. At each step, pseudo-labels $y_{u}$ are generated for the entire unlabelled dataset by aligning them with the predicted class ratio $\bm{r^{u}}$ and the current number of objects per image $n_{o}$. The mean confidence score $p_{mean}$ is then calculated across the pseudo-labels, and the loop terminates when this value falls below the target value $\tau$.

Scenarios. We validate our proposed method using two scenarios previously used for UDA-OD, and one novel scenario that captures a more extreme class distribution shift.

• •

  Adaptation from small to large dataset (C2B): as in previous work [4, 11, 15], we utilise Cityscapes [18] as a small source dataset and BDD100k daytime [20] images as the large and diverse target domain. The shift in location in this scenario produces significant class distribution shift.

• •

  Adaptation from normal to foggy weather (C2F): this commonly studied scenario uses Cityscapes [18] as the source domain and Foggy Cityscapes [19] as the target domain. To compare with the optimal performance of the current state of the art [4], we use all three levels of fog.

• •

  Adaptation from daytime to night (C2N): we propose a novel benchmark to explore the impact of more extreme class distribution shift on UDA-OD. We utilise Cityscapes [18] as the source domain and BDD100k night [20] as the target domain.

Network Architecture. A standard object detection architecture is used across all UDA-OD works to enable fair comparison between methods [1]. Following this trend, we use Faster-RCNN [28] as the detector architecture with VGG-16 [29] pre-trained on ImageNet [30] as the backbone. The inference rate of this network is quoted at 5Hz on a Nvidia K40 GPU [28], while our experiments return 20Hz on a Nvidia GeForce RTX 3090. However, our method could be easily extended to novel architectures, making it useful for diverse robotic applications [13].

Implementation Details. We follow the default implementation provided in [13] to test our method, which leverages MMDetection [31]. Thus, we use a batch size of 16 for the labelled and unlabelled data and a learning rate of 0.016. The optimiser used is SGD, with a momentum rate of 0.9 and a weight decay of 0.0001. For all experiments, the model is pre-trained using source data for 4,000 iterations before beginning self-training. For the C2B and C2N scenarios, we implement 42,000 iterations of self-training. Due to slower convergence on C2F, we self-train for 72,000 iterations. Lastly, we use a target mean confidence score $\tau$ of 0.6 and a step size $\Delta n_{o}$ of 0.1 for selecting the number of objects per image.

Evaluation. Standard mean average precision (mAP) at an IOU threshold of 0.5 is used to compare performance with that reported by existing works. In additional to implementing our approach, we conduct the following evaluations:

• •

  For comparison on the novel C2N scenario, we implement Probabilistic Teacher [4] as per the released code and report the peak performance recorded across 24,000 iterations. This method is the current state of the art on the C2B and C2F scenarios, and thus provides a strong baseline for evaluating our method under more extreme class distribution shift.

• •

  We evaluate the original ACT [13] on all scenarios using the same implementation as our method.

• •

  Lastly, for the C2N scenario, we evaluate the model trained for 48,000 iterations on the source data, and target data only.

Our method matches or outperforms existing approaches across all three scenarios (Table I, II). Strong performance occurs on scenarios with more extreme class distribution shift, a case that has been largely overlooked in existing benchmarks. We outperform the strongest baseline Probabilistic Teacher [4] by 4.3 mAP on the previously studied C2B scenario (Table I). We further return a 4.7 mAP improvement on the novel C2N scenario (Table I), which is more challenging due to extreme class distribution shift (Figure 5) and the discrete appearance change from daytime to night. A clear improvement is also noted on both scenarios relative to using the original ACT [13]. These results highlight that by modelling shifts in a robot’s deployment context, pseudo-label reliability can be improved in Mean Teacher. Furthermore, it emphasises that class distribution shift must be considered when designing UDA-OD methods for robotics applications.

Interestingly, our approach remains competitive with existing methods on the normal to foggy scenario (C2F) (Table II), where there is a large change in image appearance but no class distribution shift. The average performance across five trials of our method is identical to the optimal performance returned by Probabilistic Teacher. The benefit of our approach is prominent in low probability classes such as train, truck and bus, indicating that we are effectively mitigating the class imbalance problem. Furthermore, we return a 4.1 mAP improvement on this scenario relative to the original ACT, emphasising the benefit of accounting for pseudo-label reliability by dynamically updating the number of objects per image.

To further explore the proposed approach, we assess the pseudo-label reliability generated using alternative threshold selection techniques on the C2B scenario. To do so, we take the teacher model after 1000 iterations of self-training and calculate the F1 score of the generated pseudo-labels when different class ratios and number of objects per image are used (Figure 3a). This plot firstly highlights the benefit of using an accurate class distribution prediction to inform pseudo-label selection. When the predicted class ratio is used, there is a clear shift to higher F1 scores relative to using the naive prior provided by the labelled data. We further see that targeting a mean confidence score of 0.6 results in an F1 score that is close to the peak. Consequently, the proposed approach results in an improvement in the F1 score of 0.06 relative to the original ACT. We also plot how the selected number of objects per image evolves through training for the C2B and C2N scenarios, highlighting how the proposed method responds to the variable reliability of the teacher model (Figure 3b-c). As the pseudo-label accuracy improves with training, the number of objects per image is dynamically increased. We also see that for the more challenging C2N scenario, the number of pseudo-labels per image is lower to account for the poorer performance of the teacher in the target domain.

We further investigate the accuracy of the proposed class ratio prediction method to assess if it is robust to different forms of domain shift. To do so, we utilise the four driving datasets used to assess UDA-OD performance; Cityscapes [18], foggy Cityscapes [19], BDD100k daytime and BDD100k night [20]. The permutation of these datasets into labelled and unlabelled pairs leads to 12 scenarios for predicting class distribution shift (Figure 5). Additionally, we use the Wanderlust dataset [17] to assess realistic class distribution shift encountered across time by an egocentric agent. This dataset contains object annotations of an egocentric videostream collected by a graduate student over nine months of their life. The seven most common classes of bicycle, bus, car, chair, dining table, person and potted plant are considered. Temporally coherent splits of the training data are created by splitting it at the 2.5%, 5%, 10%, 25%. 50%, 75% and 90% mark of the datastream. We use the labels for all data before the split, resulting in 7 scenarios for assessing class distribution prediction (Figure 4).

For each scenario, we report the Kullback-Leibler (KL) divergence between the labelled class ratio, our merged and squared predictions, and the true class ratio. Generally, the squared prediction is much more accurate than using the labelled data alone, reducing KL divergence by 3-4 fold. On the driving scenarios, our method is robust to large and small class distribution shifts and significant appearance changes due to foggy weather and night driving (Figure 5). This emphasises that the representation of semantic context used to predict the class ratio is sufficiently resistant to domain shift. On the Wanderlust scenarios, our prediction remains accurate across a variable labelling budget (Figure 4).

We also investigate the regression models to ascertain why the squared prediction is consistently more accurate than the merged prediction (Figure 5, 4). We find that the intercept coefficients of these models vary significantly across datasets. This implies that the class ratio is influenced by exogenous factors not captured by the CLIP similarity scores. As a result, a model fit to the labelled data will consistently be in error when making predictions on the shifted, unlabelled data. However, this intercept error is strongly correlated with the class distribution shift between the datasets ($r=0.98$ for the driving scenarios in Figure 5). Thus, by predicting the relative change in class ratio between datasets, we are indirectly estimating the intercept error. Scaling up the predicted change in class ratio therefore improves accuracy by incorporating this prediction of the intercept error into our method. In future, prompt learning techniques [25, 27] could be investigated to make the CLIP similarity scores more expressive of the class ratio, avoiding the need to heuristically account for the intercept error.

Using the C2B scenario, we study the independent effect of using a more accurate class ratio and the dynamic adjustment of the number of objects per image. We first introduce the dynamic adjustment of the number of pseudo-labels per image, but continue using the labelled class distribution prior. The resulting 5.1 mAP improvement relative to the original ACT [13] can therefore be accredited to using the mean confidence of the teacher to select the number of pseudo-labels per image (Table III). When the predicted prior is introduced instead of the labelled class ratio, a further improvement of 3.8 mAP results. Lastly, we test the performance of our method if the true class ratio of the unlabelled data was used as the prior, finding that it leads to only a minor increase in performance, from 39.2 to 40.2 mAP (Table III).

We also study the sensitivity of our approach under different settings of the target mean pseudo-label confidence $\tau$. For values between 0.5 and 0.7, our approach maintains performance superior to the strongest baseline (Table IV). These values perform optimally as they lead the number of pseudo-labels per image to converge to the true number of objects per image in the target dataset (Figure 3b-c). We therefore recommend setting $\tau=0.6$ as a starting point on novel scenarios, before confirming that the number of pseudo-labels per image is converging to an appropriate value. In future work, merging class distribution information with probabilistic self-training approaches [14, 4] may reduce sensitivity to key parameters.