Dynamic Supervisor for Cross-dataset Object Detection

Dataset. We sample images from the MS COCO dataset [18] to establish two mini datasets, namely miniCOCO-Alpha and miniCOCO-Beta. Each of them contains 8K images for training and 5K images for validation. We preserve 10 category labels on miniCOCO-Alpha: {car, handbag, truck, light, bench, chair, horse, bicycle, cup, plant}, and we preserve an additional 10 category labels on miniCOCO-Beta: {person, bottle, motorcycle, bird, boat, umbrella, sheep, wine glass, table, tv}. For simplicity, miniCOCO-Alpha is denoted as $(\mathbf{I}^{\alpha},\mathbf{C}^{\alpha}_{1})$ and miniCOCO-Beta is denoted as $(\mathbf{I}^{\beta},\mathbf{C}^{\beta}_{2})$, where $\mathbf{I}$, $\mathbf{C}_{1}$ and $\mathbf{C}_{2}$ denote the image set, the first 10-category annotation set, and the second 10-category annotation set, respectively. Numbers 1 and 2 correspond to the first 10-category and the second 10-category, respectively. This setting models the central issue when multiple datasets with different category sets are merged.

As we discussed in Section 1, the key step in cross-dataset training for object detection involves seeking the accurate annotation sets $\mathbf{C}_{2}^{\alpha}$ and $\mathbf{C}_{1}^{\beta}$, and thus, transforming the cross-dataset $(\{\mathbf{I}^{\alpha},\mathbf{I}^{\beta}\},\{\mathbf{C}^{\alpha}_{1},\mathbf{C}^{\beta}_{2}\})$ into a complete form $(\{\mathbf{I}^{\alpha},\mathbf{I}^{\beta}\},\{\mathbf{C}^{\alpha}_{1},\mathbf{C}^{\alpha}_{2},\mathbf{C}^{\beta}_{1},\mathbf{C}^{\beta}_{2}\})$. Next, we conduct a detailed experimental analysis on the generation of the missing annotation sets, $\mathbf{C}_{2}^{\alpha}$ and $\mathbf{C}_{1}^{\beta}$, with high quality.

Training Details. ResNet-50 [32] pretrained on ImageNet [33] is chosen as the backbone of the SSD in this section. During training, we use five image scales {448, 480, 512, 544, 576} (the aspect ratio of the image is 1:1) randomly. The network is trained using the stochastic gradient descent (SGD) algorithm for 100 epochs with 0.9 momentum, 0.0005 weight decay, and 32 batch sizes on two NVIDIA V100 GPUs. The initial learning rate is 0.0026 and it decays at epochs 66 and 83. The loss functions used during training are the cross-entropy loss for the classification branch and the smooth-L1 loss for the regression branch.

Inference Details. During the inference phase, the input image is resized to 576 $\times$ 576, after which it is input into the entire network to output the predicted bounding boxes with a predicted category. To filter out the redundant background bounding boxes, the confidence threshold is set to 0.01, and non-maximum suppression (NMS) is applied, with an IoU threshold of 0.6 per class, when evaluating the detection performance of the network.

A submodel trained using miniCOCO-Alpha generates the first 10-category annotation set for miniCOCO-Beta, which is denoted as $\mathbf{iC}_{1}^{\beta}$. Correspondingly, another submodel trained using miniCOCO-Beta generates the second 10-category annotation set for miniCOCO-Alpha, which is denoted as $\mathbf{iC}_{2}^{\alpha}$. It is worth noting that each element in $\mathbf{iC}_{2}^{\alpha}$ and $\mathbf{iC}_{1}^{\beta}$ consists of bounding box coordinates, a category label, and the corresponding confidence score. Additionally, the confidence score of each element should be larger than the threshold $T_{c}$. The next conventional operation involves integrating $\mathbf{iC}_{2}^{\alpha}$ and $\mathbf{iC}_{1}^{\beta}$ into the datasets and establishing a complete one: $(\{\mathbf{I}^{\alpha},\mathbf{I}^{\beta}\},\{\mathbf{C}^{\alpha}_{1},\mathbf{iC}^{\alpha}_{2},\mathbf{iC}^{\beta}_{1},\mathbf{C}^{\beta}_{2}\})$. Soft-label training is the usual choice for this type of “noisy-annotated” dataset. During training, the loss function for the classification branch is expressed as follows: $L_{cls}=-\sum_{c=0}^{20}y_{c}\log p_{c}$. The label for a pseudo-annotation of category $j$ and confidence $s$ is a soft form ${\boldsymbol{y}}=\left[y_{0},y_{1},...,y_{20}\right]^{T}$, where $y_{0}=1-s$ and $y_{j}=s$. Through the dataset $(\{\mathbf{I}^{\alpha},\mathbf{I}^{\beta}\},\{\mathbf{C}^{\alpha}_{1},\mathbf{iC}^{\alpha}_{2},\mathbf{iC}^{\beta}_{1},\mathbf{C}^{\beta}_{2}\})$, the soft-label-training submodel achieves 34.4$\%$ mAP on the validation set (5K images from miniCOCO-Alpha and 5K images from miniCOCO-Beta).

Because miniCOCO-Alpha and miniCOCO-Beta are both sampled from MS COCO, we can obtain the true annotations directly, namely $\mathbf{tC}_{2}^{\alpha}$ and $\mathbf{tC}_{1}^{\beta}$, where $\mathbf{t}$ denotes true. Using the dataset $(\{\mathbf{I}^{\alpha},\mathbf{I}^{\beta}\},\{\mathbf{C}^{\alpha}_{1},\mathbf{tC}^{\alpha}_{2},\mathbf{tC}^{\beta}_{1},\mathbf{C}^{\beta}_{2}\})$, a theoretically optimal SSD model is trained, and it achieves 38.2$\%$ mAP on the same validation set.

We then analyze the predictions of both models on the validation set. As shown in Figure 2, the statistical characteristics have attracted our attention. The quantity of true positives (TP) from the soft-label-training model is similar to that from the theoretically optimal model (the quantity from the theoretically optimal model is approximately 5% higher than that from the soft-label-training model), which means that the maximum recall between the two models is close. However, when we observe the quantities of false positives (FP), we find that the theoretically optimal model outputs much more FP than the soft-label-training model, which is counter-intuitive.

We suggest that the performance gap between the two models is caused by the distribution of detection predictions rather than the quantity of detection outputs. Figure 3 shows the output distribution of category person, where the figure on the left is for the theoretically optimal model and the one on the right is for the soft-label-training model. The TP from the theoretically optimal model tend to have a higher confidence score than those of the soft-label-training model. In contrast, more TP from the soft-label-training model are concentrated in the low-score region and severely mixed up with FP, which should be the direct cause of the performance gap between the two detectors. Quantitatively, the TP of the theoretically optimal model that have a confidence score higher than 0.2 account for 68.2% of all TP, whereas those of the soft-label-training model account for a mere 36.3%. As for FP with a confidence score higher than 0.2, their number in the theoretically optimal model is almost four times that in the soft-label-training model.

To verify the difference between the two models, we conduct hard-label training on the dataset $(\{\mathbf{I}^{\alpha},\mathbf{I}^{\beta}\},\{\mathbf{C}^{\alpha}_{1},\mathbf{iC}^{\alpha}_{2},\mathbf{iC}^{\beta}_{1},\mathbf{C}^{\beta}_{2}\})$ for further analysis. The true positive distributions of category person for the theoretically optimal model, soft-label-training model, and hard-label-training model are illustrated in Figure 4. The distribution curves of the theoretically optimal model and the hard-label-training model are similar. They output more TP in the high-score region, and the peaks of their curves are more to the right than the peak of the soft-label-training model’s curve. The TP of the hard-label-training model that have a confidence score higher than 0.2 account for 64.7% of all TP, which is close to that of the theoretically optimal model (68.2%). Additionally, the high-scoring (>0.9) TP account for 6.8% and 9.5% for the theoretically optimal model and the hard-label-training model, respectively, which is higher than that of the soft-label-training model (2.1%). Based on these statistics, we suggest that the prediction of the hard-label-training model (the theoretically optimal model is also trained using hard labels) has a higher mean score and contains more FP, whereas that of the soft-label-training model has a lower mean score and contains fewer FP.

Considering the different characteristics of hard-label training and soft-label training, we propose taking advantage of these two strategies to improve the quality of annotations $\mathbf{iC}_{2}^{\alpha}$ and $\mathbf{iC}_{1}^{\beta}$ from two perspectives. 1) We can expand two annotation sets by adding new predictions that can increase the recall of annotations. 2) We can shrink two annotation sets by filtering out unreliable predictions to improve the precision of annotations.

Specifically, images from $\mathbf{I^{\alpha}}$ and $\mathbf{I^{\beta}}$ are input into the hard-label-training model and the soft-label-training model to obtain new detection results, $\left(\mathbf{H}_{2}^{\alpha},\mathbf{H}_{1}^{\beta}\right)$ and $\left(\mathbf{S}_{2}^{\alpha},\mathbf{S}_{1}^{\beta}\right)$, respectively. $\mathbf{H}$ denotes the predictions produced using the hard-label-training model, and $\mathbf{S}$ denotes the predictions produced using the soft-label-training model. Therefore the expanding operation using hard-label training is formulated as follows:

where $\mathbf{hC}_{2}^{\alpha+}$ denotes the expanded annotation set on dataset miniCOCO-Alpha and $\mathbf{hC}_{1}^{\beta+}$ denotes that on miniCOCO-Beta. The operation $\setminus$ minuses the annotation in $\mathbf{H}_{2}^{\alpha}$, which has an IoU larger than a threshold $T_{e}$, with any annotation in $(\mathbf{C}_{1}^{\alpha}\cup\mathbf{iC}_{2}^{\alpha})$. Similarly, the expanding operation using soft-label training is formulated as follows:

We formulate the shrinking operation using hard-label training as follows:

where $\mathbf{hC}_{2}^{\alpha-}$ denotes the shrunk annotation set on dataset miniCOCO-Alpha and $\mathbf{hC}_{1}^{\beta-}$ denotes that on miniCOCO-Beta. The operation $\cap$ preserves the annotation in $\mathbf{iC}_{2}^{\alpha}$, which has an IoU larger than a threshold $T_{s}$, with any annotation in $(\mathbf{H}_{2}^{\alpha}\setminus\mathbf{C}_{1}^{\alpha})$. It is inspired by the intuition that the bounding boxes detected using both detectors are more reliable. Similarly, the shrinking operation using soft-label training is formulated as follows:

Based on the true annotations $\mathbf{tC}_{2}^{\alpha}$ and $\mathbf{tC}_{1}^{\beta}$, we can obtain the relative variation of recall and precision for new annotation sets when we expand or shrink them using different models. The results are listed in Table 1, and they confirm our conclusion that the hard-label-training model is conducive to improving the recall of annotation sets, whereas the soft-label-training model is useful for improving the precision of annotation sets. The performance of the models trained using different annotation sets is also shown in Table 1.

The structure of the proposed dynamic supervisor framework is shown in Figure 5, and two datasets annotated with different category sets are merged in our illustration (the notations in Section 3 are followed). There are three steps for generating the final annotation set. The first step of the dynamic supervisor framework is similar to the one employed in previous studies [16, 8], where two detection models are trained using miniCOCO-Alpha or miniCOCO-Beta individually, after which each detection model generates an initial annotation set for images from the other dataset (Initial labeling, as shown in Figure 5).

By combining the ground truths ($\mathbf{C}_{1}^{\alpha}$, $\mathbf{C}_{2}^{\beta}$) with the generated annotation sets ($\mathbf{iC}_{2}^{\alpha}$, $\mathbf{iC}_{1}^{\beta}$), miniCOCO-Alpha and miniCOCO-Beta can be merged. One optional structure of the dynamic supervisor framework is shown in Figure 5 (a), where two datasets are merged after the generation of the initial annotation sets. Next, in the second step (Expanding in Figure 5), a detection model trained using $(\{\mathbf{I}^{\alpha},\mathbf{I}^{\beta}\},\{\mathbf{C}^{\alpha}_{1},\mathbf{iC}^{\alpha}_{2},\mathbf{iC}^{\beta}_{1},\mathbf{C}^{\beta}_{2}\})$ generates new annotations through hard-label training to expand the initial annotation sets ($\mathbf{iC}_{2}^{\alpha}$, $\mathbf{iC}_{1}^{\beta}$) into ($\mathbf{hC}_{2}^{\alpha+}$, $\mathbf{hC}_{1}^{\beta+}$). In the third step (Shrinking in Figure 5), another detection model trained using $(\{\mathbf{I}^{\alpha},\mathbf{I}^{\beta}\},\{\mathbf{C}^{\alpha}_{1},\mathbf{hC}_{2}^{\alpha+},\mathbf{hC}_{1}^{\beta+},\mathbf{C}^{\beta}_{2}\})$ filters out unreliable annotations through soft-label training to shrink the expanded annotation sets ($\mathbf{hC}_{2}^{\alpha+}$, $\mathbf{hC}_{1}^{\beta+}$) into ($\mathbf{sC}_{2}^{\alpha-}$, $\mathbf{sC}_{1}^{\beta-}$). Based on the dataset $(\{\mathbf{I}^{\alpha},\mathbf{I}^{\beta}\},\{\mathbf{C}^{\alpha}_{1},\mathbf{sC}_{2}^{\alpha-},\mathbf{sC}_{1}^{\beta-},\mathbf{C}^{\beta}_{2}\})$, we can obtain the final model for cross-dataset object detection.

In the first structure of the dynamic supervisor discussed above, the detection models (in the second and third steps) must update their supervision information independently. Considering the risk that the models are prone to overfitting to noisy annotations in this “self-annotated mechanism”, we propose another structure, which is a “cross-annotated mechanism,” as shown in Figure 5 (b). Unlike in the first structure, the two datasets are not merged after the generation of the initial annotation sets in the first step. When the two original datasets are augmented with the generated annotation sets $(\mathbf{I}^{\alpha},\mathbf{C}^{\alpha}_{1},\mathbf{iC}^{\alpha}_{2})$ and $(\mathbf{I}^{\beta},\mathbf{iC}^{\beta}_{1},\mathbf{C}^{\beta}_{2})$, they are responsible for training two models individually through hard-label training. The two hard-label-training models then generate new annotations for images from the other dataset to expand their initial annotation set in the second step. Because these two models are trained using the augmented datasets, their detection distributions are different from those of the previous two models in the first step. Therefore, they can detect objects that are ignored, thereby improving the recall of the annotation sets. Although the recall is improved, the expanding operation probably results in a decrease in precision. Accordingly, two other models are then trained through soft-label training using the expanded annotation sets ($(\mathbf{I}^{\alpha},\mathbf{C}^{\alpha}_{1},\mathbf{hC}_{2}^{\alpha+})$ or $(\mathbf{I}^{\beta},\mathbf{hC}_{1}^{\beta+},\mathbf{C}^{\beta}_{2})$). The new annotations generated using two soft-label-training models are used to filter out the unreliable annotations of the expanded annotation sets in the third step. The objects detected through the two different models are more likely to be TP than FP. Therefore, there will be an improvement in the precision of annotation sets after the shrinking operation.

Finally, miniCOCO-Alpha and miniCOCO-Beta are merged, and the final model is trained using this multiple-updated dataset: $(\{\mathbf{I}^{\alpha},\mathbf{I}^{\beta}\},\{\mathbf{C}^{\alpha}_{1},\mathbf{sC}_{2}^{\alpha-},\mathbf{sC}_{1}^{\beta-},\mathbf{C}^{\beta}_{2}\})$. We suppose that the recall and precision of the annotation sets can be promoted comprehensively through the multiple update steps.

Cross-dataset Setting. To verify the effectiveness of the proposed dynamic supervisor framework, we define three different combinations of datasets that cover diverse scenarios. In Table 2, we summarize the dataset settings used in our experiments. Setting A contains two mini datasets sampled from MS COCO, which we used in Section 3. Setting B combines PASCAL VOC [17] (20 categories) and MS COCO [18] (without VOC categories) to verify the performance of the proposed method on large-scale datasets. Setting C consists of three datasets from different scenarios: PASCAL VOC is a general dataset containing 20 common categories, SUN-RGBD [19] includes 18 categories of indoor scenes, and LISA-Signs [20] is a driving scenes dataset, which contains 4 traffic signs. This setting combines multiple datasets with large gaps, and frequent scene overlapping exists among these datasets.

Comparison with Other Methods. We apply the proposed dynamic supervisor framework to the Faster-RCNN [36] model. ResNet-50 [32] with FPN [37] is used as the backbone, and it is pretrained on ImageNet [33]. We follow the implementation presented by  [38], where input images are resized to keep their shorter side as one of $\left\{640,672,704,736,768,800\right\}$ randomly and their longer side less than or equal to 1,333 during training.

For setting B, we evaluate our model on two datasets: COCO and MIX [8] (a combination of VOC and COCO). The experimental results are shown in Table 3. Compared with the naive combination of the two datasets (the first row in Table 3), the partial loss method proposed by  [34] only takes a small step forward. The following pseudo-labeling method proposed by  [8] achieves much better performance (50.3% vs. 42.6%, 52.2% vs. 43.7%), which demonstrates the effectiveness of pseudo-annotation. The performance of the static supervisor is comparable to that of the one proposed by  [8] (53.3% vs. 50.3%, 51.5% vs. 52.2%). However, the dynamic supervisor framework goes further and achieves a new state-of-the-art performance (56.2% on COCO and 55.8% on MIX). This performance demonstrates the effectiveness of the dynamic supervisor framework in a large-scale dataset setting.

For setting C, there are four overlapping categories between PASCAL VOC and SUN-RGBD. Therefore, the entire merged dataset has 38 categories in total. A validation set [8] of 1,500 images (from three datasets) is annotated for all 38 categories for evaluation. The results of the experiments conducted in this setting are listed in Table 4. The static supervisor framework achieves a performance that is comparable to that of the pseudo-labeling method [8] (60.7% vs. 61.1%). Based on the static supervisor framework, the proposed dynamic supervisor framework enhances the performance further by 1% in mAP (61.7% vs. 60.7%). The best performance in this setting demonstrates the generality of our method in a complicated setting (multiple datasets with frequent scene overlapping).

Comparing the promotions brought by the dynamic supervisor in setting B and setting C, the performance superiority in setting B is larger than that in setting C. This inapparent superiority in setting C results from the slight annotation missing. The more annotations are lost, the more promotions the dynamic supervisor will bring. Quantitively, there are 4.8 predicted instances per image for setting B, whereas there are only 1.0 predicted instances per image for setting C. In other words, the training task in setting C is closer to a fully-supervised problem, resulting in minor performance gaps among all competitors.

We follow the controlled experimental setting in Section 3.1 to analyze the proposed dynamic supervisor framework. The implementation details are already described in Section 3.1. Ablation results are shown in Table 5 - Table 7, Figure 6, and Figure 7. Details about ablation studies are discussed in the following.

Dynamic Hyperparameters: Figure 6 shows the detection performance of models trained using initial annotation sets with a different confidence threshold $T_{c}$. It is hard for us to determine which threshold is the best (it should be between 0.15 and 0.3 in this setting), especially when encountering other datasets or detection frameworks. Meanwhile, considering the recognition abilities of a detection model vary from one category to another, we propose selecting the confidence threshold adaptively. The confidence threshold for each category is determined when the F1-score of this category reaches the maximum on the validation set. This method achieves the best performance, as shown in Figure 6, which is simple but effective.

Figure 7 shows the detection performance for different choices of $T_{e}$ and $T_{s}$. All these hyperparameter combinations can improve the detection performance compared with the baseline model (34.4% mAP). From Eq. (1) and (4), we can observe that a larger $T_{e}$ introduces more new pseudo-annotations in the expanding operation. Additionally, a smaller $T_{s}$ preserves more old pseudo-annotations in the shrinking operation. As shown in Figure 7, it is better to apply a large $T_{e}$ to obtain more recall improvement and a small $T_{s}$ to avoid many true annotations being filtered out. In the remainder of this paper, we use $T_{e}=0.7$ and $T_{s}=0.5$ for ablation studies.

Dynamic Supervisor vs. Static Supervisor: We report the detection performance of models with different operations in Table 5. The model trained using the naive combination of two datasets, without any other strategies, performs the worst. When the static supervisor framework is applied, the corresponding model enhances the performance by 7.8% in mAP, showing the effect of pseudo-labeling on the cross-dataset object detection task. Furthermore, a dynamic supervisor framework is proposed to update the initial annotation sets multiple times using two types of submodels. From the last three models shown in Table 5, the promotions resulting from the expanding operation, the shrinking operation, or both are clear. In Table 6, we record the variation in recall and precision when new submodels apply expanding, shrinking, or a combination of both to the annotation set. A single expanding operation on the annotation sets increases the recall but decreases the precision, and the shrinking operation is simply the opposite. Nevertheless, when we combine the advantages of these two operations and apply them sequentially, both the recall and precision can be improved.

Cross-annotated vs. Self-annotated: In Section 4.1, we introduce two types of the dynamic supervisor framework, which is a “self-annotated mechanism” for the first one and a “cross-annotated mechanism” for the second one. In the self-annotated mechanism, two datasets are merged, and the detection model is trained using this merged dataset to expand or shrink the initial annotation sets after they are generated. In the cross-annotated mechanism, there are two models trained using their respective datasets, which are already augmented with annotation sets, after which they are utilized to expand or shrink the annotation set of each other. The detection performance of these two types of dynamic supervisors is shown in Table 7. When the expanding operation is applied to the annotation set, both mechanisms result in performance improvements. However, when the shrinking operation is applied subsequently, the detection performance of the self-annotated mechanism tends to degrade (mAP decreases from 34.8% to 34.6%). We suggest that the model trained using a noisy dataset is prone to overfitting to incorrect annotations. Consequently, in the self-annotated mechanism, the knowledge learned from such a noisy dataset is that it is difficult to eliminate the noise of this dataset continuously. In contrast, the cross-annotated mechanism avoids this dilemma and can progressively improve performance (mAP increases from 34.4% to 35.5% step by step).

Operation Sequence: There are two operations in the proposed dynamic supervisor framework for improving the recall or precision of the annotation set. Here, we attempt to update the annotation set through a different sequence of operations to explore the relationship between quality variation and performance improvement. The results of the experiments on the two types of operation sequences are shown in Table 6. First, and most importantly, the two types of operation sequences can improve the quality of the annotation set and achieve similar detection performance (35.5% vs. 35.2%). However, the different operation sequences would result in different improvements in recall and precision. When we first expand the annotation set and then shrink it, we obtain more improvement in precision and less improvement in recall. This is because the later shrinking operation is prone to missing TP. Therefore, it suppresses recall. Conversely, the improvement in the recall will be higher. The results show the flexibility of the dynamic supervisor framework, and the goal is to improve the quality of the annotation set comprehensively.

Finally, we visualize the pseudo-labeling results in different steps of our dynamic supervisor framework, as shown in Figure 8. These images are sampled from miniCOCO-Beta, and their original ground-truth boxes are not labeled here. As described in previous sections, the pseudo-labeling results of a single model are biased, and it is difficult for a single model to achieve high recall and high precision (the first row of Figure 8). Therefore, we propose updating the annotations set multiple times in a dynamic framework. The expanding operation is effective for increasing the number of TP. However, it also facilitates the introduction of new FP (the second row of Figure 8). Consequently, the shrinking operation is applied to discard them and obtain a cleaner annotation set (the third row of Figure 8). However, compared with the ground truth boxes, objects, such as the cup (on the table) and handbag (hanging on the sofa), which are small or hard to distinguish, still need to be found.