CrossRectify: Leveraging Disagreement for Semi-supervised Object Detection

Many of the existing SSOD methods utilize the consistency regularization proposed in semi-supervised learning (SSL), such as CSD [25], ISD [26], PL [27], etc. The key idea of consistency regularization is to require the detector to predict consistently on both weak- and strong- augmented versions of the same input. We point out that the consistency-regularization-based methods can be regarded as the special case of the self-labeling training scheme, because the detector is supervised by the pseudo labels on weak-augmented image predicted by itself. Note that these studies report large performance improvements over the fully-supervised baselines, but we point out that the experimental settings are somehow unfair. In fact, under the same data augmentation, the performance increments brought by consistency regularization are always observed limited. We analyze the reason behind such phenomenon in this paper, and point out the inherent limitations of the self-labeling training scheme, as well as the consistency regularization.

Beyond the consistency regularization, there are a few SSOD works based on the teacher-student mutual learning framework [21]. In an offline or online manner, the pseudo labels are generated by the teacher model, instead of by the student model itself. As for the former, [16, 17] first pre-train the teacher model with available labeled data, then utilize the teacher to annotate the entire unlabeled data. However, the pseudo labels are generated only once and remain fixed during semi-supervised training, thus the final performance of student model is limited by that of the teacher model. As for the latter,  [18, 19, 20] compute the exponential moving average of student as teacher and the teacher is utilized to generate pseudo labels during the semi-supervised training process. However, we observe that the teacher is converged to the student in the late stage of training, which indicates that the labeling process degenerates into the self-labeling manner and suffers from the same limitations.

As another research line of semi-supervised learning, co-training methods [12, 13, 14, 15] are proposed to train two models in a collaborative manner. Each model can learn from the pseudo labels predicted by its counterpart, which seems a promising way to mitigate the limitations of self-labeling methods. The only prior work taking the idea of co-training for the SSOD task is co-rectify [22]. In [22], the pseudo boxes for unlabeled data are first predicted by one detector, then refined by the corresponding predictions from another model, with probability scores and coordinates being averaged. Besides, a recent study [28] also adopts co-training and proposes the cross pseudo supervision (CPS) for the semi-supervised semantic segmentation task, where each model is supervised by the pseudo maps predicted by the other model. We note that CPS can be adapted for the SSOD task. However, we conduct quantitative comparisons to show that both co-rectify and CPS cannot fully exploit the advantages of multiple models and improve the quality of pseudo labels.

In each training iteration, we utilize two detector models $f_{A}$ and $f_{B}$ to predict on the unlabeled inputs, then select the bounding boxes with max probability scores higher than threshold $\tau$, denoted as $\overline{\boldsymbol{y}}_{A}=(\overline{\boldsymbol{p}}_{A},\overline{\boldsymbol{t}}_{A})$ and $\overline{\boldsymbol{y}}_{B}=(\overline{\boldsymbol{p}}_{B},\overline{\boldsymbol{t}}_{B})$. $\overline{\boldsymbol{p}}$ and $\overline{\boldsymbol{t}}$ denote the probability scores and coordinates of all predicted bounding boxes.

For each box in $\overline{\boldsymbol{y}}_{A}$, we search its best match box among all boxes in $\overline{\boldsymbol{y}}_{B}$. For example, for i-th box $(\overline{\mathbf{p}}_{A,i},\overline{\mathbf{t}}_{A,i})$ in $\overline{\boldsymbol{y}}_{A}$, the matching process is formulated as:

where $M(\cdot,\cdot)$ stands for the matching metric and is slightly different for kinds of detectors. For the Faster-RCNN [2], $M((\overline{\mathbf{p}}_{A,i},\overline{\mathbf{t}}_{A,i}),(\overline{\mathbf{p}}_{B,j},\overline{\mathbf{t}}_{B,j}))$ is the area of intersection over union (IoU) between two boxes as:

Specifically, if the IoU areas between $(\overline{\mathbf{p}}_{A,i},\overline{\mathbf{t}}_{A,i})$ and all boxes in $\overline{\boldsymbol{y}}_{B}$ are all below a certain threshold $\delta$, we create a virtual bounding box $\overline{\mathbf{t}}_{B,j^{\ast}}$ to match it, as $(\overline{\mathbf{p}}_{B,j^{\ast}},\overline{\mathbf{t}}_{j^{\ast}})=(\overline{\mathbf{p}}_{A,i},\overline{\mathbf{t}}_{A,i})$ and the matching metric $M(\cdot,\cdot)$ equals 1. For the SSD structure [1], $M((\overline{\mathbf{p}}_{A,i},\overline{\mathbf{t}}_{A,i}),(\overline{\mathbf{p}}_{B,j},\overline{\mathbf{t}}_{B,j}))$ equals 1 if two boxes are based on the same anchor, otherwise 0. Note that $M(\cdot,\cdot)$ for SSD can also be specified as the IoU areas as like that for Faster-RCNN, but we find it more effective to adopt the anchor correspondence in our experiments. For the VoteNet [24], the matching metric $M(\cdot,\cdot)$ is specified as the negative Euclidean distance between the centers of two bounding boxes, computed as:

When the training process ends, we evaluate the performance of one single detector. Moreover, to exploit the different detection abilities, we propose to adopt the weighted boxes fusion (WBF) [29] strategy to ensemble two predictions sets. The corresponding performance is denoted as CrossRectify^∗.

As discussed in Section 2.2, some recent SSOD works [16, 17, 18, 19, 20] are established on the offline/online teacher-student mutual learning [21]. These works can alleviate the self-errors in self-labeling process by introducing another separate object detector for generation of pseudo labels. However, as for offline methods [16, 17], the pseudo labels are generated only once and remain fixed when training student detector, so the student performance is upper-bounded by that of teacher. For instance, we conduct experiments with Faster-RCNN-FPN detector on the MS-COCO benchmark dataset under 10% degree of supervision. The AP₅₀ performance of teacher detector after fully-supervised pre-training is 23.86%, while that of student detector supervised by teacher detector only increases with a 3.30% absolute gain, far away from the results in Table 4 (34.89% AP₅₀). Similar phenomenon can be observed with respect to the SSD300 structure on the Pascal VOC benchmark dataset in Table 3 (obtaining only a 0.79% AP₅₀ gain).

As for online methods [18, 19, 20], the teacher detector is converged to the student detector and yields similar predictions in the late stage of training, thus the pseudo label generation process degenerates to the self-labeling process and suffers from the same limitations. For instance, we conduct online teacher-student mutual learning based on SSD300 and Pascal VOC. As shown in Fig. 4(a), the average KL-divergence between probability scores predicted by teacher and student detector reaches zero in the last 40k iterations. Correspondingly, the detection performance shown in Table 3 also indicate the ineffectiveness of online teacher-student mutual learning.

Recently, a co-training based SSOD method named co-rectify has been proposed in [22], which is the only prior work taking the idea of co-training in the SSOD task to the best of our knowledge. In co-rectify, the pseudo bounding boxes are first predicted by one detector, then refined by corresponding predictions from another model, with probability scores and coordinates being averaged. Besides, we notice that a recent work proposes cross pseudo supervision (CPS) [28] for the semi-supervised semantic segmentation task and achieves the state-of-the-art performances, where each model directly takes the predictions from the other model as pseudo labels. CPS can be adapted to the SSOD task, as each detector is supervised by the other detector. However, as shown in Fig. 4(b), the precision values of pseudo label of these methods are inferior to that of CrossRectify (conducting semi-supervised training with SSD300 on Pascal VOC). We consider the reason behind such phenomenon lies in that simply averaging multiple predictions (co-rectify) or directly taking predictions from other models as supervision (CPS) cannot fully exploit the advantages of multiple models, in comparison to our cross-rectifying mechanism. Their inferior performances shown in Table 3 and Table 6 also validate our consideration. Besides, we also investigate more alternative strategies on pseudo label rectification and observe that cross-rectifying turns out to be most effective strategy (as detailed in Sec. 5.4).

We evaluate the proposed CrossRectify on two widely-used benchmark datasets, i.e., Pascal VOC[11] and MS-COCO [30]. Pascal VOC has 20 object categories. We take the VOC07 trainval set (5,011 images) as labeled and the VOC12 trainval set (11,540 images) as unlabeled. The detection performance is evaluated on the VOC07 test set (4,952 images) using the VOC style AP₅₀ metric. MS-COCO has 80 object categories. We follow the same settings as that in [16, 18, 20, 22, 31] to randomly sample 1/2/5/10% of the COCO2017 train set (118,287 images) as labeled and take the remaining part as unlabeled. Also, we create five data folds under each degree of supervision, and finally report the mean and standard deviation from five results. The detection performance is evaluated on the COCO2017 val set (5,000 images) using the COCO style AP_50:95 metric.

We follow [32] to conduct experiments on the SUN-RGBD benchmark dataset [33]. We randomly sample 5% of 5,285 training samples as labeled and take the remaining part as unlabeled. The detection performances is evaluated on 5,050 validation samples, using both AP₂₅ and AP₅₀ metrics.

We carry out experiments on the Pascal VOC dataset with two detector structures, that are SSD300 [1] with VGG-16 backbone and Faster-RCNN-FPN [2, 34] with ResNet-50 backbone. The latter structure is also utilized in experiments on the MS-COCO dataset. As for 3D detection, we utilize VoteNet [24] with PointNet++ backbone [35].

We utilize the Pytorch implementation²²2https://github.com/amdegroot/ssd.pytorch to train SSD300 on Pascal VOC. Within a total of 120k iterations, we conduct fully-supervised training in the first 12k iterations as warm-up. We ramp-up/down the unsupervised loss weight $\lambda_{U}$, and set threshold $\tau$ and batch size as 0.5 and 32 according to [26]. We utilize the Detectron2 platform³³3https://github.com/facebookresearch/detectron2 to train Faster-RCNN-FPN on Pascal VOC. We train a total of 36k iterations with first 6k being fully-supervised warm-up, and adopt the same data augmentation strategy as that in [18]. We set $\lambda_{U}$ as 2.0 and threshold $\tau$ as 0.7 following [18]. The batch sizes for labeled data and unlabeled data are both 16. The threshold $\delta$ on matching metric is 0.5. To show the generality across different platforms of our CrossRectify method, we adopt the MMdetection⁴⁴4https://github.com/open-mmlab/mmdetection to train Faster-RCNN-FPN on MS-COCO. We train a total of 180k iterations and adopt the data augmentation strategies in [20]. Under 1% degree of supervision, we conduct fully-supervised warm-up in the first 80k iterations to ensure the stability of training. We set $\lambda_{U}$ as 4.0 and threshold $\tau$ as 0.9 according to [20]. The batch sizes for labeled data and unlabeled data are respectively 8 and 32. The threshold $\delta$ on matching metric is 0.5. To train VoteNet on SUN-RGBD, we first conduct fully-supervised pre-training by 900 iterations, then conduct semi-supervised training by 1k iterations, following [32].

Note that the exponential moving average (EMA) strategy is commonly used for the pseudo label-based methods, since a detector model aggregated by EMA can yield more conservative and stable predictions than the detector itself [18, 19, 20]. For fair comparisons, We follow such common practice in our experiments, as we utilize the EMAs of two detectors to conduct the detector feed-forward process.

Table 3 shows the results of our CrossRectify method compared with other training frameworks on Pascal VOC. As for the SSD300 detector, we take the 71.73% AP₅₀ performance of fully-supervised training as the baseline. As can be seen, our proposed method can obtain a 73.65% AP₅₀ result, while the results of all compared approaches are only about 72.50%. Such comparison validates the effectiveness of our CrossRectify on improving pseudo label quality. Besides, the WBF-merged [29] results from both detectors can further boost the final performances, denoted as CrossRectify^∗. Under the mix-up data augmentation [10], our CrossRectify method can still show better performance than the self-labeling-based method, ISD [26] (by a 1.41% margin).

As for the Faster-RCNN-FPN detector, we compare CrossRectify with previous methods, and our method can improve the AP₅₀ result with a 4.66% margin over fully-supervised baseline, achieving the state-of-the-art performance. Note that Unbiased Teacher [18] reports the performance based on the COCO style AP₅₀ metric in their paper. For a fair comparison, we instead adopt the VOC style AP₅₀ metric for Unbiased Teacher, and the AP₅₀ raises from 77.37% to 80.51%, still surpassed by that of CrossRectify with a 0.59% margin.

Table 4 shows the performances of our CrossRectify method compared with previous state-of-the-arts on the MS-COCO dataset. Under different degrees of supervision 1%, 2%, 5% and 10%, the proposed CrossRectify can obtain consistent and substantial improvements, surpassing those of Soft Teacher [20] by 1.46%, 0.50%, 0.96%, and 0.85% AP_50:95 margins. These comparative results further verify the effectiveness of the proposed method. Moreover, we visualize the pseudo bounding boxes for some unlabeled images in Fig. 5. Compared with self-labeling training scheme, our method can yield more accurate pseudo boxes.

Table 5 shows the comparison with all previous works (i.e., SESS [36] and 3DIoUMatch [32]) on the SUN-RGBD benchmark dataset. Under 5% degree of supervision, the performance of our CrossRectify can outperform that of the state-of-the-art 3DIoUMatch method by 3.1 AP₂₅ and 1.9 AP₅₀ margins. The results validate the efficiency of CrossRectify on 3D semi-supervised object detection task. We omit the WBF-merging performance CrossRectify^∗, because WBF does not support on 3D bounding boxes with different rotation angles.

Now we investigate the alternative strategies on rectifying the pseudo labels, including (a) only utilizing the intersection of two prediction sets from two detectors as pseudo labels, which is composed of the objects classified as the same classes. (b) Only utilizing the difference set of two prediction sets from two detectors as the pseudo label, which is composed of objects classified as different classes. (c) Directly taking all the predicted bounding boxes from the other detector as the pseudo label, which turns out to be the cross pseudo supervision (CPS) [28]. As observed in Table. 6, all these strategies cannot ensure the pseudo label quality and finally lead to inferior performances.

Our proposed CrossRectify method can be easily extended to train more than two detectors simultaneously. Specifically, during the pseudo label rectification process, each pseudo bounding box is re-labeled by the majority of all predicted classes, and re-located by the average of all predicted coordinates. As displayed in Table 7, CrossRectify over four SSD300 detectors can bring a 0.06% AP₅₀ improvement for each detector on average. We believe that two-detector scenario is already able to cross-rectify the misclassified pseudo labels adequately.