Boosting Semi-Supervised 3D Object Detection with Semi-Sampling

Gt-sampling is a data augmentation method introduced by [35], which is widely used in outdoor 3D object detection. It uses ground truth bounding boxes to crops ground truth samples from labeled frames, and the cropped object samples will be collected to generate a gt database $\mathcal{G}=\{s_{i}^{l}\}_{i=1}^{N_{gt}}$. $N_{gt}$ is the number of ground truth boxes in all labeled frames and $s_{i}^{l}$ denotes the $i^{th}$ ground truth sample. For $s_{i}^{l}$, it consists of the corresponding point clouds ${p}_{i}^{l}$ in labeled frames and its ground truth bounding box ${b}_{i}^{l}$. In the training stage, gt-sampling will randomly select ground truth samples from $\mathcal{G}$ and pastes them to the current frame. To prevent pasted samples from overlapping with existing objects in the current frame, collision detection is needed.

For the KITTI dataset, we directly follow existing outdoor 3D object-detecting methods to build a gt database. For indoor scenes, as far as we know, we are the first to use gt-sampling. There are some differences between outdoor and indoor scenes. In indoor scenes, 3D object bounding boxes may overlap with each other, such as a chair under a table. Using ground truth bounding boxes to crop the gt samples will involve points from other objects, resulting in a noisy gt database. For the ScanNet[4] dataset, as it provides an instance mask for each ground truth object, we can use instance masks instead of gt boxes to crop gt samples. In this way, only points belonging to the instance are cropped. For the SUN-RGBD[25] dataset, there is no instance mask for each object, so we have to use ground truth bounding boxes to crop gt samples. As a result, the gt database of SUN-RGBD is not clean, which will produce unreal scenes when performing gt-sampling. In the experiments on SUN-RGBD, we find that gt-sampling can improve model performance at the early training epochs. However, we observe accuracy degradation at the end of training. We address this problem by using the fade strategy [29], which disables gt-sampling when the model is near convergent. Please refer to Table 6 for more details.

To build a pseudo database, we firstly pre-train a backbone detector on labeled frames $\{\boldsymbol{x}^{l}_{i},\boldsymbol{y}^{l}_{i}\}_{i=1}^{N_{l}}$. Then, the pre-trained detector is used to forward unlabeled frames$\{\boldsymbol{x}^{u}_{i}\}_{i=1}^{N_{u}}$. The predictions are used as pseudo labels $\{\boldsymbol{\tilde{y}}^{u}_{i}\}_{i=1}^{N_{u}}$. Different from the gt database, we use pseudo labels to crop pseudo samples from unlabeled frames. With these pseudo samples, we can generate a pseudo database $\mathcal{P}=\{s_{i}^{u}\}_{i=1}^{N_{pse}}$, where $N_{pse}$ is the number of all pseudo samples and $s_{i}^{u}$ denotes the $i^{th}$ pseudo sample. For $s_{i}^{u}$, it consists of corresponding point clouds ${p}_{i}^{u}$ in unlabeled frames, predicted bounding box ${b}_{i}^{pred}$, and predicted classification score ${c}_{i}^{pred}$. Since there are many low-quality pseudo labels, when using the pseudo database, we will filter out high-quality pseudo samples by their scores.

For gt-sampling on labeled frames, we follow the vanilla gt-sampling [35]. For pseudo-sampling on labeled frames, we randomly select some pseudo samples along with their pseudo labels from the pseudo database. The pseudo labels of selected pseudo samples serve two purposes. Firstly, they are used to perform collision detection between pseudo samples and ground truth objects on labeled frames. Secondly, we use them as labels of pseudo samples in labeled frames to supervise the student network at the training time, which requires the pseudo-label to be of high quality. Therefore, we set a pseudo label score threshold $\tau_{pseudo\_sample}$. When performing pseudo-sampling, we only use pseudo samples whose pseudo label scores $c>\tau_{pseudo\_sample}$.

When employing semi-sampling on unlabeled frames, there are no ground truth bounding boxes on unlabeled frames for collision detection with pasted samples. Considering that pseudo labels can provide approximate localization of foreground objects in unlabeled frames, we directly use them for collision detection. However, the massive pseudo labels may take up quite a lot of space. So we set a threshold $\tau_{unlabeled\_frame}$ to filter the pseudo labels in unlabeled frames with pseudo label scores $c>\tau_{unlabeled\_frame}$. It is worth noting that the ground truth labels and pseudo labels of the pasted samples on the unlabeled frames will not be used to supervise the student network. All supervising information of unlabeled frames is produced by the teacher network online.

When performing semi-sampling, we select several samples for each category from the gt database or pseudo database. However, indoor scenes are crowded and there are many categories, such as 18 for ScanNet and 10 for SUN-RGBD. If we choose samples according to a fixed category queue like outdoor methods, there may be no space for the samples of the category at the end of the queue, which will cause inter-class sample imbalance. To solve the issue, we simply shuffle the category queue every time we use semi-sampling.

For indoor scenes, we evaluate our method on ScanNet[4] and SUN-RGBD[25]. ScanNet is an indoor dataset consisting of 1,513 reconstructed meshes, among which 1,201 are training samples and the rest are validation samples. SUN-RGBD contains 10,335 RGB-D images of indoor scenes, which are split into 5,285 training samples and 5,050 validation samples. We follow a standard evaluation protocol [15] by using mean Average Precision (mAP) under iou thresholds of 0.25 and 0.50.

For outdoor scenes, we use KITTI[5] for evaluation. KITTI is a popular dataset for autonomous driving, which consists of fine annotations for 3D object detection. There are 7481 outdoor scenes for training and 7518 for testing. The training samples are generally divided into a training split of 3712 samples and a validation split of 3769 samples. We report the mAP with 40 recall positions, with a rotated IoU threshold of 0.7, 0.5, and 0.5 for the three classes, car, pedestrian, and cyclist, respectively.

Omni-supervised learning[17] is a special regime of semi-supervised learning, which uses all labeled data as well as large-scale unlabeled data. It is lower-bounded by the accuracy of training on all labeled data and it aims to boost the fully-supervised methods. To validate the effectiveness of our method under the omni-supervised setting, we construct the Omni-KITTI dataset with KITTI raw data. Specifically, KITTI raw data consists of many sequences containing all frames that emerge on the KITTI training and validation set. Therefore, we can use the KITTI training set as all labeled data (including 3712 samples) and take the remaining data that excludes KITTI validation sequences as unlabeled data (including 33507 samples). The final performance is evaluated on the KITTI validation set with the same evaluation protocol as the KITTI dataset.

We re-implement the indoor 3DIoUMatch [30] on the OpenPCDet[28], achieving similar results as the official repo. Then we use the 3DIoUMatch as our baseline. In 3DIoUMatch, the backbone detector is an IoU-aware VoteNet which is a VoteNet equipped with a 3D IoU estimation module. Our backbone detector is also the same as 3DIoUMatch. All the experiments are trained on 4 GPUs. At pre-training stage, we train the IoU-aware VoteNet with a batch size of 8 and use the same data augmentation as 3DIoUMatch[30]. The IoU-aware VoteNet is optimized by an adam optimizer with an initial learning rate of 0.008 for 900 epochs. The learning rate is decayed by a factor of 0.1 at the $400^{th}$, $600^{th}$, $800^{th}$ epoch. The pre-trained model is not only used for the initialization of the training stage but also used to generate pseudo labels for unlabeled data and build a pseudo database. At the training stage, we train our method with a batch size of $\{4+8\}$, where 4 is the number of labeled frames and 8 is the number of unlabeled frames. For the ScanNet dataset, our model is optimized by an adam optimizer with an initial learning rate of 0.004 for 1000 epochs. We decay the learning rate by 0.3, 0.3, 0.1, 0.1 at the $400^{th}$, $600^{th}$, $800^{th}$, $900^{th}$ epoch, following the training strategy of 3DIoUMatch. For the SUN-RGBD dataset, the training is time-consuming. To speed up the convergence, we train our model for 300 epochs with an adam one-cycle optimizer and a maximum learning rate of 0.008. At the inference stage, we use the student IoU-aware VoteNet to forward input data.

There is an official implementation of outdoor 3DIoUMatch on OpenPCDet. We can directly use it as our baseline. We employ PV-RCNN as our backbone detector following 3DIoUMatch. At pre-training stage, the backbone detector is optimized by an adam one-cycle optimizer with the batch size 8, learning rate 0.01 for 80 epochs. For all ratios of labeled data, we repeat the labeled data 10 times. At the training stage, we repeat the labeled data 5 times and train the model for 60 epochs with the same optimizer and learning rate as the pre-training stage. For the experiments under 1% and 2% labeled data, we set $\tau_{unlabeled\_frame}$ = 0.5 to filter abundant pseudo labels on unlabeled frames for collision detection. The $\tau_{pseudo\_sample}$, which is used for selecting pseudo samples for pseudo-sampling, is set to 0.8, 0.7, 0.7 for car, pedestrian and cyclist, respectively. For the experiments under 5% labeled data, we use higher $\tau_{unlabeled\_frame}$ and $\tau_{pseudo\_sample}$. Because with more labeled data, the pre-trained model will produce more high-quality pseudo labels, which usually have higher scores. In practice, we set $\tau_{unlabeled\_frame}$ to 0.8 and $\tau_{pseudo\_sample}$ to 0.9, 0.8, 0.8 for car, pedestrian and cyclist. At the inference stage, we use the student PV-RCNN network to forward input data.

We compare our methods with previous state-of-the-art works SESS[37] and 3DIoUMatch[30] on ScanNet and SUN-RGBD dataset under 5%, 10%, 20% and 100% labeled data. Our method outperforms the current state-of-the-art, 3DIoUMatch[30], under all ratios of labeled data and all listed datasets. For example, with the same backbone detector and the same split of 10% labeled data, our approach achieves 50.3 [email protected] and 34.7 [email protected] on ScanNet, outperforming previous best results by 3.1 and 6.4 points. For the SUN-RGBD dataset, we also attain decent results. With the same backbone detector and split of 5% labeled data, we improve 3DIoUMatch by 1.1 and 3.2 points in terms of [email protected] and [email protected]. When utilizing 100% labeled data, there are still gains on two datasets, suggesting that semi-supervised learning is powerful with only labeled data. It should be noted that because the gt and pseudo databases of the SUN-RGBD dataset are noisy, we do not use semi-sampling when training 10%, 20% and 100% SUN-RGBD. Instead, we perform gt-sampling with the fade strategy[29] on the pre-trained model. The good pre-trained models can also benefit the semi-supervised learning on 10%, 20% and 100% SUN-RGBD, achieving promising results.

We also provide a comparison with 3DIoUMatch on the outdoor dataset KITTI. As shown in Table 2, our semi-sampling method yields significant improvement. In particular, we outperform 3DIoUMatch by 3.5 mAP, 6.7 mAP and 14.1 mAP on car, pedestrian and cyclist classes under 1% labeled data. With only 5% labeled data, the performance of our method is close to the methods trained with 100% labeled data, as can be seen in Table 8.

We first ablate the effectiveness of each object sampling strategy of semi-sampling on the KITTI dataset. Our semi-sampling consists of gt-sampling on labeled frames, gt-sampling on unlabeled frames, pseudo-sampling on labeled frames, and pseudo-sampling on unlabeled frames. In outdoor scenes, the effectiveness of gt-sampling on labeled frames has been verified by a lot of prior works, so we use it by default. All experiments in Table 3 use the same pre-trained weight. In Table 3, experiment (a) is our baseline, namely 3DIoUMatch. Experiment (b) suggests that the pseudo-sampling on labeled frames can improve our baseline by a large margin. Further,

In order to be consistent with 3DIoUMatch, we ablate our method on ScanNet with 10% labeled data and on SUN-RGDB with 5% labeled data. We use pre-trained weights provided by 3DIoUMatch on ScanNet 10% and SUN-RGBD 5% to initialize our model in Table 4 and Table 5 for a fair comparison. And we use 3DIoUMatch as our baseline, corresponding to experiment (a) in both tables. Experiment (b) in Table 4 and Table 5 demonstrate the effectiveness of gt-sampling on labeled frames. Experiment (f)-(h) in Table 4 suggest that pseudo-sampling on labeled frames is also helpful while gt-sampling or pseudo-sampling on unlabeled frames can not lead to improvement on ScanNet. By contrast, Experiment (f)-(h) in Table 5 shows that pseudo-sampling on labeled and unlabeled frames benefits our baseline while gt-sampling on unlabeled frames can not bring improvement on SUN-RGBD. Overall, the best performance on both datasets is achieved when using gt-sampling on labeled and unlabeled frames.

As discussed in Section 3.3, gt-sampling is a very useful data augmentation method in outdoor scenes while there is no investigation on it in indoor scenes. Here we conduct an experiment to study the effectiveness of gt-sampling in indoor scenes. To provide more value to the community, we use a fully-supervised state-of-the-art method RBGNet[31] as our baseline. As shown in experiment (b) of Table 6, gt-sampling boosts the strong baseline by 0.9 [email protected] and 2.2 [email protected] on ScanNet, achieving a new state-of-the-art on ScanNet. However, there is a decrease on SUN-RGBD. As depicted in Figure 2, in the early stage, gt-sampling is actually beneficial to the model performance. However, the performance degrades at the end compared to the baseline. We attribute the phenomenon to the noisy gt database of SUN-RGBD as mentioned in Section 3.3. In the early epochs, the model is not well-trained, and gt-sampling can augment the training data largely. In the late epochs, cleaner data are needed to refine the well-trained model, so the noisy gt samples become harmful for training. To make use of gt-sampling on SUN-RGBD, we use the fade strategy [29], which disables gt-sampling when the model is near convergent. We use three different epochs to study the best time to fade gt-sampling. As shown in experiments (c)-(e), when fading from the $30^{th}$ epoch, we get the best result and achieve a new state-of-the-art on SUN-RGBD.

When performing semi-sampling on unlabeled frames, we use pseudo labels of unlabeled frames to do collision detection with the pasted samples. To remove low-quality pseudo labels in unlabeled frames, we set a score threshold $\tau_{unlabeled\_frame}$ to filter them. A low threshold will leave too many abundant pseudo labels, making there no space on unlabeled frames to place object samples. A high threshold will filter pseudo labels of some foreground objects on unlabeled frames. Without pseudo labels for collision detection, the pasted samples may overlap with them. In Table 7, we study three different thresholds, and the experiment shows that 0.5 is a good choice for our method.

To further verify the effectiveness of our method under the omni-supervised setting, we evaluate our method on the Omni-KITTI dataset. We use the pre-train weight provided by PV-RCNN to initialize 3DIoUMatch and our method. Table 8 summarize the results. Our method pushes the fully-supervised PV-RCNN to a new level. Specifically, we outperform PV-RCNN by 0.8 mAP, 9.7 mAP and 3.9 mAP on car, pedestrian and cyclist classes. For 3DIoUMatch, its improvement on PV-RCNN is relatively small. In addition, we use another backbone detector CT3D[21] to validate our method. We use the pre-train weight provided by CT3D as initialization. As shown in Table 8, the improvement is also considerable, especially on the pedestrian class.