Breaking the SSL-AL Barrier: A Synergistic Semi-Supervised Active Learning Framework for 3D Object Detection

To optimize the extraction of confident objects for model pre-training, we employ a multi-step approach that enhances the quality of our training data. Initially, we utilize a Confident Object Filtering module to extract confident objects from unlabeled scenes, providing crucial information for model training. These extracted objects are then stored in a box bank to preserve and manage the collected object information. Furthermore, we incorporate an Iterative Refinement mechanism that iteratively generates confident boxes from the unlabeled data, integrating them into the box bank to create high-quality pseudo labels for the models. This process is essential for improving the robustness and accuracy of the model.

Confident Object Filtering. To ensure compatibility with active learning(AL), we utilize the same uncertainty measure employed in the AL process. By applying this uncertainty measure to the objects, we collect their uncertainty scores and employ clustering techniques on these scores to identify the group with the lowest uncertainty scores. Traditional semi-supervised learning (SSL) methods typically use fixed thresholds or top-k selections to filter confident objects. However, we find that score distributions vary greatly across classes and models, making it difficult to set an appropriate class-specific threshold. Additionally, model performance fluctuations complicate the selection of a consistent top-k value—if $k$ is too small, too few objects are selected; if $k$ is too large, noise increases. To address this, we leverage clustering techniques, such as KMeans (Liu et al., 2023), on uncertainty scores to select confident objects. Clustering groups objects with similar patterns, and by choosing the highest number of cluster centers, we enhance the reliability of confident object selection. Once confident groups are identified, we filter objects within these groups for selection. The object information is represented as $O=\{cls,loc,score,scene_{id},pc\}$, where $cls$ denotes the class label, $loc\in\mathbb{R}^{7}$ represents the object’s location and orientation, $score\in\mathbb{R}^{1}$ is the uncertainty score, $scene_{id}$ is the scene index, and $pc\in\mathbb{R}^{n\times 3}$ contains the point cloud of the object. We also extract background objects likely to be false positives, as done in (Oh et al., 2024). All extracted information is stored in a box bank for easy access and management.

Iterative Refinement. To continuously improve the quality of the box bank, we implement an iterative refinement mechanism that selectively incorporates newly extracted confident objects. Let $O_{\text{new}}$ denote the set of newly extracted objects and $O_{\text{bank}}$ the set of objects already stored in the box bank. For each new object $o_{\text{new}}\in O_{\text{new}}$, if it overlaps with an existing object $o_{\text{bank}}\in O_{\text{bank}}$, we compare their uncertainty scores, $U(o_{\text{new}})$ and $U(o_{\text{bank}})$, respectively. The object with the lower uncertainty score is retained as follows:

This ensures that only higher-confidence objects are kept in the box bank, minimizing the risk of introducing noisy or uncertain objects into the training data.

Despite retaining objects with lower uncertainty scores, errors may still occur in uncertainty estimation, potentially leading to the inclusion of mislabeled or erroneous objects. To mitigate this risk, we introduce a deletion mechanism. After each training iteration, when new confident boxes are extracted, the newly added objects are compared to existing objects in the box bank. The deletion mechanism checks for overlaps between objects, using the Intersection over Union (IoU) metric:

If the overlap is below a specified threshold $\tau_{\text{overlap}}$ for all newly extracted objects, those objects are deleted.

Additionally, mislabeled or erroneous objects are identified and removed by evaluating their performance in subsequent training iterations. This helps ensure that the objects in the box bank remain reliable for training purposes. The iterative process of adding, comparing, and refining the box bank improves its overall quality. This ensures that the uncertainty estimation becomes more reliable over time, formally expressed as:

where $O_{\text{bank}}^{t}$ is the set of objects in the box bank at iteration $t$, and $O_{\text{remove}}$ is the set of objects identified for deletion during that iteration. Through this iterative refinement, the box bank evolves to contain higher-quality pseudo-labels for training.

In typical training scenarios, scenes often contain both confident and unconfident objects. To enhance the model’s focus on confident objects while minimizing the impact of unconfident ones, we draw inspiration from the Reliable Background Mining Module in (Liu et al., 2022), which excludes point clouds associated with potential object detections. This method proves highly effective for our task, where we aim to train only on confident objects. Furthermore, we establish a relatively high threshold to preserve background point clouds that are prone to be falsely identified as positive detections. We then construct Pseudo Scenes by merging point clouds from the selected confident boxes in the box bank with these background scenes. These Pseudo Scenes are composed solely of confident objects, excluding any unconfident ones from the training data. This strategy ensures that the pre-training dataset is fine-tuned to help the model make stable and reliable predictions, providing a solid foundation for the active learning process.

We use entropy to measure the uncertainty of each predicted box. Considering the collective influence of all the boxes, the overall uncertainty of the entire scene is represented by calculating the average entropy. This approach enables us to capture the overall uncertainty and make informed decisions based on the entropy measure.

Specifically, the uncertainty for a point cloud scene $S$ is computed as:

where $b$ represents the predicted boxes, $p_{bc}$ is the predicted class probability of class $c$ for box $b$, $N_{b}$ is the total number of predicted boxes, and $|C|$ is the number of object classes.

While this entropy-based measure captures uncertainty, it does not fully leverage the potential of the CPSP model. We observe that the CPSP pre-trained model may overlook some confident objects, leading to potential gaps in uncertainty measure. To mitigate this, we propose an ensemble strategy that combines the high-confidence predictions from the normal pre-trained model with all the boxes from the CPSP pre-trained model. Redundant boxes are then removed using the Non-Maximum Suppression (NMS) technique, ensuring more accurate and reliable predictions.

Diversity is essential for reducing redundancy in the selected samples. We achieve this by measuring the similarity between boxes using cosine similarity and assigning each box to its closest counterpart. The similarity score for each scene is computed by averaging the cosine similarity between box features from the current scene and features from previously selected scenes.

Formally, let $S_{a}$ be a scene with box features $F_{a}=\{f_{a,i}\mid\text{box}_{a_{i}}\in S_{a}\}$, and $S=\{S_{c}\}$ be the set of selected scenes with features $F=\{f_{i}\mid\text{box}_{i}\in S_{c}\}$. We calculate the similarity of scene $S_{a}$ as:

During the sample selection process, if the similarity score between a new sample and previously selected samples exceeds a threshold, the sample is excluded from selection to avoid redundancy. Given the potentially large size of $|F|$, we apply clustering to retain the most representative features.

Outdoor LiDAR scenes pose significant challenges due to the presence of rare classes, which are difficult to sample and annotate. The scarcity of these rare classes makes their annotation disproportionately expensive, especially when they coexist with more frequent classes within the same scene. Additionally, many existing methods (Luo et al., 2023b; Wu et al., 2022; Yang et al., 2022; Luo et al., 2023a) fail to address the inherent difficulty models face in accurately estimating the number of objects in complex outdoor scenes, often resulting in high rates of false positives (FP) and false negatives (FN).

To mitigate these issues, we focus on boxes that are predicted by both the normal pre-trained model and the CPSP pre-trained model. This intersection likely represents true objects, filtering out background noise and reducing false positives, thereby improving the reliability of the sampling process. To further address the class imbalance problem, we propose a class balance algorithm that dynamically adjusts the sampling process by setting an upper limit $U_{c}$ for the number of objects selected from each class. If the number of objects from a class exceeds this limit, the weight assigned to that class is reduced from 1 to 0.1, encouraging the model to focus on unconfident objects from underrepresented classes. Specifically, the upper limits for each class $c$ are determined as:

where $N_{c}$ is the number of labeled samples for class $c$, $N_{\text{total}}$ is the total number of labeled samples, $N_{i}$ is the number of labeled samples for class $i$, $C$ is the total number of classes, and $B$ is the total number of samples to be selected. To prevent unrealistically small or negative values, a minimum threshold is applied to $U_{c}$. This ensures that the model maintains balanced sampling by prioritizing underrepresented classes while avoiding over-sampling of more frequent ones.

This approach ensures rare classes are well-represented and prevents over-sampling of easier classes, achieving a balanced class distribution and improving model robustness.

We conducted evaluations of our methods on the KITTI 3D detection benchmark (Geiger et al., 2013), using the standard train split comprising 3,712 samples and the validation split containing 3,769 samples (Shi et al., 2020). In our semi-supervised active learning framework, we initially trained an initial model using randomly selected frames consisting of approximately 200 boxes. Subsequently, we leveraged the remaining unlabeled training data for further model refinement. During active learning, we specifically selected frames that contained around 150 boxes for effective training. The total number of labeled boxes in our approach is approximately 350 boxes, which accounts for less than 2% of the total boxes present in the KITTI train split. For evaluation, we calculate the mean average precision (mAP) at 40 recall positions for the Car, Pedestrian (Ped), and Cyclist (Cyc), employing 3D IoU thresholds of 0.7, 0.5, and 0.5, respectively, across different difficulty levels: easy, moderate (mod), and hard.

A significant portion of the dataset contains “DontCare” regions, as illustrated in Fig. 4. These regions often overlap with challenging objects that contribute significantly to uncertainty but are unlabeled, making them unsuitable for active learning. To mitigate this, we project the predicted 3D boxes onto 2D images since “DontCare” areas only have 2D annotations. If more than two predicted boxes lie within the “DontCare” regions, we exclude the corresponding frame from the active learning process.

We conducted evaluations of our methods on the Waymo dataset (Sun et al., 2020), a widely used benchmark in autonomous driving. It offers diverse real-world driving scenarios with high-resolution sensor data. The dataset comprises 798 training sequences and 202 validation sequences. Notably, the annotations provide a full 360° field of view. Additionally, the prediction results are categorized into LEVEL 1 and LEVEL 2 for 3D objects based on the presence of more than five LiDAR points and one LiDAR point, respectively. To optimize efficiency, we adopted a time-saving approach by setting a sample interval of 20 from the training set to generate a pool of frames. From this pool, we selected frames for our divided datasets. Similar to our approach in the KITTI dataset, we employed a similar strategy for the Waymo dataset. In the initial stage, we randomly sampled frames with approximately 5000 boxes, and in the active learning stage, we again selected frames with around 5000 boxes. The total number of boxes, which amounts to 10,000, is less than 1% of the total boxes present in the Waymo train set. For evaluation, we use mean average precision (mAP) for Vehicle (Veh), Pedestrian (Ped), and Cyclist (Cyc) in LEVEL_1 (L1) and LEVEL_2 (L2), along with average mAP and heading accuracy weighted AP (mAPH).

Table 3 presents the results of the ablation study for CPSP, illustrating the impact of different components—iteration mechanism (ITER), deletion mechanism (DEL), and false-positive (FP)—on model performance. The results indicate that each component contributes positively to the overall performance.

Specifically, adding FP greatly enhances Pedestrian detection, improving mAP from 56.3 to 56.9. This improvement is consistent with the fact that detecting Pedestrian often generates more false positives (e.g., trees or background elements), necessitating the inclusion of additional false positives to help refine the model’s ability to distinguish between true and false detections. On its own, ITER leads to a decrease in performance, with mAP dropping to 66.3, mainly due to the noise spread during the iterative process. This highlights the importance of the other components in mitigating the negative effects of noisy data. However, when ITER is combined with the DEL mechanism, performance improves, as the DEL mechanism effectively removes unwanted or erroneous objects from the box bank, enhancing the model’s overall performance. Finally, when all three mechanisms (ITER, DEL, and FP) are used together, the model achieves its best performance, reaching 68.6 mAP, demonstrating improved detection across all classes.

Furthermore, Confident objects provide crucial information with minimal noise, while unconfident ones introduce numerous incorrect pseudo-labels that mislead the model. We utilize two types of pre-training in our approach CPSP: UNC (Unconfident) and CON (Confident). The visualization results can be seen in Fig. 5. In UNC pre-training, objects with a low number of clustering centers (2) are filtered, allowing us to include more unconfident objects in the model training process. In CON pre-training, we train confident objects using the same methods but with a higher number of clustering centers (20). This approach allows us to focus on objects that exhibit a higher level of certainty and reliability during the training process.

Additional results are shown in Table 4, where we test different numbers of clustering centers (2, 5, 10, 20, and 50). These results highlight the impact of the number of centers on performance, with 20 centers yielding the highest mAP at 68.6. Using fewer centers (2, 5, or 10) introduces noise and reduces the model’s ability to accurately measure uncertainty. Conversely, using more centers (50) fails to effectively leverage the unlabeled data, limiting the model’s optimization and performance improvement.

The ablation study, as shown in Table 5, highlights the importance of the uncertainty measure, class balance methods, and diversity methods in CAL for improving performance.

The CBS method significantly addresses class imbalances, particularly for harder classes like Pedestrian (improving mAP by 1.7%) and Cyclist (improving mAP by 10.1%), which results in better performance on these challenging categories. The $E^{2}\_Unc$ component plays a critical role in active learning by selecting informative samples, enabling the model to focus on challenging instances. This leads to an overall 2.4% mAP improvement. Additionally, the $B\_Div$ method reduces redundancy in the selected samples, allowing the model to capture a broader range of object variations. This further enhances detection capabilities, contributing to the overall improvement.

By incorporating these CAL components, the overall semi-supervised active learning framework becomes more effective, resulting in better performance in 3D object detection.

We conducted experiments using both single-round and multi-round approaches to assess their potential for improving performance. Specifically, we performed experiments on the Waymo dataset, utilizing a total of 20,000 annotated boxes to compare the outcomes of single-round versus three-round approaches. As shown in Table 6, increasing the number of rounds while maintaining the same annotation budget results in a 0.9% improvement in mAP. We attribute this enhancement to the model’s improved ability to select better data over multiple rounds, underscoring the positive impact of utilizing multiple rounds in our approach.

In addition, to facilitate a comprehensive comparison of our methods with existing approaches, we employ active learning over multiple rounds. In each round, we annotate 5000 boxes, conducting a total of 5 rounds, which culminate in 30,000 annotated boxes. We compare the following methods in our evaluation: Joint3D (Hwang et al., 2023), NAL (Elezi et al., 2022), 3DIoUMatch (Wang et al., 2021) combined with our CAL, Normal pre-training combined with CAL, and our CPSP pre-training combined with CAL. For a fair comparison, all methods utilize CPSP in the Final Model Delivering Stage. As depicted in Fig. 6, our method (CPSP + CAL) consistently outperforms all other approaches across every round of active learning, demonstrating substantial performance gains even in the final evaluation round. The sustained advantage of our method highlights its effectiveness in selecting and leveraging superior data throughout the active learning process.

We replace the SSL methods in Table 1 with our Collaborative PseudoScene Pre-training (CPSP) approach in the Final Model Delivering Stage to examine how well our method performs when compared with different pre-train methods. As presented in Table 7, the results demonstrate that our CPSP method not only achieves the best performance but also generates a more significant performance gap in comparison to other pre-train methods. The improvement is particularly noticeable across challenging classes such as pedestrians and cyclists. This highlights the robustness of CPSP in handling diverse object detection tasks.