ReDAL: Region-based and Diversity-aware Active Learning for Point Cloud Semantic Segmentation

In the past decade, many supervised point cloud semantic segmentation approaches have been proposed [13, 19, 20, 30, 27, 6, 3, 15, 26]. However, despite the continuous development of supervised learning algorithms and the simplicity of collecting 3D point cloud data in large scenes, the cost of obtaining manual point-by-point marking is still high. As a result, many researchers began to study how to achieve similar performance with less labeled data.

Some have tried to apply transfer learning to this task. Wu et al. [33] developed an unsupervised domain adaptation approach to make the model perform well in real-world scenarios given only synthetic training sets. However, their method can only be applied to a single network architecture [32] instead of a general framework.

Some others applied weakly supervised learning to reduce the cost of labeling. [34] utilized gradient approximation along with spatial and color smoothness constraints for training with few labeled scattered points. However, this operation does not save much cost, since annotators still have to switch viewpoints or zoom in and out throughout a scene when labeling scattered points. Besides, [31] designed a multi-path region mining module to help the classification model learn local cues and to generate pseudo point-wise labels at subcloud-level, but their performance is still far from the current state-of-the-art method compared with the fully-supervised result.

Still some others leveraged active learning to alleviate the annotation burden. [16] designed an active learning approach to reduce the workload of CRF-based semantic labeling. However, their method can not be applied to current large-scale datasets for two reasons. First, the algorithm highly relies on the result of over-segmentation preprocessing and the algorithm cannot perfectly cut out small blocks with high purity in the increasingly complex scenes of the current real-world datasets. Second, the computation of pair-wise CRF is extremely high and thus not suitable for large-scale datasets. In addition to the above practice, [14] proposed segment entropy to measure the informativeness of single point cloud scan in a deep active learning pipeline.

To the best of our knowledge, we are the first to design a region-based active learning framework general for many deep learning models. Furthermore, our idea of reducing redundant annotation through diversity-aware selection is totally different from those previous works.

Sufficient labeled training data is vital for supervised deep learning models, but the cost of manual annotation is often high.

Active Learning [24] aims to reduce labeling cost by selecting the most valuable data for label acquisition. [28] proposed the first active learning framework on deep learning where a batch of items, rather than a single sample in traditional active learning, is queried in each active selection for acceleration. Several past deep active learning practices are based on model uncertainty. [28] is the first work that applied least confidence, smallest margin [21] and maximum entropy [25] to deep active learning. [29] introduced semi-supervision to active learning, which assigned pseudo-labels for instances with the highest certainty. [8, 9] combined bayesian active learning with deep learning, which estimated model uncertainty by MC-Dropout.

In addition to model uncertainty, many recent deep active learning works take in-batch data diversity into account. [23, 12, 2] stated that neglecting data correlation would cause similar items to appear in the same querying batch, which further leads to inefficient training. [23] converted batch selection into a core-set construction problem to ensure diversity in labeled data; [12, 2] tried to consider model uncertainty and data diversity at the same time. Empirically, uncertainty and diversity are two key indicators of active learning. [11] is a hybrid method that enjoys the benefit of both by dynamically choosing the best query strategy in each active selection step.

To the best of our knowledge, we design the first 3D deep active learning framework combining uncertainty, diversity and point cloud domain knowledge.

We use a sub-scene region as the fundamental query unit in our proposed ReDAL method. In traditional deep active learning, the smallest unit for label querying is a sample, which is a whole point cloud scan in our task. However, based on the prior experiment shown in Figure 2, we know that some labeled regions may contribute little to the model improvement. Therefore, we change the fundamental unit of label querying from a point cloud scan to a sub-scene region in a scan.

Instead of using model uncertainty as the only criterion to determine the selection common in 2D active learning, we leverage the domain knowledge from 3D computer vision and include two informative cues, color discontinuity and structural complexity, in the selection indicators. Moreover, to avoid redundant labeling caused by multiple duplicate regions in a querying batch, we design a simple yet effective diversity-aware selection strategy to mitigate the problem and improve the performance.

Our region-based and diversity-aware active learning can be divided into 4 steps: (1) Train on current labeled dataset $D_{L}$ in a supervised manner. (2) Calculate the region information score $\varphi$ for each region with three indicators: softmax entropy, structure complexity and color discontinuity as shown in Figure 3 (a) (Sec. 3.2). (3) Perform diversity-aware selection by measuring the similarity between all regions and using a greedy algorithm to penalize similar regions appearing in a querying batch as shown in Figure 3 (b) (Sec. 3.3). (4) Select top-K regions for label acquisition, and move them from the unlabeled dataset $D_{U}$ into the current labeled dataset $D_{L}$ as shown in Figure 3 (c) (Sec. 3.4).

We divide a large-scale point cloud scan into some sub-scene regions as the fundamental label querying units using VCCS [17] algorithm, an unsupervised over-segmentation method that groups similar points into a region. The original purpose of this algorithm was to cut a point cloud into multiple small regions with high segmentation purity to reduce the computational burden of the probability statistical model. Different from the original purpose of requiring high purity, our method merely utilizes the algorithm to divide a scan into sub-scenes of median size for better annotation and learning. An ideal sub-scene consists of several but not complicated semantic meanings, while preserving geometric structures of point cloud.

In each active selection step, we calculate the information score of a region from three aspects: (1) softmax entropy, (2) color discontinuity, and (3) structural complexity, which is described in detail as follows.

With the sorted region information list $\varphi$, a naive way is to select the top-ranked regions for label acquisition directly. Nevertheless, this strategy results in multiple visually similar regions being in the same batch as shown in Figure 4. These regions, though informative individually, provide less diverse information for the model.

To avoid visually similar regions appearing in a querying batch, we design a diversity-aware selection algorithm divided into two parts: (1) region similarity measurement and (2) similar region penalization.

After getting the final scores $\varphi^{*}$ by considering region diversity, we select regions into a querying batch in decreasing order according to $\varphi^{*}$ for label acquisition until the budget of this round is exhausted. Note that in each label acquisition step, we set the budget as a fixed number of total points instead of a fixed number of regions for fair comparison since each region contains different number of points.

For experiments, after selecting the querying batch data, we regard the ground truth region annotation as the labeled data obtained from human annotators. Then, these regions are moved from unlabeled set $D_{U}$ to labeled set $D_{L}$. Note that different from the 100% fully labeled initial training point cloud scans, since we regard a region as the basic labeling unit, many point cloud scans with only a small portion of labeled region are appended to the labeling data set $D_{L}$ in every active selection step as shown in Figure 3 (c).

Finishing the active selection step containing region information estimation, diversity-aware selection and region label acquisition, we repeat the active learning loop to fine-tune the network on the updated labeled dataset $D_{L}$.

In order to verify the effectiveness and universality of our proposed active selection strategy, we conduct experiments on two different large-scale datasets and two different network architectures. The implementation details are explained in the supplementary material due to limited space.

We compare our proposed method with 7 other active selection strategies, including random point cloud scans selection (RAND), softmax confidence (CONF) [28], softmax margin (MAR) [28], softmax entropy (ENT) [28], MC-dropout (MCDR) [8, 9], core-set approach, (Core-Set) [23] and segment-entropy (SEGENT) [14]. The implementation details are explained in the supplementary material.

The experimental results can be seen in Figure 5. In each subplot, the x-axis means the percentage of labeled points and the y-axis indicates the mIoU achieved by the network. Our proposed ReDAL significantly surpasses other existing active learning strategies under any combination.

In addition, we observe that random selection (RAND) outperforms any other active learning methods except ours on four experiments. For uncertainty-based methods, such as ENT and MCDR, since the model uncertainty value is dominated by background area, the performance is not as expected. The same, for pure diversity approach, such as CSET, since the global feature is dominated by the background area, simply clustering global feature cannot produce diverse label acquisition. The experimental results further verify our proposal of changing the fundamental querying units from a scan to a region is a better choice.

On the S3DIS [1] dataset, our proposed active selection strategy can achieve more than 55 % mIoU with 15 % labeling points, while others cannot reach 50 % mIoU under the same condition. The main reason for such a large performance gap is that these room-sized point cloud data in the dataset is very different. Compared with other active selection methods querying a batch of point cloud scans, our region-based label acquisition make the model be trained on more diverse labeled data.

As for the SemanticKITTI [5], we find that with merely less than 5% of labeled data, our active learning strategy can achieve 90% of the result of fully supervised methods. With the network architecture of MinkowskiNet, our active selection strategy can even reach 95 % fully supervised result with only 4 % labeled points.

Furthermore, in Table 1, the performance of some small or complicated class objects like bicycles and bicyclists is even better than fully supervised one. Table 2 shows the reason that our selected algorithm focuses more on those small or complicated objects. In other words, our ReDAL does not waste annotation budgets on easy cases like uniform surfaces, which again realizes the observation and motivation in the introduction. Besides, our ReDAL selection strategy makes it more friendly to real-world applications, such as autonomous driving, since it puts more emphasis on important and valuable semantics.

We verify the effectiveness of all components in our proposed active selection strategy on S3DIS dataset [1]. The results are shown in Figure 8.

First, changing the labeling units from scans to regions contributes the most to the improvement as shown from the comparison of the purple line (ENT), the yellow line (ENT+Region), and the light blue line (RAND+Region). By applying region-based selection, the mIoU performance improves over 10 % under both networks.

Furthermore, our diversity-aware selection also plays a key role in the active selection process shown from the comparison of the yellow line (ENT+Region) and the green line (ENT+Region+Div). Without the aid of this component, the performance of region-based entropy is even lower than random region selection under SPVCNN network architecture as shown from the comparison of the yellow line (ENT+Region) and the light blue line (RAND+Region).

As for adding extra information of color discontinuity and structure complexity, it contributes little to SPVCNN but is helpful to MinkowskiNet when the percentage of labeled points is larger than 9 % as shown in the comparison of the green line (w/o color and structure) and the red line (w/ color and structure).

Note that the performance of “ENT+Region” (the yellow line) is similar to “ENT+Region+Color/Structure” (the dark blue line). The reason is that without our diversity module, the selected query batch is still full of duplicated regions. The result also validates the importance of our diversity-aware greedy algorithm.