MaskRange: A Mask-classification Model for Range-view based LiDAR Segmentation

According to different LiDAR data representations, LiDAR-based segmentation algorithms can be categorized into three types [10], i.e., the point-, voxel-, and projection-based methods. The point-based methods [5, 11, 12] can extract features directly from the raw point cloud, but have the inefficient neighbor-searching problem as well as computational and memory limitations. The voxel-based methods [13, 14, 15] partition the space into a finite number of regions to accelerate the feature extraction process, but suffer serious information loss when resolution is reduced. The projection-based methods [6, 16, 17, 18, 19] also have information loss, but can benefit from some efficient 2D CNN architectures. Due to their efficiency, projection-based methods are more attractive for practical applications. Our work belongs to this category, specifically to the range-view-based methods.

Different from the conventional per-pixel classification paradigm, the mask-classification methods formulate the segmentation problem from another perspective by predicting a set of binary masks and assigning a single class to each mask in parallel. Following DETR [20] framework, Maskformer [7] uses a set of learnable tokens to query the dense embeddings for mask prediction and shows promising results on both semantic and panoptic segmentation tasks. Similarly, K-Net [9] also unifies segmentation tasks in the view of dot product between kernel and feature maps. Mask2Former [8] further improves the components of MaskFormer and outperforms most specialized architectures on all considered tasks and datasets. Our work utilizes the MaskFormer [7] meta architecture and aims to achieve LiDAR segmentation in a unified manner.

Data augmentation is an important way to enhance model performance by increasing the diversity of training sets. In the 3D area, simple data augmentation schemes such as random rotation, translation, point dropping, and flipping are widely used in common practice [5, 11, 21]. PointMixup [22] and PointCutMix [23] extend the 2D augmentation methods Mixup [24] and CutMix [25] into 3D point cloud, respectively. These methods have shown better performance compared with the original simple augmentation methods. However, they cannot generally be applied to large-scale point cloud understanding. Mix3D [26] proposes an effective method by simply mixing two data items and can significantly improve the out-of-context generalization ability. To alleviate the class-imbalance problem, RPVNet [10] and Second [27] augment data by pasting the rare-class instances to the scene with the consideration of plausibility and reality. Our Weighted Paste Drop data-augmentation scheme is similar to these works but more efficient and unified. We will compare our method with the above-mentioned methods in subsection 3.2.

We build up our model based on the mask-classification meta architecture [7, 8] with consideration of the efficiency and accuracy. A simple meta architecture generally consists of three components: Backbone, Pixel Decoder and Transformer Decoder. Figure 2 is the overview of MaskRange. Each component is introduced in detail in the following paragraphs.

Backbone. The backbone is to extract low-resolution features from the input $R_{in}$. Without loss of generality, we use a modified ResNet-34 [30, 31, 18] as our backbone to extract global features from a range image (see supplementary material Section A for more details). Note that we do not utilize any pre-trained parameters for two reasons. One is that we hope our method is general enough so that different types of backbones can be incorporated into it, whereas there may be no pre-trained models for these backbones. The other is that the ImageNet [32] pre-trained model is not suitable for our range image since the data distribution is quite different.

Pixel Decoder. The pixel decoder is used to get high-resolution per-pixel embeddings. Since we have no pre-trained models, our backbone is required to be trained from scratch and we also need to optimize the pixel decoder and transformer decoder simultaneously. Thus, it would take relatively long time to train the network. To speed up the feed-forward time and the convergence rate, we utilize the Fully Interpolation Decoding (FID) [18] as our decoder module. The FID module is a parameter-free decoder, which only contains bilinear upsampling operation and can facilitate the optimization of our network. To further improve the performance, we replace the bilinear upsampling in FID with the Data-dependent Upsampling (DUpsampling) [33] to get the high-resolution feature maps. Note that we design our decoder in FID format only for simplicity and efficiency. Other decoders can also be used.

Transformer Decoder. For DETR-like frameworks [20, 7], the transformer decoder acts like first asking what objects are in the image, then finding where these objects are. To achieve that, a set of learnable object embeddings are used to interact with the image features. The outputs of the transformer decoder are the corresponding class predictions and mask embeddings. The binary mask predictions can be obtained via a simple matrix multiplication operation. We do not incorporate more advanced designs (e.g., Mask2Former [8]) into our architecture for simplicity. With this architecture, we can achieve the semantic segmentation and panoptic segmentation tasks in a unified manner. We refer readers to MaskFormer [7] for more details and better comparison.

To prevent model overfitting, the simple data augmentation schemes are widely used in common practice, including random rotation, translation, flipping, and point dropping [17, 18, 19]. We also train our model directly with these common data augmentation schemes. However, the performance of our MaskRange even cannot be on par with that of the per-pixel baseline (CENet [31]). We analyze that the unsatisfactory results (see supplementary material Section B for experimental analysis) are due to three problems: overfitting, context reliance, and class imbalance.

Overfitting. We restrict the overfitting problem here only related to the amount and diversity of training data. It has been pointed out by some previous works [34, 35] that transformer architectures usually require more training data compared with convolutional neural networks. However, the public available LiDAR segmentation datasets are relatively limited and small, and the common data augmentation schemes cannot satisfy the requirement of MaskRange.

Context Reliance. In terms of “Context”, it generally refers to the strong configuration rules present in man-made environments [26]. For example, with context knowledge, it can be inferred that traffic signs tend to be at the roadside, but not within a parking lot. However, relying too heavily on contextual cues may result in poor model generalization ability to rare or unseen situations [26, 36]. This problem is particularly acute for MaskRange since the transformer decoder only queries the global context embeddings to find objects. Intuitively, paying more attention to local geometrical information can be helpful to deal with the context-reliance problem. Unfortunately, most LiDAR segmentation methods adopt encoder-decoder architectures to aggregate context information without the consideration of balancing the global context priors and the local geometry information.

Class Imbalance. Generally, the number of points corresponding to “road” is significantly larger than that of “pedestrian”. Thus, the network is more biased to the high-frequency classes and may perform badly in some rare classes. This problem is widespread in LiDAR-based segmentation datasets [6] and can usually be dealt with by adding statistical weights to the cross-entropy loss, i.e., weighted cross entropy [6, 17, 18, 19]. However, this simple method cannot handle the class-imbalance problem well.

To deal with the aforementioned problems, we propose a novel data-augmentation method, which is constituted by three meta operations: “Weighted”, “Paste” and “Drop”. Initially, two frames are randomly selected from the dataset (noted as the first frame and the second frame) and the common data augmentation is applied. The paste operation selects the long-tail objects from the second frame firstly, then adds them to the first frame. The drop operation selects the non-long-tail class points in the first frame, then deletes these points. The weighted operation is to add a probability to the paste and drop.

Our WPD data augmentation significantly enlarges the size and diversity of the dataset, therefore alleviating the overfitting problem. The core idea of our WPD scheme to alleviate the context-reliance problem is to weaken the role of the context priors. Our “paste” operation can create unusual or even impossible scene scenarios to the training set. For example, we can “paste” a “trunk” in the middle of a road, which never appears in the original dataset and cannot be created by common data augmentation. This means that the “paste” operation can weaken the context priors. To further reduce the context bias, we “drop” the points with high context information. For example, we hope our network can recognize cars without the road background. For the class-imbalance problem, we drop the high-frequency classes, such as road and car, with high probability and paste less-frequency classes frequently.

Note that, different from Mix3D [26] which pastes all components, we only paste the long-tail objects and drop the non-long-tail objects. As Figure 3(c) shows, after the spherical projection, it is hard to distinguish the mixed points by Mix3D from the original points of the first frame. Though the context priors can be “misled” by Mix3D [26], it makes the model confused. In addition, Mix3D cannot handle the class-imbalance problem. RPVNet [10] and Second [27] also copy the long-tail instances from a database and paste them above the ground-class points. However, their methods are time-consuming and unrealistic (See Figure 3(a) and 3(b)). Our paste combines the actual scanning points, which inherently considers the orientation of the LiDAR sensor and the sparsity of the scanned points at different distances.

Our loss functions contain classification loss and binary mask losses. The mask-classification loss is the cross-entropy loss. The binary mask losses contain focal loss $L_{focal}$ [37], Lovász loss $L_{ls}$ [38] and boundary loss $L_{bd}$ [39]. Our total loss can be computed as

where $\lambda_{1}$, $\lambda_{2}$, $\lambda_{3}$ and $\lambda_{4}$ are hyper parameters to balance these losses.

Different from other works [17, 18, 19, 31] which utilize the weighted cross-entropy loss, we utilize the weighted focal loss to re-balance the training set since the statistical weights have been largely changed by our data augmentation. Our weighted focal loss can be computed as

where $p_{t}$ is the probability of prediction for the corresponding label. $\gamma$ is the focusing parameter and $(1-p_{t})^{\gamma}$ is the modulating factor [37]. $f_{i}$ stands for the proportion of the class $i$ according to the number of points. $\epsilon$ is a small constant to prevent the exception value. $ins_{i}$ and $sem_{i}$ are the number of instance and semantic segments of the class $i$, respectively. Thus, $\alpha_{i}$ is the semantic re-balance factor and $\beta_{i}$ is the panoptic re-balance factor. For semantic segmentation, $\beta_{i}$ is equal to 1 since each class has only one “instance”. As for panoptic segmentation, $\beta_{i}$ is equal to 1 for stuff classes and larger than 1 for thing classes. This unified weighted focal loss provides a general promotion on both semantic and panoptic segmentation tasks.

In MaskFormer [7], auxiliary losses are added to every transformer decoder layer and the per-pixel embeddings in the final pixel decoder layer. Our FID-like pixel decoder is not fully decoded, which may result in lower performance with only supervision on the final layer. Thus, inspired by deep supervision [40, 41, 31], we add auxiliary losses on the feature embeddings in intermediate pixel decoder layers (as illustrated in Figure 2).

Data augmentation. Initially, two frames are randomly selected and the common data augmentation schemes (random point dropping, flipping, rotation, and translation) are applied to them. Then we paste the long-tail objects in the second frame to the first frame and drop the non-long-tail class points in the first frame. We set motorcyclist, bicyclist, bicycle, person, motorcycle, traffic-sign other-vehicle, truck, pole, other-ground, and trunk as long-tail classes according to the frequency of the points in each class (the detailed statistical results are presented in supplementary material Section C). The other classes are set as non-long-tail classes. We add the probability to the paste and drop operations, known as the weighted operation. We similarly obtain these weights as presented in our weighted focal loss Eq. 3, i.e., $\alpha_{i}\beta_{i}$. We rescale these weights to [0, 1] by $\alpha_{i}\beta_{i}$ / $\max$($\alpha_{1}\beta_{1}$, $\alpha_{2}\beta_{2}$, …, $\alpha_{k}\beta_{k}$), where $k$ is the number of classes.

Training settings. The input resolution of our range image is $64\times 2048$. CENet [31] is used as our pixel-level module, which provides us with the backbone and the pixel decoder. We use 4 transformer decoder layers with 100 queries by default. We use AdamW as the optimizer [42] and the poly [43] learning rate schedule to optimize our model. The initial learning rate of transformer decoder parameters is set as $0.0001$. For backbone and pixel decoder, the initial learning rate is $0.001$. The batch size is set as $16$. All models are trained for $150$ epochs with $4$ A100 GPUs.

Inference and post-processing. In inference time, we use the same strategy as proposed in MaskFormer [7]. With the predictions in range view, we need to further project these predictions back to the 3D point cloud. Following some previous works [6, 17, 19], we employ a KNN post-processing to alleviate the “shadow” effect. The inference latency is measured on a platform with an AMD Ryzen 9 3950X CPU and an RTX 2080Ti GPU.

For semantic segmentation, we use mean Intersection over Union (mIoU) [47] as our evaluation metric to compare our MaskRange with recently published works. As shown in Table 1, among all projection-based methods, our MaskRange achieves the best performance. Compared with point/voxel-based methods, our performance is still competitive with only $40\ ms$ runtime cost.

For panoptic segmentation, we use Panoptic Quality (PQ) [54] as the main metric and we also report Recognition Quality (RQ) and Segmentation Quality (SQ) for better comparison. These metrics are calculated separately for thing and stuff classes. Besides, we also report PQ^† [55] which uses IoU as PQ for stuff classes. As shown in Table 2, among all listed methods, our MaskRange is the most efficient one with 25 FPS and achieves comparable performance with $53.1$ PQ. Benefiting from the mask classification paradigm and our unified WPD augmentation, we can extend our MaskRange to panoptic segmentation without any bounding-box supervision or extra hyperparameters tuning.

We do ablation studies on the WPD data augmentation, upsampling and deep supervision strategies. We also compare our weighted focal loss (WF) with the weighted cross-entropy loss (WCE) to show the effectiveness of our loss functions. We choose CENet [31] as the per-pixel baseline and train it with the official code. All the ablation studies are conducted with the $64\times 2048$ resolution.

Data augmentation. We first study the effectiveness of our WPD data augmentation. To show the superiority of our WPD, we also compare our method with Mix3D [26] and Instance Paste (I.P.) [10]. As shown in Table 3 (row 3-6), our WPD outperforms compared methods by at least $1.2$ mIoU. It can be seen that, with the common data augmentation, our model cannot even be on par with the CENet [31] (row 1). When incorporating the Mix3D [26], the performance is improved by a large margin. This motivates us to investigate better data augmentation methods. Thus, we propose WPD, which can alleviate the overfitting, context-reliance, and class-imbalance problems simultaneously. We refer readers to the supplementary materials Section B and D for more details.

We also add our WPD to the CENet (row 2), but the improvement is limited ($64.25\%-62.70\%=1.55\%$). In contrast, the improvement attributed to the WPD in our MaskRange is $66.80\%-59.10\%=7.70\%$. This means that our MaskRange is more expressive. Compared with CENet, MaskRange introduces $1.113$ M extra parameters and only $2.2$ more GFlops. Another possible explanation is that the transformer decoder collects global information from the image features to generate class predictions. This setup reduces the need of heavy context aggregation in per-pixel module [7] and can help this module focus more on the shape or geometry information.

WCE vs. WF. Since the statistical weights have been largely changed by our WPD data augmentation, we use focal loss to re-weight the cross-entropy loss. Additionally, we also introduce extra re-balance factors to the original focal loss. Rows $6$-$7$ of Table 3 show that our weighted focal loss is superior to the weighted cross-entropy loss.

Deep supervision strategy. We compare our improved implementation of deep supervision (notated as DeepB) with the default deep supervision (DeepA). As illustrated in Figure 2, the red dashed lines represent DeepA and the red solid lines represent DeepB. As shown in row $7$ and $8$ of Table 3, the mIoU of DeepB can be slightly higher than DeepA. Moreover, the convergence rate of DeepB is faster than that of DeepA. Thus, We choose DeepB as our deep supervision strategy.

Upsampling strategy. The interpolation-based upsamling (IU) is efficient but may result in coarse predictions. With data-dependent upsampling (DU), we can achieve better performance by $0.24$ mIoU (row $8$-$9$) with only $2\ ms$ extra runtime cost.

We also do the ablation on the re-balance strategies, which contain the comparisons of no balance, class balance, and unified balance strategies on both semantic and panoptic segmentation tasks. We refer readers to the supplementary material Section E for more details.