Panoptic-PHNet: Towards Real-Time and High-Precision LiDAR Panoptic Segmentation via Clustering Pseudo Heatmap

As effective data representation is the foundation of learning based tasks, for irregular and sparse point clouds, there are generally two ways to learn representations in previous studies. One is to learn features directly at the point level, while the other regularizes raw point clouds at first before feature extraction. Based on PointNet [26] and PointNet++ [27], KPConv [30] and RandLA [16] process the irregular point clouds straightforwardly, which, however, takes a time-consuming preprocess to build the graph among points. VoxelNet [39] first projects point clouds to regular voxels and utilizes 3D CNNs to learn features. SECOND [35] introduces sparse convolution to promote learning efficiency for voxel-wise features. To further optimize the latency of feature extraction as well as the memory consumption, PointPillars [20] collapses the height dimension via PointNet and then treats the input as a BEV image. PolarNet [37] takes the imbalanced distribution of points in physical space into consideration and encodes point clouds into a polar BEV map. Range image [9, 34, 23, 33] is another common projection space for efficient feature encoding, yet the 3D topological relations are also weakened. Methods [32, 38, 29] propose to fuse information from different perspectives.

As a newly proposed research domain, panoptic segmentation unifies semantic segmentation and instance segmentation. In terms of the approaches of processing ID information, two frameworks, i.e., proposal-based and proposal-free, are designed.

As pixels are the essential elements for Unet, voxels are the basic units in our network. Following the design of Unet, which fuses low-level and high-level features for each pixel, our network aggregates both 2D semantic features under different receptive fields and fine-grained 3D features for each voxel. The former quickens the convergence of the task and the latter facilitates distinguishing different voxels from each other.

The framework of our Panoptic-PHNet is shown in Fig. 2. The input LiDAR point cloud is first encoded through a voxel encoder to be 3D voxel representations, which are further transformed into size-fixed 2D representations by a BEV encoder. A Unet-like 2D backbone network is utilized to extract BEV features with different receptive fields. The BEV features are gathered for each voxel according to their coordinates. New features are generated by concatenating the gathered BEV features with the low-level fine-grained voxel features, which are then fed into two branches for semantic and instance segmentation respectively. In the instance branch, a knn-transformer module is introduced for modeling the interaction among thing voxels to enhance the feature representation. We predict offsets towards instance centers to shift thing voxels, subsequently a clustering pseudo heatmap is generated by projection according to the quantitative density of shifted voxels to yield instance centers. The possible redundant centers are integrated through a center grouping module. At last, combining the outputs of two branches, we obtain the final panoptic segmentation results via a voting-based scheme as [40]. In the following sections, we first elaborate on two components in the instance branch of our Panoptic-PHNet, then the backbone design is presented.

After the center offsets prediction and the voxels shifting, the remaining work in the instance branch can be regarded as a clustering task. For efficient instance clustering, we proceed in the BEV space based on the assumption as [40] that the interested thing objects are separated from each other and do not overlap under the bird’s eye view.

To address the current issues of existing methods as analyzed in Sec. 1, we propose a clustering pseudo heatmap to yield instance centers by projecting the shifted thing voxels onto a BEV map without extra learning branches. Since the number of voxels is used as the score for each BEV grid, the location with the most voxels in a local area corresponds to a local peak on the pseudo heatmap, which can be naturally taken as an instance center. Such a bottom-up design ensures the consistency between the instance centers and the shifted voxels. More specifically, we project the shifted thing voxels $V^{\prime}\in\mathbb{R}^{M\times 3}$ to a BEV pseudo image $I^{\prime}\in\mathbb{R}^{H\times W\times C_{n}}$ based on their locations on the BEV map, where $M$ is the number of thing voxels, $H$ and $W$ are the size of the BEV map and $C_{n}$ is the number of the semantic categories. By summing the voxel number along the dimension of $C_{n}$, a class-agnostic pseudo heatmap $I\in\mathbb{R}^{H\times W\times 1}$ is created. As a non-maximum suppression, a window-based 2D max pooling is adopted to efficiently pick out the local centers. Compared with the dense learning-based heatmap in [40], our clustering pseudo heatmap is sparse so that the top-$k$ operation for centers filtering is no longer necessary. Finally, each shifted thing voxel can be clustered to its closest center according to their spatial distance on the BEV space. We use the grid size 0.2m $\times$ 0.2m for our clustering pseudo heatmap in all the experiments.

To deal with such cases, we introduce a size-based center grouping module. We first use a 2D average pooling on the pseudo image $I^{\prime}\in\mathbb{R}^{H\times W\times C_{n}}$ to count the number of thing voxels within a sliding window for each category. Majority voting is applied to determine the class of each grid. We then give each center a minimum radius empirically based on its category. Fig. 3 (b) illustrates the grouping operation: given a certain base center $C_{b}\in\mathcal{ID}_{b}$ as well as a radius $r_{b}$, where $\mathcal{ID}_{b}$ indicates a group of centers with the same instance ID as $C_{b}$. If a target center $C_{t}\in\mathcal{ID}_{t}$ with the same semantic label as $C_{b}$ appears within the radius $r_{b}$, the set $\mathcal{ID}_{t}$ are then regrouped into $\mathcal{ID}_{b}$. We traverse all centers with this operation, through which multiple redundant centers are integrated together as shown in Fig. 3 (c). Since the number of instance centers per LiDAR frame is limited, the time cost can almost be ignored.

At the beginning of the instance branch, we use the thing mask generated from semantic segmentation to pick out the feature vectors $F\in\mathbb{R}^{M\times C}$ for thing voxels. Since the number of the voxels to be processed is significantly reduced, accurately modeling the interaction among these elements becomes possible. Similar to a natural sentence, the disorderd and number-unfixed thing voxels are suitable for Transformer [31] to deal with.

Inspired by Swin-Transformer [21] where the local attention mechanism is introduced, we propose a knn-transformer to efficiently model the interaction among thing voxels. We basically follow the design of self-attention layer from [31], except that we take spatial distance as prior to construct the similarity matrix among local thing voxels. More specifically, given the features of thing voxels with shape $M\times C$ as shown in Fig. 4, we calculate the indices of $k$ nearest neighbors for each thing voxel on GPU based on its spatial location, by which the input vectors are broadcasted to be a feature matrix with shape $M\times k\times C$. Through linear transformation, a query matrix $Q$, a key matirx $K$ and a value matix $V$ are generated respectively. Afterwards, an attention matrix with shape $M\times k$, describing the interaction between each thing voxel and its $k$ nearest neighbors, is computed as [31]:

$$\vspace{-0.2em}\operatorname{Attention}(Q,K,V)=\operatorname{softmax}\left(\frac{QK^{T}}{\sqrt{C^{\prime}}}\right)V$$

(1)

where $C^{\prime}$ denotes the size of the channel dimension. Compared with the vanilla self-attention layer, our knn-based design reduces the computational complexity per layer from $O\left(M^{2}\cdot C^{\prime}\right)$ to $O\left(M\cdot k\cdot C^{\prime}\right)$. We maintain the structure of multi-head attention and feed-forward layer in [31]. Since the position information is encoded in each voxel by nature, the positional embedding is not adopted in our module. We use $k=25$ in our experiments.

We have four decoders in our 2D backbone network, where the first two are shared for the semantic and instance branches. We gather the BEV features for the corresponding voxels in both branches respectively according to their BEV locations. The gathered BEV features are then concatenated with the fine-grained voxel features as the final voxel representation. All prediction results and supervision signals are at the voxel level. Finally, we map the voxel results into point level based on the coordinate of each point.

Following the recent official evaluation protocol [10], we further report our results on nuScenes validation as shown in Tab. 3 without any test-time augmentation techniques. Our approach outperforms the best baseline Panoptic-PolarNet [40] by 11.3$\%$ PQ and 14.8$\%$ PQ${}^{\text{Th}}$.

As shown in Fig. 5 (a), both of the two modules contribute to the final performance. Regarding the center grouping module, it presents a significant quality improvement for instance segmentation (+4.4$\%$ PQ${}^{\text{Th}}$). As mentioned in Sec. 3.2, our clustering pseudo heatmap indeed achieves high recall for center generation: as long as there is a cluster of points, a highlight peak shows up certainly. However, it also brings the issue of multiple redundant centers. Solving this problem effectively, the proposed center grouping further ensures the high precision for instance centers. Note that a possible limitation of this module lies in the introduced hyper-parameters, i.e., the prior size for each category, which can be left in further work.

In addition, by enhancing the feature representations of thing voxels for more accurate offset regression, our knn-transformer further improves the performance by 1.2$\%$ PQ${}^{\text{Th}}$, further analysis is shown later.

We still compare our method with DS-Net [14] and Panoptic-PolarNet [40]. Since the latter only has 2D offset predictions, all the EPE results in Tab. 5 are calculated in the cartesian 2D BEV space for a fair comparison. As shown in the table, our approach has an obviously smaller offset error, which means larger spatial distance among instances to facilitate the following clustering process. Although the index of mIoU also contributes to PQ, it is clear that our network benefits from such high-precision offset predictions. Moreover, high-quality offset regression presents greater impact on crowded urban scenes as illustrated in Fig. 6, where only our approach segments three close instances correctly. Such crowded scenes are more common in nuScenes dataset, on which, thus, our method shows more significant improvement. As demonstrated in Tab. 3 and Tab. 5, for example, compared with Panoptic-PolarNet, the promotions of our approach are 14.8% PQ${}^{\text{Th}}$ vs. 3.6% PQ${}^{\text{Th}}$ on the two datasets respectively.