PointInst3D: Segmenting 3D Instances by Points

DyCo3D [14] has three output branches: semantic segmentation, centroid offset prediction, and mask features. The breadth-first-searching algorithm is used to find out the homogenous points that have identical semantic labels and close centroid predictions. Each cluster is sent to the instance head and generates a set of convolution parameters for decoding the mask of the corresponding instance. Formally, the mask $\hat{M_{k}}$ predicted by the $k$-th cluster can be formulated as:

The input features to convolution contains two parts: $F_{m}$ and $C_{\text{rel}}^{k}$. $F_{m}$ is the mask features shared by all instances. $C_{\text{rel}}^{k}\in\mathbb{R}^{N\times 3}$ is the instance-specific relative coordinates, which are obtained by computing the difference between the center of the $k$-th cluster and all input points. $F_{m}$ and $C_{\text{rel}}^{k}$ are concatenated (‘$\oplus$’) along the feature dimension. The convolutional weights are dynamically generated by an mlp layer, whose input is the feature of the $k$-th cluster. The clustering algorithm $G(\cdot)$ takes the semantic prediction $P_{s}\in\mathbb{R}^{N}$ and centroid prediction $P_{c}\in\mathbb{R}^{N}$ as input and finds out a set of homogenous clusters. The $k$-th cluster is denoted as $G(\cdot)_{k}$. Besides, the dynamic convolution in DyCo3D is conditioned on the results of semantic segmentation. For example, DyCo3D can only discriminate one specific ‘Chair’ instance from all points that are semantically categorized as ‘Chair’, instead of the whole point set. It is implemented by an element-wise production (‘$\odot$’) with a binary mask (‘$\mathds{1}(\cdot)$’). $s_{k}$ is the semantic label of the $k$-th cluster. Finally, the target mask for $\hat{M_{k}}$ is decided by the instance label of the $k$-th cluster. More details can be found in [14].

Although promising, DyCo3D [14] involves a grouping step to get the instance-related clusters, depending on the accuracy of semantic segmentation and offset prediction. Besides, the conditional convolution also forces the instance decoding to rely on the results of semantic segmentation. These inter-task dependencies cause error accumulation and lead to sub-par performance (See Fig. 1). In this paper, we propose a clustering-free and dependency-free framework in a per-point prediction fashion. Total $K$ points are selected via the farthest point sampling strategy. The instance head takes as input both the mask feature $F_{m}$ and point-wise feature $f_{b}^{k}$. The $k$-th mask $\hat{M_{k}}$ predicted by the instance head can be formulated as:

where $f_{b}^{k}$ is the feature of the $k$-th sampled point from output of the backbone. $C_{\text{rel}}^{k}\in\mathbb{R}^{N\times 3}$ is the relative position embedding, obtained by computing the difference between the coordinate of the $k$-th point and all other points. More details about the instance head can be found in supplementary materials.

However, building such a simplified pipeline is non-trivial. Removing the clustering step and conditional convolution causes the mAP of DyCo3D to drop dramatically.

To summarize, the loss function includes two terms for training, including the auxiliary loss term $\mathcal{L}_{\text{a}}$ and the main task loss term $\mathcal{L}_{\text{m}}$:

where $\{M^{a}_{\text{k}}\}^{K}\in\{0,1\}^{K\times N}$ is the ground truth masks for the $K$ predictions. These targets are static and decided by the instance labels of the $K$ sampled points. $\{M^{m}_{\text{k}}\}^{K}\in\{0,1\}^{K\times N}$ is the set of the calibrated targets for the main instance head. $\{\hat{M}^{a}_{\text{k}}\}^{K}$ and $\{\hat{M}^{m}_{\text{k}}\}^{K}$ are the predictions from auxiliary and main instance heads, respectively. $w_{a}$ is the loss weight for the auxiliary task. We set $w_{a}$ to 1.0 with a decaying rate of 0.99. Early in the training phase, the static targets for the auxiliary task play a significant role in stabilizing the learning process. The loss of the main task is involved until the end of a warming-up period, which is set to 6k steps. So far, we have obtained a set of binary masks. There are many ways to obtain the corresponding categories, for example, adding a classification head for each mask proposal. In our paper, we implement it by simply introducing a semantic branch. The category $c_{k}$ of the $k$-th instance is the majority of the semantic predictions within the foreground mask of $\hat{M}^{m}_{k}$. Instances with a number of points less than 50 are ignored.

ScanNet has 1613 scans in total, which are divided into training, validation, and testing with a size of 1201, 312, and 100, respectively. The task of instance segmentation is evaluated on 18 classes. Following  [14], we report the results on the validation set for ablation study and submit the results on the testing set to the official evaluation server. The evaluation metrics are mAP (mean average precision ) and AP@50.

S3DIS contains more than 270 scans, which are collected on 6 large indoor areas. It has 13 categories for instance segmentation. Following the previous method [34], the evaluation metrics include: mean coverage (mCov), mean weighed coverage (mWCov), mean precision (mPrec), and mean recall (mRec).

The backbone model we use is from [9], which maintains a symmetrical UNet structure. It has 7 blocks in total and the scalability of the model is controlled by the channels of the block. To prove the generalization capability of our proposed method, we report the performance with both small and large backbones, denoted as Ours-S and Ours-L, respectively. The small model has a channel unit of 16, while the large model is 32. The default dimension of the mask features is 16 and 32, respectively.

For each input scan, we concatenate the coordinates and RGB values as the input features. All experiments are trained for 60K iteration with 4 GPUS. The batch size for each GPU is 3. The learning rate is set to 1e-3 and follows a polynomial decay policy. In testing, the computation related to the auxiliary head is ignored. Only Non-Maximum-Suppression (NMS) is required to remove the redundant mask predictions for inference, with a threshold of 0.3.

In this section, we verify the effectiveness of the key components in our proposed method. For a fair comparison, all experiments are conducted on the validation set of ScanNet [6] with the smaller model.

Baseline. We build a strong baseline by tailoring CondInst [30] for the 3D point cloud. It works in a per-point prediction fashion and each sampled point has a static target, which is consistent with the corresponding instance label. As shown in Tab. 1, our method achieves 33.7% 52.4%, and 65.0% in terms of mAP, AP@50, and AP@25, respectively. With a larger number of sampled points and longer iterations, our baseline model surpasses the implementation of DyCo3D [14] by a large margin.

Center Prior in 3D. To demonstrate the difficulty of selecting informative samples in 3D, we tailor the center prior [31] to 3D point cloud. As points are collected from the surface of the objects, centers of 3D instances are likely to be in empty space. To this end, we first predict the offset between each point and the center of the corresponding object. If the distance between the center-shifted point and the ground truth is close ($\leq 0.3$m), the point is regarded as positive and responsible for the instance. If the distance is larger than 0.6m, the point is defined as negative. Other points are ignored for training. As presented in Tab. 1, selecting positive samples based on the 3D center prior only boosts 0.4% and 0.8% in terms of mAP and mAP@50, respectively. The incremental improvement demonstrates the difficulty of selecting informative samples in 3D. In contrast, we propose to apply a dynamic strategy, by which the target for each candidate is determined based on its prediction.

Dynamic Targets. To show the effectiveness of the dynamic strategy, we implement an experiment by removing the auxiliary head. As the predictions are basically random guesses in the early stage of the training, we first warm up the model for 12k iterations with a static assignment to avoid the trivial solution. In the remaining steps, targets are calibrated by the Optimal Solution. As shown in Tab. 1, our approach boosts the performance of the baseline model by 3.1%, 2.4%, and 0.9%, in terms of mAP, AP@50, and AP@25, respectively.

Auxiliary Supervision. As illustrated in Fig. 2, we propose to regularize the intermediate layers by introducing an auxiliary instance head for decoding the instance masks. The targets for this task are static and consistent with the instance labels. Besides, as the generated parameters are convolving with the whole point set, large context and instance-related knowledge are encoded in the point-wise features. To remove the influence of the dynamic assignment, both auxiliary and the main task are applying a static assignment strategy. As shown in Tab. 1, the auxiliary supervision brings 2.8%, 1.9%, and 0.7% improvement in terms of mAP, mAP@50, and mAP@25, respectively. In addition to the encoded large context, the predicted instance masks are also applied to the Optimal Solution to obtain calibrated targets. Combining with the proposed dynamic assignment strategy, it further boosts mAP, AP@50, and AP@25 for 3.1%, 4.4%, and 4.5%, respectively, achieving 39.6% in terms of mAP with a small backbone.

Analysis on Efficiency. Our method takes the whole scan as input, without complex pre-processing steps. Similar to DyCo3D [14], the instance head is implemented in parallel. To make a fair comparison, we set K equal to the average number of clusters in DyCo3D. Using the same GPU, the mAP of our proposed method is 1.8% higher than DyCo3D and the inference time is 26% faster than DyCo3D.

Number of Random Selected Samples. We randomly select $K$ points, each of which is responsible for one specific instance or the background (all zeros). In this part, we study the influence of the value of $K$. The performance is shown in Fig. 4. We set K to 256 for its highest mAP.

The Dimension of the Mask Feature. The mask feature contains the knowledge of instances. We conduct experiments to show the influence of different dimensions of the mask feature. We find the fluctuation of the performance is relatively small when the dimension is greater than 8, showing the strong robustness of our method to the variation of $d^{\prime}$. We set $d^{\prime}$ to 16 in our experiments.

We compare our method with other state-of-the-art methods on both S3DIS and ScanNet datasets.

3D Detection. Following [14, 7], we evaluate the performance of 3D detection on the ScanNet dataset. The results are obtained by fitting axis-aligned bounding boxes for predicted masks, as presented in Tab. 2. Our method surpasses DyCo3D [14] and 3D-MPA [7] by 4.8% and 1.8% in terms of mAP, respectively. The promising performance demonstrates the compactness of the segmentation results.

Instance Segmentation on S3DIS. Following the evaluation protocols that are widely applied in the previous approaches, experiments are carried out on both Area-5 and 6-Fold cross-validation. As shown in Tab. 3, our proposed method achieves the highest performance and surpasses previous methods with a much simpler pipeline. With 6-fold validation, our method improves HAIS [4] by 4.5%, 3.7%, 3.2%, and 4.6% in terms of mConv, mWConv, mPrec, and mRec, respectively. The proposed approach works in a fully end-to-end fashion, removing the error accumulation caused by the inter-task dependencies.

Instance Segmentation on ScanNet. The performance of instance segmentation on the validation and testing sets of ScanNet [6] is reported in Tab. 4 and Tab. 5, respectively. On the validation set, we report the performance with both small and large backbones, denoted as Ours-S and Ours-L, respectively. It surpasses previous top-performing methods on both architectures in terms of mAP, demonstrating strong generalization capability. Compared with DyCo3D [14], our approach exceeds it by 4.2% in terms of mAP. The qualitative result is illustrated in Fig. 5. We also make a fair comparison with HAIS [4], the highest mAP is achieved on the validation set.