Semantic Dense Reconstruction with Consistent Scene Segments

Early 2D semantic segmentation methods, including Faster R-CNN [5], Mask R-CNN [4], make instance mask predictions before object semantic recognition. More recent networks [19, 20, 21] are anchor-based approaches that predict boxes’ offsets relative to a collection of fixed boxes. Although semantic instance segmentation has achieved reliable results, more and more segmentation tasks have put forward requirements for efficiency. YOLACT [6] is the real-time (more than $30$ fps) instance segmentation algorithm that is updated as YOLACT++ [7] by incorporating deformable convolutions into the backbone network. However, those approaches focus on the single image processing topics, leading to inconsistent scene interpretation issues due to illumination changes, occlusions and other variations over time. To solve the problem, video-based instance segmentation (VIS) [22] tracks object instances interested in a video sequence, but these methods require the target information determined in the first frame. Different from them, each RGB-D pair is segmented in geometric and semantic manners to obtain correct boundaries in this paper. Moreover, we build a global sparse semantic map in real-time to maintain 2D consistent semantic segments.

3D scenes are usually represented by point clouds since this unstructured type of data is efficient and contains rich geometric information compared with 2D images. 3D ShapeNet [23] is one of the first works in this area. Via training a 3D convolutional deep belief network from a ground truth shape database. Inspired by ShapeNet, PointNet [24] and PointNet++ [1] exploit a more efficient representation of 3D surfaces. To solve issues in spatio-temporal perception tasks, MinkowskiNet [9] making use of 4D dimensional convolutional neural layers outperforms efficient and accurate performance on semantic segmentation. In addition to extracting deep features from point clouds, the researchers also proved that the joint 2D and 3D features complement each other in local areas and obtain better performance. 3DMV [11] is presented as a joint 3D-multi-view approach built on the core idea of combining spatial and RGB features in an end-to-end architecture. Based on 3DMV, 3D-SIS [12] fuses 2D color images with 3D geometry features by projecting deep features from 2D RGB view into the voxel grid for instance segmentation. BPNet [3] enables the bidirectional feature interacting between 2D and 3D CNNs in multiple pyramid levels via the proposed bidirectional projection module. In this paper, after obtaining 2D consistent semantic maps from the system, the SP-Block is proposed to capture features from those 2D semantic segments and transform generated feature volumes to another feature domain encoded from MinkowshiNet [9].

Based on RGB-D images, tracking and mapping methods [25] are used to build global 3D maps that are bridges for complete scene understanding systems. Multi-feature-based trackers [18, 17] achieve robust estimations in indoor scenes, but those mapping parts aim to maintain sparse features for removing camera pose drift rather than reconstructing dense maps. Different from those methods, KinectFision [26] and BundleFusion [27] focus their mind on dense reconstruction by using GPUs to obtain on-the-fly 3D scene reconstruction. In our system, both tracking and dense mapping are important to be implemented via multi-threads on a CPU, which does not have high requirements for hardware.

Texturing 2D semantic maps to dense maps is a traditional way to build online semantic reconstruction systems. SemanticFusion [28] is a pioneer 3D semantic segmentation system, which fuses semantic surfels labeled by convolutional neural networks incrementally. PanopticFusion [29] extends the 2D-to-3D mapping framework to TSDF voxels where semantic labels come from pixel-wise panoptic prediction networks. Those 2D-to-3D architectures are direct to follow, but they rely on the performances of 2D semantic segmentation methods, which limits further improvement. Instead of using 2D semantic results, SceneGraphFusion [15] estimates a semantic scene graph based on a 3D instance map while the performance also relies on the instance fusing method. Instead of sticking those 2D labels in the dense model or capturing deep features from point clouds only, we try to make the 2D objects’ labels consistent and let our 3D segmentation network segment the mesh model in one shoot, which tends to be more accurate since global structures and local details are considered together.

To achieve robust pose estimation performances in the general indoor environments, the camera tracking module used in this paper is our Manhattan-SLAM [17] that exploits points, lines, plane features, and spatial constraints between them, including parallel and perpendicular planes. Note that this tracker can be replaced by other tracking methods.

For the mapping part, we have two main changes proposed. Compared with the surfel map generated by Manhattan-SLAM, 3D objects’ semantic labels are obtained from geometric and semantic segments of every keyframe, which are fused into a global sparse semantic map, as shown in Figure 4(e), to support consistent 2D semantic mask generation. Furthermore, a dense mapping module is created to propose a smooth dense mesh map. Since sparse maps cannot provide enough information for robots, our system generates a dense mesh map incrementally based on CPU. When a new keyframe is generated from the tracking thread, we make use of the estimated camera pose and the RGB-D pair to build a dense TSDF [30, 31] model. After that, the marching cubes method [32] is exploited to extract the smooth surface from voxels.

The consistent semantic segmentation strategy in this architecture is also an important module that is responsible to provides stable 2D semantic instance predictions of different views. In this module, two segmentation branches, learned [6] (Figure 4(b)) and geometric [16] (Figure 4(c&d)) methods, are used to deal with RGB and depth maps, respectively. The geometric part segments objects’ areas based on the 3D shape by computing normals from depth maps, while the learned one predicts the semantic instance masks directly after training convolutional neural networks on large datasets. To remove the boundary noise of semantic labels, only union regions of those two maps are accepted by the system.

As we all know, images that only capture part information of objects are useful for 3D semantic segmentation since more details and boundaries information can be obtained from there. Those partial scans, however, bring huge challenges to 2D semantic segmentation networks. To keep the consistency of segments, we take advantage of camera poses and the global semantic sparse map to correct those ill-posed results.

In this module, an encoder-decoder network is implemented for the final 3D dense semantic segmentation task. In the encoder module, the proposed SP-Block is connected with the original encoder of MinkowskiNet to build deep embedding features that are decoded in semantic predictions.

Benefiting from the SP-Block, deep features from 2D and 3D domains can be fused to predict 3D semantic segments from the 3D dense reconstruction.

In this paper, each RGB image is fed to YOLACT [6] to segment instances and predict objects’ labels, there are two types of outputs, label $R_{rgb}$ and probability maps $R_{p}$, from the network, where the first one codes each index of detected objects as shown in Figure 4(a) while the three channels of the second map (see Figure 4(b)) is used to save the corresponding probabilities.

Since boundaries of semantic masks generated from an RGB image are commonly noisy, we extract areas with discontinuous depth information from the corresponding depth map. Given depth maps, geometric-based shape segmentation methods [2, 16] are used to segment the scene into different instances according to the normal edge analysis. As shown in Figure 4, the TV is segmented from a wall since the normal map detects disconnection regions between them. Therefore, a filtered segmentation map $R^{*}$ is obtained by

here $R_{d}$ is a binary map where instance-covered pixels are denoted as 1. We have to note that those segments extracted from the RGB image will be removed if they do not exist in the geometric map, which will be completed when the information appears in both semantic and geometric images.

To keep the consistency of objects captured in different views, we maintain a semantic sparse point clouds map containing different geometric landmarks [17, 18] and semantic objects.

Each 3D object of the map needs to be initialized and updated in the whole tracking process. First, 3D objects $O_{i},i\in(1,n)$ where $n$ is the number of 3D objects saved in the map are re-projected to a new keyframe to obtain 2D re-projections regions $o_{i}^{rp}$. We compute the IoU (Intersection over Union) between $o_{i}^{rp}$ and semantic labels $o_{j},j\in(1,m)$, here $m$ is the number of detected objects of $R^{*}$. If the IoU is more than a threshold $t_{iou}=0.4$, we continue to check the probability of $o_{j}$. $o_{i}^{rp}$ and $o_{j}$ are matched when index and probability of $o_{j}$ are matched with $o_{i}^{rp}$ at the same time. Otherwise, we continually check the probabilities of semantic labels in $R^{*}$, if they are more than the threshold $t_{p1}=0.9$, those objects will be noted as new comes and be fused into the map.

In the objects’ fusion process, when the re-projected information is matched with the current keyframe’s semantic labels, we will also update the probability of $O_{i}$. If the probabilities of those 2D semantic segments are more than $t_{p1}$, the probability of $O_{i}$ will be increased since the object is reconfirmed in different views. If those new segments that satisfy the IoU requirement are in high probabilities but in different semantic labels, the weights of related 3D objects $O_{i}$ will be decreased, which will be removed from the map when the weights are less than $t_{p2}=0.7$.

3DMV [11] and 3D-SIS [12] propose differentiable projection layers mapping 2D features to corresponding 3D voxels. However, this operation brings noisy to the 3D branch if features from 2D origins have no matches in 3D models. Therefore, we re-project [3] each 3D point $P=[x,y,z]$ of the model to 2D images to detect intersections that are represented into binary masks $B_{k}$ by using,

where $\ocircle$ is the $and$ operation between the $k^{th}$ semantic input image $R_{k}^{*}$ and the re-projected image from the 3D model. $T_{kw}$ is the 6 DoFs pose matrix from the world to the $k^{th}$ camera coordinate and $K$ is the intrinsic matrix.

As shown in Fgiure 5, the embedding $\widetilde{F}_{3D}$ is constructed by two branches, $F_{3D}$ from the encoder of MinkowskiNet [9] and $\hat{F}_{3D}^{V}$ from our semantic projection block introduced in this section.

First, we extract deep feature pyramids by using ResNet-18 [33] from multi-view semantic images. There are four levels of feature maps $F_{2D}^{Li},i\in[1\dots 4]$ extracted from each image. To maintain compatibility with deep features from the encoder of 3D-Net [9], we project each feature channel of $F_{2D}^{Li}$ to the shape of $N\times 1$ based on the camera pose and intrinsic matrix, where $N$ is the number of voxels in the 3D model. Therefore, each level’s feature maps (with $C$ channels) are transferred to a shape as $N\times C$. And then at the same level of $V$ views, following [3] we concatenate those transferred shapes along the channel direction to obtain $\hat{F}_{3D}^{v,l}$ with the size of $N\times C\times V$.

Then the DoT operation that is constructed by four 3D sparse convolutional layers and a sparse max-pooling layer is proposed to aggregate feature volumes $\hat{F}_{3D}^{v,l}$ from different views.

where $L$ is the size of feature levels while $V$ is the number of views. Via the DoT operation, the shape of $\hat{F}_{3D}^{v,l}$ is transferred with the same size of $F_{3D}$. Moreover, the spatial and semantic information from different views is fused.

Finally, corresponding levels of encodes’ features $F_{3D}$ based on MinkowskiNet and $\hat{F}_{3D}^{V}$ based on our SP-Block are fused by a concatenation operation as

where $\widetilde{F}_{3D}$ is the fusion embedding that is fed to the decoder to obtain semantic prediction results.

The decoder used in this paper comes from MinkowskiNet [9], which generates a semantic label for every 3D point. Readers can refer to MinkowskiNet [9] for details. Furthermore, we make use of the conventional cross entropy [34] to supervise the whole encoder-decoder pipeline. During the raining process, we exploit the ground truth 2D semantic labels and 3D reconstruction models of each scenario as the input of our network.

The network is implemented on the PyTorch platform, which exploits the SGD optimizer with a learning rate of $0.01$ and a momentum of $0.9$. Furthermore, the network is trained on the machine with 4 NVIDIA GeForce 2080TI GPUs and 64GB RAM, where the batch size is set to 12 in 100 training epochs.

The size of RGB-D images fed to our tracking system is $480\times 640$, while the size of semantic images for SP-Block is downsampled to $240\times 320$. The channels’ size of four layers in the feature pyramid are $512$, $256$, $128$ and $96$, respectively. After the DoT operation, the channels’ size of four layers in $\hat{F}_{3D}^{V}$ are $256$, $128$, $128$ and $96$, respectively.

First, the reconstruction accuracy is evaluated between different approaches. ElasticFusion [38], BundleFusion [27] and InfiniTAM [39] are dense reconstruction methods, where the first one reconstructs surfel-based models and the others obtain dense mesh model as well as ours. In the lr2 and lr3 sequences, a complete room can be found. Therefore, we compare the reconstruction errors among those methods. Compared with those methods, our method is more robust in different sequences and we only need a CPU for the on-the-fly dense reconstruction.

Moreover, we compare the qualitative reconstruction results between ours and the ground truth of ScanNetv2 that is built from BundleFusion. As shown in Figure 7(a), our method can reconstruct the chair completely. Benefiting from our accurate pose estimation module and the smooth dense reconstruction strategy, the refrigerator is reconstructed more accurately than the ground truth. Related semantic segmentation results are shown in Figure 7(b).

Following the common evaluation metrics in previous works, the standard mean Intersection over Union (mIoU) and mean Accuracy (mAcc) as used to evaluate the performance of our network. Our 3D semantic segmentation results are shown in Table II,III, where we compare our network with state-of-the-art pipelines. Similar to our methods, SF [28], SR [40], SPV [43] and PF [29] are semantic reconstruction systems based RGB-D images, while BPNet [3] deals with point clouds.

Semantic segmentation results of 20-class objects/scenarios from different approaches are listed in Table II, FPC [42] achieves good predictions in $5$ classes, especially in bath,bed and wall instances. However, it does not understand bookshelves existed in scenes. Compared with those methods, our network achieves more robust performances. For most object classes, our method obtains better results. To be specific, we obtain the best results at chair, desk, sofa and toilet instances since our 2D semantic masks based on Yolact [6] can detect those objects from RGB images, which proves that the proposed SP-Block improves the segmentation results. Similar to our method, BPNet is a 2D-3D joint method. But, it feeds the network with RGB images. According to the results of Ours$+$, which makes use of the 2D semantic masks provided by ScanNetV2 to take place of Yolact’s results, the performances of our network are improved shapely and perform better over BP-Net in the majority of categories listed in Table II.

Furthermore, we compare the mIoU and mAcc between different methods as shown in Tabel III, where BPNet and ours results are based on the voxel size of $5$ cm while SPV makes use of $1$ cm voxels. When the voxel size becomes smaller, more detailed information is saved in models, which tends to produce accurate predictions. However, it brings more intensive computation and higher requirements for hardware.

To verify the generality of our algorithm, the network is tested on the 3RScan dataset. As shown in Figure 8(b), our approach accurately predicts the table and chair of the scene, but MinkowskiNet fails to detect them.