FM-Fusion: Instance-aware Semantic Mapping
Boosted by Vision-Language Foundation Models

The image tagging foundation model RAM [7], recognizes the semantic categories in the image and generates related tags. The open-set object detector, such as GLIP [10] and GroundingDINO [6], reads a text prompt to detect the objects. The text prompt can be a sentence or a series of semantic labels. It extracts the regional image embeddings and matches the image embedding to the phrase of the text prompt through a grounding scheme. The network is trained using contrastive learning to align the image embeddings and text embeddings. The detection results contain a bounding box and a set of open-set label measurements. SAM[4] can precisely segment any object with a geometric prompt. It is trained with 11M images and evaluated in zero-shot benchmarks. SAM demonstrates strong generalization ability across data distribution without fine-tune. The combined foundation models read an image and detect objects with open-set labels and masks. We denote them as RAM-Grounded-SAM. ¹¹1https://github.com/IDEA-Research/Grounded-Segment-Anything.

The foundation models have been applied in a series of downstream tasks without fine-tuning. Without semantic prediction, SAM3D [11] projects the image-wise segmentation from SAM to a 3D point cloud map. It further merges the segments with geometric segments generated from graph-based segmentation [12]. SAM is also combined with a neural radiance field to generate a novel view of objects [13]. On the other hand, combining the SAM or other foundation models with semantic mapping is still an open area.

SemanticFusion [14] is a pioneer work in semantic mapping. It trains a lightweight CNN-based semantic segmentation network [15] on the NYUv2 dataset. SemanticFusion incrementally fuses the semantic labels, ignoring the instance-level information, into each surfel of the global volumetric map. In Bayesian fusing the label measurement, the semantic probability is directly provided by the object detector. Relying on a pre-trained Mask R-CNN on the COCO dataset, Kimera [16] uses similar methods to fuse semantic labels into a voxel map. It clusters the nearby voxels with identical semantic labels into instances. Kimera further constructs a scene graph, which is a hierarchical map representation. Based on Kimera, Hydra [2] utilizes the scene graph to detect loops more efficiently.

On the other hand, Fusion++ [17] directly detects semantic instances on images and fuses them into instance-wise volumetric maps. It further demonstrates that semantic landmarks can be used in loop detection. Later methods use similar methods to construct semantic instance maps but utilize the semantic landmarks in novel methods to detect loops [1][18].

Rather than a pure dense map such as a surfel map or voxel map, Voxblox++ [19] first generates geometric segments on each depth frame[20]. If the object detection masks the complete region of an instance, it can merge those broken segments generated from geometric segmentation. Then, the merged segments with their labels are fused into a global segment map through a data association strategy.

The main limitation of the current semantic mapping methods is the lack of ability to generalize. The supervised object detection networks are trained with limited source data. Considering the majority of target SLAM scenarios do not provide annotated semantic data, object detection can not be fine-tuned on the target distribution. To avoid the issue of generalization, Kimera has to experiment in a synthetic dataset[16], including some experiments that rely on ground-truth segmentation. Lin etc. [1] sets up an environment with sparsely distributed objects to reconstruct a semantic map. Voxblox++ evaluates a few of the 9 semantic classes in 10 scans. Although they propose novel semantic SLAM methods, the semantic mapping module prevents their methods from being used in other real-world scenes.

To enhance robustness in the distribution shift, our method fuses the object detections from foundation models to reconstruct the instance-aware semantic map. We evaluate its zero-shot performance on the ScanNet semantic instance segmentation benchmark. It involves 20 classes in the NYUv2 label-set and evaluates their average precision(AP) in 3D space. We also show the qualitative results in several SceneNN scans, which have been used by the previous semantic mapping works.

As shown in Figure 2, FM-Fusion reads an RGB-D sequence and reconstructs a semantic instance map. Each semantic instance is represented as $\mathbf{f}=\{L_{s},\mathbf{v}\}$, where $L_{s}$ is its predicted semantic class and $\mathbf{v}$ is its voxel grid map. $L_{s}$ is predicted as a label $c_{n}$ over the NYUv2 label-set $\mathcal{L}_{c}$. At each RGB-D frame $\{I^{t},D^{t}\}$ at frame index $t$, RAM generates a set of possible object tags. The valid tags are encoded into the text prompt $q^{t}$. GroundingDINO generates object detections with each of the detection $z^{t}_{k}=\{y_{i},s_{i},q^{t}\}_{i}$, where $y_{i}$ is the predicted open-set label, $s_{i}$ is the corresponding similarity score and $q^{t}$ is the frame-wise text prompt. For each $z^{t}_{k}$, SAM generates an object mask $m_{k}$.

We first construct open-set labels of our interests $\mathcal{L}_{o}$. RAM generates various tags. Many of them are not correlated with the pre-defined labels $\mathcal{L}_{c}$. The labels of interest can be selected by sampling a histogram of measured labels for each semantic class in $\mathbf{L}_{c}$. In the ScanNet experiment, we select $38$ open-set labels to construct $\mathcal{L}_{o}$. Only the tags belonging to $\mathcal{L}_{o}$ are encoded into the $q^{t}$ and sent to GroundingDINO. GroundingDINO matches each detected object with the tags in the text prompt. The tags in $q^{t}$ and label measurements $\{y_{i}\}_{i}$ in each $z^{t}_{k}$ are all from the label-set $\mathcal{L}_{o}$.

In a single image frame, RAM can miss some objects in its generated tags due to occlusion. The missing tags further cause GroundingDINO to detect objects incorrectly. It is a natural limitation of running foundation models on a single image. To address it, we encode the detected labels in adjacent frames into the text prompt. The augmented text prompt $q^{t}=\bar{q^{t}}\cup{U}^{t}$, where $\bar{q^{t}}$ is the valid tags from RAM and $U^{t}$ is a set of measured labels in previous adjacent frames. All the tags in $\bar{q^{t}}$ and labels in $U^{t}$ belong to the $\mathcal{L}_{o}$. The text prompt augmentation can reduce the missing tags generated from a single image. More complete tags improve the detection performance of GroundingDINO.

In our system, each instance maintains an individual voxel grid map $\mathbf{v}$, similar to Fusion++ [17]. Meanwhile, the SLAM module integrates a global TSDF map [21] separately. The advantage of separating semantic mapping and global volumetric mapping is that false or missed object detection can not affect the global volumetric map. So, in each RGB-D frame, all the observed sub-volumes are integrated into the global TSDF map despite the detection variances.

In each detection frame, data association is conducted between detection results and volumes of the existing instances. Specifically, the observed instance voxels are first queried. They can be searched by projecting the depth image into the voxel grid map of all the instances. If an instance is observed, its voxels are projected to the current RGB frame. For a detection $z^{t}_{k}$ and a projected instance $\mathbf{f}_{j}$, their intersection over union (IoU) can be calculated $\Omega(z^{t}_{k},\mathbf{f_{j}})=\frac{m_{k}\cap r_{j}}{m_{k}\cup r_{j}}$, where $m_{k}$ is a detection mask and $r_{j}$ is the projected mask of an existed instance. If $\Omega(z^{t}_{k},\mathbf{f}_{j})$ is larger than a threshold, the detection $k$ is associated with instance $j$.

After data association, we integrate the voxel grid map of matched instances accordingly. Those unmatched detections initiate new instances. An instance voxel grid map $\mathbf{v}$ is integrated using the traditional voxel map fusion method [21]. Specifically, we raycast the masked depth of a detected object and update all of its observed voxels.

As shown in Figure 3, an object is observed by GroundingDINO across frames. Each generated detection result $z^{t}_{k}=\{y_{i},s_{i},q^{t}\}_{i}$ contains multiple label measurements $y_{i}$, the corresponding similarity score $s_{i}$ and a text prompt $q^{t}$, where $y_{i}=o_{m},o_{m}\in\mathcal{L}_{o}$. Based on the associated detections, we predict a probability distribution $p(L_{s}^{t}=c_{n})$, where $c_{n}\in\mathcal{L}_{c}$ and $t$ is the index of the image frame.

We follow the Bayes filter algorithm [22] to fuse open-set label measurements and propagate them along the image sequence. The input to the Bayesian label fusion is detection result $z^{t}_{k}$, semantic probability distribution at the last frame $p(L_{s}^{t-1})$, and a uniform control input $u^{t}$. And it predicts the latest semantic probability distribution $p(L_{s}^{t})$.

The key part in our Bayesian label fusion module is the likelihood function $p(y_{i},s_{i},q^{t}|L_{s}^{t}=c_{n})$, as shown in equation (5). The score likelihood $p(s_{i}|y_{i},q^{t},L^{t}_{s}=c_{n})$ is given by GroundingDINO, while label likelihood $p(y_{i},q^{t}|L^{t}_{s}=c_{n})$ should be statistic summarized. Since GroundingDINO can only detect a label $y_{i}$ if it is given in the text prompt, the label likelihood can be transmitted as equation (6). $\exists o_{m}\in q^{t}$ denotes the detected label $o_{m}$ exists in the text prompt $q^{t}$.

Here, we further expand the label likelihood in equation (6) into two conditional probabilities,

The first term is a detection likelihood while the second term is a tagging likelihood. They can be statistically summarized using the detection results from GroundingDINO and tagging results from RAM. We follow the equation (7) to construct a label likelihood matrix over $o_{m}\in\mathcal{L}_{o}$ and $c_{n}\in\mathcal{L}_{c}$. In the ScanNet training set, we sample $35,000$ image frames with tagging results, detection results, and ground-truth annotation to summarize the statistics. In the Bayesian update step in equation (5), the label likelihood between each pair of $\{o_{m},c_{n}\}$ can be queried from the constructed label likelihood matrix.

As shown in Figure 4(a), parts of the constructed label likelihood matrix are visualized, while the complete likelihood matrix involves the entire $\mathcal{L}_{o}$ and $\mathcal{L}_{c}$. For comparison, we construct a manually assigned label likelihood matrix similar to Kimera. As shown in Figure 4, the statistic summarized likelihood matrix is quite different from the manually assigned one. In the statistical label likelihood, each semantic class can be detected by its similar open-set labels at various probabilities. Those cells beyond the diagonal can also have likelihood values, indicating the probability of falsely measured labels. The summarized likelihood matrix following equation (7) describes the probability distribution of label measurements reasonably.

In actual implementation, the multiplicative measurement update in equation (3) frequently generates over-confident probability distribution, which is also reported in Fusion++[17]. It causes $p(L_{s}^{t})$ can be easily dominated by the latest measurement $z^{t}_{k}$ even if previous label measurements are all different with $z^{t}_{k}$. As a result, in the measurement update, we propagate the probability distribution by weighted addition.

Then, the predicted semantic class for each instance at frame $t$ is $\arg\max_{c_{n}}p(L_{s}^{t}=c_{n})$.

Although SAM has demonstrated promising segmentation on a single image, it generates inconsistent instance masks at changed viewpoints, as shown in Figure 5. The inconsistent masks prevent a correct data association between detections and observed instances. Those mismatched detections are initialized as new instances and cause over-segmentation, as shown in Figure 6(a).

The inconsistent instance mask is a natural limitation for image-based segmentation networks, including SAM and Mask R-CNN. To address it, we utilize spatial overlap information to merge the over-segmentation. For a pair of instances $\{\mathbf{f}_{a},\mathbf{f}_{b}\}$ at detection frame $t$, where $\mathbf{f}_{a}$ is volumetric larger than $\mathbf{f}_{b}$, their semantic similarity $\sigma(\mathbf{f}_{a},\mathbf{f}_{b})$ and 3D IoU $\Omega(\mathbf{f}_{a},\mathbf{f}_{b})$ are calculated,

where $\bar{p}^{t}(L_{s}(a))$ is the normalized semantic distribution, ${\mathbf{v}}$ is an instance voxel grid map and $\hat{\mathbf{v}}$ is the inflated voxel grid map. The voxel inflation is designed to enhance 3D IoU for instances with sparse volume. It can be directly generated by scaling the length of each voxel in $\mathbf{v}$. If the semantic similarity and 3D IoU are both larger than the corresponding thresholds, $\mathbf{f}_{b}$ is integrated into the voxel grid map of $\mathbf{f}_{a}$ and further cleared from the instance map. As shown in Figure 6(b), over-segmented instances caused by inconsistent object masks are merged.

The instance-wise voxel grid map can contain voxel outliers due to noisy depth images being integrated. On the other hand, the global TSDF map is a precise 3D geometric representation. It is because the global TSDF map integrates all the observed volumes in each RGB-D frame, while instance volume only integrates a masked RGB-D frame if the corresponding instance is correctly detected. To filter voxel outliers, we fuse instance-wise voxel grid map $\mathbf{v}$ with the point cloud $\mathbf{P}$ extracted from the global TSDF map. As shown in Figure 7, those voxels in $\mathbf{v}$ that are not occupied by any point in $\mathbf{P}$ are outliers and have been removed. The fused voxel grid map represents the instance volume precisely.

In the instance segmentation benchmark, as shown in Table I, semantic mapping based on Mask R-CNN can only reconstruct a few of the semantic categories. It is because the pre-trained Mask R-CNN is trained using COCO label-set and those new semantic classes in NYUv2 label-set are predicted with $0$ AP. Even for those predictable semantic classes, the pre-trained Mask R-CNN suffers from the issue of generalization and achieve low $\text{AP}_{50}$ scores. In experiment with fine-tune Mask R-CNN, although the mean AP is improved, they still reconstruct a few of semantic classes with $0$ AP. We also notice that Kimera performs significantly better than the implemented Fusion++. We believe the difference comes from their different map management methods. Unlike Kimera ignores the instance-wise segmentation, Fusion++ maintains instance-wise volumes and requires data association. But the fine-tuned Mask R-CNN still generate detections with noisy instance masks, causing a large amount of false data association. As a result, Fusion++ generates instances with too many over-segmentation and maintains a low AP score.

The results demonstrated that semantic mapping based on supervised object detection can be easily affected by image distribution, label-set distribution and annotation quality. On the other hand, boosted by pre-trained foundational models RAM-Grounded-SAM, both Kimera and our method reconstructed semantic instances in higher quality than semantic mapping methods based on the supervised object detection.

However, simply replacing object detectors with foundation models could not utilize the maximum potential of the foundation models. Compared with Kimera using RAM-Grounded-SAM, our method achieved $+15.6$ $\text{mAP}_{50}$. The boosted performance comes from two aspects. Firstly, our probabilistic label fusion predicts semantic class in higher accuracy. As shown in Figure 8(a), Kimera predicts semantics of some sub-volumes falsely. Since Kimera updates the label measurements with a manually assigned likelihood probability and ignores the similarity score provided by GroundingDINO, it is easier to be affected by false label measurements. Secondly, Kimera ignores the instance-level segmentation and reconstructs many over-segmented instances. Some of them are predicted with different semantic labels, as shown in Figure 8(a) and 8(c). However, our method is instance-aware. Each instance volume is maintained separately. Our instance refinement module merges over-segmented instances caused by inconsistent instance masks at changed viewpoints. We further fused instance volume with a global volumetric map. Hence, our instances volumes are spatially consistent and relative precise, as shown in Figure 8(d).

The rest of the ScanNet experiment focus on evaluating each module of our method through an ablation study. As shown in Table.II, the text prompt augmentation, probabilistic label fusion with statistic summarized likelihood, and instance refinement all improve the reconstructed semantic instances.

A visualized example of the Ablation-A is shown in Figure 9. As shown in Figure 9(a), Ablation-A predicts an instance falsely, similar to Kimera. It also predicts overlapped instances with over-confident semantic probability distributions. They can not be merged during refinement due to their low semantic similarity. So, the over-segmented instances can not be merged, as shown in Figure 9(c). On the other hand, our method predicts the corresponding semantic classes correctly. The over-segmented instances are predicted with a similar semantic probability distribution and have been merged successfully, as shown in Figure 9(b) and 9(d).

As shown in Figure 10(a), RAM fails to recognize a table due to the extreme viewpoint, and GroundingDINO cannot detect it either. On the other hand, as illustrated in section III-B, our method maintains a series of labels $U^{t}$ that has been detected in previous $5$ frames. If a label in $U^{t}$ is not given in RAM tags, the corresponding label is added to the text prompt. As shown in Figure 10(b), our method detects the table correctly. Beyond miss detecting objects, the incomplete tags from RAM cause false label measurements in other frames. Hence, Ablation-B reconstructs a few instances with a false semantic class. More results can be found in our supplementary video.

To sum up, simply replacing traditional object detectors with RAM-Grounded-SAM to construct the semantic map improves the semantic mapping performance significantly. However, false label measurements, inconsistent instance masks, and missed tags in the text prompt still exist in foundation models. They limit the performance of semantic mapping. We consider those limitations of foundation models. Compared with Kimera using RAM-Grounded-SAM, our method further improves $\text{mAP}_{50}$ by $+15.6$.

In the SceneNN experiment, we kept using the label likelihood matrix $P(y_{i}=o_{m},\exists o_{m}\in q^{t}|L_{s}=c_{n})$ summarized in ScanNet and compare it with Kimera.

As shown in Figure11, Kimera reconstructed some instances with false labels and over-segmentation, similar to its reconstruction in ScanNet. On the other hand, our semantic prediction is more accurate and significantly less over-segmentation. The quantitative results can be found in Table III. Although our statistical label likelihood is summarized using ScanNet data, we have not observed a domain gap in implementing it in SceneNN. One of the reasons is that foundation models preserve strong generalization ability. RAM-Grounded-SAM maintains a similar label likelihood matrix across the image distribution. For example, a door is frequently detected as a cabinet in both ScanNet and SceneNN datasets, which are highlighted in red in Figure9(a) and Figure11(a). Hence, our statistical label likelihood can be used across domains.

So far, the system run offline. As shown in Table. IV, the total time for each frame is $1039.6$ ms. Although it is not a real-time system yet, many modules can be optimized in the future. SAM-related variants have been published to generate instance masks faster[25]. In FM-Fusion, a few modules are implemented with Python, the efficiency can be further improved by deploying it with C++. That would be one of our future works.