MOSE: Monocular Semantic Reconstruction Using
NeRF-Lifted Noisy Priors

MOSE adopts a NeRF-based representation to compactly encode scene appearance, geometry and semantics (see Fig. 2). Specifically, three MLPs are used: for spatial points $\mathbf{x}=(x,y,z)$ and their viewing direction $\mathbf{v}=(\theta,\phi)$, a color MLP encodes the appearance as a color field $\mathbf{c}(\mathbf{x},\mathbf{v})$, an SDF MLP represents the geometry as a signed distance function (SDF) field $d(\mathbf{x})$, and a semantic MLP encodes its semantic class label as a semantic field $\mathbf{s}(\mathbf{x})$.

To learn SDF field, we adopt the neural surface representation from NeuS [8], which transforms the volume density $\sigma(\mathbf{x})$ to an SDF value $d(\mathbf{x})$. We sample 3D points $\mathbf{x}_{i}$ along the camera ray $\mathbf{r}$, and utilize volume rendering to accumulate its color $\mathbf{c}_{i}$, normal $\mathbf{n}_{i}$ and semantic logits $\mathbf{s}_{i}$:

where $\hat{\mathbf{C}}$, $\hat{\mathbf{N}}$, $\hat{\mathbf{S}}$ are the predicted 2D color, normal and semantic logits of ray $\mathbf{r}$, respectively; $\alpha_{i}=1-\text{exp}(-\sigma_{i}\delta_{i})$ is the discrete opacity of $i$-th sampled point, $w_{i}=\alpha_{i}\prod_{j=1}^{i-1}(1-\alpha_{j})$ represent its transmittance, $\delta_{i}=\|x_{i+1}-x_{i}\|_{2}$ is the distance between adjacent sampled points.

We use photometric loss $\mathcal{L}_{c}$, normal loss $\mathcal{L}_{n}$ and semantic loss $\mathcal{L}_{s}$ to supervise the network:

where $\mathcal{R}$ are the sampled rays, $\mathbf{C}$ and $\mathbf{N}$ are the corresponding RGB color and normal prior, respectively. Following [9], we apply normal supervision only to pixels satisfying a pre-defined patch-wise similarity metric, indicated by $\Omega_{n}(\cdot)$. With a set of predefined $L$ semantic classes, $\hat{p}^{l}(\mathbf{r})$ is the rendered semantic probability at class $l$ after softmax normalization, and $p^{l}(\mathbf{r})$ is the input 2D semantic supervision.

It is worth to note that we prevent gradients from semantics (Eq. 4) to SDF MLP. We empirically observe that the volume density (i.e., SDF value) often distorts to accommodate input noisy labels, leading to significant geometric degradation. Similar finding has also been found in [5]. As shown in Tab. III, adopting this strategy benefits both surface reconstruction and semantic understanding.

Supervised by imperfect semantic predictions [20, 21], reasonable 2D labels can be directly rendered from a joint compact scene representation [4, 9]. However, we observe frequent discontinuity and inconsistency among semantic rendering, especially on objects shown as noisy patches and bleeding edges in Fig. 3. We suspect that the element-wise fusion lacks an explicit awareness of the local coherence of semantics. To address this issue, we propose to rely on generic images segments obeying image structures like boundaries. We introduce a locally-consistent fusion strategy (LCF), leveraging on 2D segmentation techniques (e.g. SAM[12], super-pixel segmentation [13]), to perceive neighboring pixels and output locally consistent semantics.

Without loss of generality, we use SAM [12] to process each multi-view image and generate class-agnostic masks $\{\mathcal{M}_{k}\}$. We also ablate the choice of using super-pixels [13] in Sec. IV-D. During training, we encourage learned semantic distribution to be consistent within each local segment mask (Fig. 3). Specifically, sampled rays located within the same mask $m$ are clustered into a group $\mathcal{R}_{m}$. For each group, we vote the most probably predicted class $K_{m}$ by counting the number of occurrences of each class. We also tried other voting strategies like soft voting but find negligible difference. We hence stick to current simple yet efficient choice. During training, after obtaining the voted labels of $\mathcal{R}_{m}$, the semantic consistency loss is defined as follows:

where $\hat{p}^{K_{m}}(\mathbf{r})$ is the rendered semantic probability of the voted class $K_{m}$. Although 2D masks $\{\mathcal{M}_{k}\}$ vary across frames, we incorporate LCF in a frame-wise manner and learn a more consistent and accurate semantic field.

Accurately reconstructing scene geometry from images alone is difficult even with normal prior, since degradation usually occurs in low-texture regions due to their inherent shape ambiguity [17]. As shown in Fig. 4, though the rendered normal maps have closely matched input normal priors, the reconstructed walls still exhibit obvious fragmentation.

Recall that previous methods [7, 8] usually use Eikonal loss [22] to regularize the SDF field:

where $\mathcal{X}$ denotes a set of sampled points $\mathbf{x}$ and $d(\mathbf{x})$ is the associated SDF field. One straightforward solution to enhance the continuity of such texture-less planar regions is to increase the weighting of Eikonal loss. However, aggressively applying excessive strength to the Eikonal term worsens geometry by over-smoothing objects (Fig. 4). Thanks to the accurate semantics encouraged by LCF, we could adaptively adjust the strength of Eikonal loss based on learned semantic classes, and propose a semantically-weighted geometric regularization (SGR), defined as follows:

where $\hat{p}_{\mathbf{x}}$ denotes the rendered multi-class semantic probability of sampled points $\mathbf{x}$, $\Phi(\hat{p}_{\mathbf{x}})$ is a semantically-adjusted weighting function, defined as:

where $\hat{l}_{\mathbf{x}}$ is the predicted semantic label of sampled points $\mathbf{x}$, we consider $\mathcal{P}$ to be dominant indoor structures like walls, floors and ceilings, which are also planar regions, and $\mathcal{O}$ to be other object classes. We dynamically adjust the strength of our geometric regularization across different semantic classes, ensuring a stronger SDF smoothness on planar regions while preserving objects’ details. Though SGR does not explicitly optimize for semantics, we observe improvements in semantics owned to better geometry quality, as shown in Tab. III.

We train our model with the following loss function:

where we set the weighting factors as $w_{c}=w_{n}=1$, $w_{s}=w_{con}=0.5$, $w_{sgr}=0.1$, respectively. We sample 512 rays for each batch and optimize using the Adam optimizer [23] with a learning rate of $2\times 10^{-4}$. Our MLPs are trained on 1 NVIDIA V100 GPU for 160,000 iterations ($\sim$10 hours). Hyperparameters of MLPs are similar to [9] and [11] for a fair comparison. To obtain the 3D semantic map, we use the Marching Cubes algorithm [24] for extracting surface mesh and compute the semantic labels of each vertex on the mesh.

In order to evaluate the effectiveness of each component in our method, we conduct ablation studies with different settings and variants: Model-A: directly integrate a semantic MLP into NeuRIS; Model-B: prevent SDF MLP of Model-A from the influence of semantics (i.e, NeuRIS*). Thus Model-B has the same geometry quality to NeuRIS [9]; Model-C: built upon Model-B, with the LCF module. Note that we use Eikonal loss (Eq. 6) to regularize the SDF field in Model-A, Model-B and Model-C; Model-D: built upon Model-B, with the SGR module; Ours: integrate both LCF and SGR modules. We report quantitative results in Tab III and present qualitative results in Fig. 8 and Fig. 9.

We also discuss the sensitivity of our system to different input labels and segmentation priors using labels and segments from Mask2Former [21] and unsupervised super-pixel techniques [13]. We use the COCO pre-trained Mask2Former [27, 5] and remap its output semantic labels to NYU-40 classes. Therefore, its predictions under NYU-40 convention are relatively lower than fine-tuned DeepLabV3+. For 2D segment masks, we take graph-based super-pixels and remove those smaller than 4000 pixels. We focus on evaluating their impact on semantics as the geometry quality is almost comparable. Qualitative results in Fig. 10 show that MOSE still exhibits commendable semantic fusion capability, indicating its robustness to input label qualities. Using super-pixels results in slightly inferior scores compared to those obtained using SAM, mainly due to less accurate segment masks. Tab. IV confirms our observations and shows that our approach is not sensitive to the specific choices of prior predictors and maintains consistent performance across various types of prior predictors.