MonoNeuralFusion: Online Monocular Neural 3D Reconstruction with Geometric Priors

Most of previous online reconstruction works represent the 3D geometry as resolution-dependent SDFs [22, 44] or explicit depth maps [38, 39]. These two representations have disadvantages in the ability of detail expression and result coherence, compared with neural implicit representations, and they are not suitable for volume rendering optimization. Although some methods based on volume rendering optimization achieve impressive and fine-grained surface reconstruction [25, 26, 28, 27], they require a time-consuming optimization from scratch and is hard to extend to an online version without a good initial prediction. Besides, those encoding an entire scene to the parameters of MLP as global features is difficult to learn general geometry priors from datasets. Based on the above, to both leverage the advantages of stronger expression ability of neural implicit scene representations with volume rendering for fine-grained reconstruction, we propose to formulate a neural implicit scene representation (NISR) for a more flexible and effective geometry representation.

Our NISR encodes an entire 3D scene as a sparse feature volume, each voxel containing a continuous SDF decoded by a latent vector fused from image features of multi-view images. Note that compared with the continuous occupancy field in [23], the continuous SDF has a stronger ability for detail expression and is reasonable to supervise on the gradient domain. Besides, the sparse structure would lead to more efficency and effectivness for higher resolution. Finally, the surface mesh can be extracted using Marching Cubes [60] at an arbitrary resolution. One benefit of our NISR is the enabled efficient geometry learning based on volume rendering optimization in the space of latent vectors, which is suitable for the online 3D reconstruction task. Besides, we further pre-train our NISR with the guidance of geometric priors, leading to more effective feature latent vector extraction for fine-grained surface reconstruction.

Sparse Feature Volume Construction. We adopt the sparse feature volume data structure from NeuralRecon [22] to organize the 3D scene’s geometry content, where the entire 3D space is divided into a set of sparse voxels. Each voxel defines a neural SDF decoded by a feature latent vector trilinearly interpolated by the features from its eight corners. Specifically, we gradually construct the sparse feature volumes $F_{\theta}^{l},l\in\{c,m,f\}$($c,m,f$ denotes the coarse, middle and fine levels, respectively) in a coarse-to-fine fashion, with $F_{\theta}^{l}$ being feature latent vectors fused by the 2D CNN-based image encoder from the input multi-view images (Fig. 2). To further improve the overall quality, we adopt a spatial-aware feature fusion based on a transformer module [61] instead of channel-wise average feature fusion used in NeuralRecon. The transformer-based fusion helps to build a more expressive feature latent vectors $F_{\theta}^{l}$. These feature latent vectors $F_{\theta}^{l}$ are further merged with the previous feature latent vectors using a GRU module after sparse 3D CNN, as global feature latent vectors [22]. For brevity, we also notate the final global feature latent vectors as $F_{\theta}^{l}$ in each sparse feature volume.

Sparse Feature Volume Decoder. We decode the feature latent vector $F_{\theta}^{f}$ in the sparse feature volume in the fine level as a continuous SDF, which implicitly represents the geometry content. Specifically, for any query point $\mathbf{x}\in R^{3}$, we use an MLP-based decoder $f_{\theta}$ to predict its signed distance $s$ as:

where $interp$ means the trilinear interpolation in $F_{\theta}^{f}$. Besides, we additionally decode a radiance field $F_{\omega}$ for $F_{\theta}^{f}$ using an MLP-based radiance decoder $f_{\omega}$:

In this section, we propose a novel volume rendering approach to render the geometry content represented by our NISR to any given novel views. Besides rendering the color map, we also render a normal map, which brings extra geometric regularization for the later volume rendering optimization. Specifically, given a casting ray $\{{\mathbf{p_{i}}}=\mathbf{o}+d_{i}\mathbf{v}\}$, where $\mathbf{o}$ is the camera center and $v$ is the view direction, we render both the color map and the normal map along this casting ray by accumulating the measurements of the $N$ sampling points (see Fig. 2) as:

where $\alpha_{i}$ is the discrete opacity value transformed from the SDF value $s_{i}$ decoded by $f_{\theta}$ following NeuS [25], and $T_{i}=\prod_{j=1}^{i-1}(1-\alpha_{i})$ is the accumulated transmittance. $c_{i}$ is the color value decoded by $f_{\omega}$ and $n_{i}$ is the normal estimation that can be calculated by the automatic derivation of the SDF decoder $f_{\theta}$.

Since NISR is a sparse feature volume representation, and is different from the ordinary MLP-based representation like NeRF [24] and NeuS [25], we propose a hierarchical sampling strategy (see Fig. 3(c)) for a better point sampling inside the sparse voxels with the following two steps: (1) ray-voxel intersection sampling; and (2) inside-voxel sampling. Specifically, we first perform a ray-voxel intersection sampling to select sparse voxels that intersect with each casting ray. Then within each intersected voxel, we perform hierarchical inside-voxel sampling to sample specific points by converting the distance of a sampling point from the camera origin in the world space to the sparse space. Note that both uniformly coarse sampling and importance sampling on the top of the coarse probability estimation are processed in the sparse space. Given ray-voxel intersection pairs $\mathcal{P}=\{(r,v)\}$, with their distances of ray-voxel out position in the world space $d_{w}^{(r,v)}$ and in the sparse space $d_{s}^{(r,v)}$ from the camera origin, we perform distance conversion of any sampling point as follows:

where $v$ is the corresponding intersected voxel id of ray $r$ for position sampling. In the end, we get a total of $N_{c}$ coarse sampling points and $N_{i}$ importance sampling points for each ray rendering.

Applying the hierarchical sampling strategy proposed in NeuS without any constraint would result in invalid sampling outside voxels, as illustrated in Fig. 3(a). NSVF, as using a similar scene representation with sparse voxels, yields a self-pruning from dense voxels and samples points guided by sparse voxels which are near the surface as optimization proceeds, thus eliminating the need of hierarchical sampling. However, sparse voxels would not guide point sampling very well in our method due to its more thickness near the underlying surface, and directly applying importance sampling would also lead to invalid sampling outside sparse voxels (Fig. 3(b)). Instead, our hierarchical sampling strategy can achieve more accurate point sampling, i.e., all samplings locate inside voxels. The second row of Fig. 3(a-c) illustrates the rendering results of these three sampling strategies. We can see that some rendered points (in blue boxes) in (a) and (b) are in the air due to the sampled points out of the sparse voxel space, while the rendered points of our method are on the surface (c).

In this subsection, we propose a geometric prior guided geometry learning mechanism, by taking full advantage of the geometric priors both in NISR pre-training and volume rendering optimization for high-fidelity reconstruction. In the NISR pre-training, we leverage effective regularization from geometric priors, such as normal prior and eikonal prior, to learn the encoder-decoder parameters of our NISR for a more detailed surface representation. In the volume rendering optimization, since it does not provide ground-truth surface normals sampled from a ground-truth mesh as in the pre-training stage, we leverage the geometric cues from the extra normal map prediction from RGB images to enhance the geometric details of reconstruction.

NISR Pre-training. For better feature latent vector extraction, we propose to pre-train our NISR beforehand with the geometric priors. Here we formulate the geometric prior as the surface normal loss with eikonal regularization as:

where $f_{\theta}$ is the MLP-based decoder introduced in Equation 1. We sample points on the surface of a ground-truth mesh and compute their normals $\hat{n}$ for supervision. Finally, we train our NISR similar to NeuralRecon[22] but with a geometric prior guided loss function:

where $\mathcal{L}_{o}=\sum_{l=1}^{L}\lambda_{o}^{l}\rm{BCE}(o^{l},\hat{o}^{l})$ denotes the binary cross-entropy (BCE) loss, $\mathcal{L}_{s}=|clamp(s,\tau)-clamp(\hat{s},\tau)|$ represents the clipped L1 loss with $clamp(x,\tau)=\min(\tau,\max(-\tau,x))$, and $\mathcal{L}_{f}=\frac{1}{M}\sum_{i=1}^{M}||f_{i}||_{2}^{2}$ is used to regularize scene feature vectors $f_{i}\in F_{\theta}^{l}$.

Volume Rendering Optimization. Based on the novel volume rendering from our NISR, we propose to optimize the latent vector of the sparse feature volume to perform accurate geometry learning for high-fidelity surface reconstruction. Specifically, we perform the optimization with the following loss function:

where $\mathcal{L}_{rgb}=\frac{1}{M}\sum|\hat{C}(\mathbf{r})-C(\mathbf{r})|$ is the color error between the rendered color $\hat{C}(\mathbf{r})$ and predicted color $C(\mathbf{r})$ commonly used in other reconstruction works [25, 29], and $\mathcal{L}_{normal}=\frac{1}{M}\sum\{|\hat{N}(\mathbf{r})-N(\mathbf{r})|+(1-\langle\hat{N}(\mathbf{r}),N(\mathbf{r})\rangle)\}$ describes the L1 distance and the cosine angle between the rendered normal $\hat{N}(\mathbf{r})$ and predicted normal $N(\mathbf{r})$. $M$ is the number of the rendered pixel in a mini-batch. We use a pretrained out-of-the-box Omnidata model [62] to predict the normal map for each image.

In this subsection, we combine all the components introduced above to build an online system to reconstruct the scene geometry. We build our system similar to a common SLAM system [2], except for the tracking component. In the implementation, We run a mapping thread and an optimization thread in parallel at the back-end, as illustrated in Fig. 4.

Mapping Thread. The mapping thread receives the incoming frames from RGB sequences and selects key-frames in sliding windows for efficiency as in NeuralRecon. Specifically, when the distance between a current frame and the last key-frame is larger than $\tau_{d}=0.1$ or the rotation angle from the last key-frame is larger than $\tau_{a}=15^{\circ}$, a key-frame will be selected. When the size of sliding windows reaches $K=9$, all images and their corresponding camera intrinsic and extrinsic parameters will be used to construct the sparse feature volume. When the feature volume is updated, Marching Cubes is applied to extract the underlying surface mesh. For every key-frame, it will be applied by a normal prediction and inserted into a database of key-frames, preparing for the subsequent volume rendering optimization.

Optimization Thread. The optimization thread runs loops. It selects $B$ optimization key-frames in the key-frame database each time and randomly samples $M$ pixels on each of them to build the optimization function as in Equation 7. We use Adam optimizer to refine the scene representation parameters for $T$ iterations in a loop.

Per-scene Fine-tuning. To further enhance the results for a specific scene, we apply an additional fine-tuning for the whole scene. The same as the optimization thread, we optimize the scene with sampling $M$ pixels on each of $B$ images for $T$ iterations. Since the reconstruction from the online system is already good, a few minutes (less than 9 minutes) fine-tuning is typically enough for getting high-fidelity results.

For the NISR, we use MnasNet [63] as a 2D CNN image encoder to extract multi-level image features and we perform the 3D sparse CNN proposed by [64]. For NISR training, we empirically set the weights of each component in the loss as $\lambda_{o}^{l}=0.8^{l-1},\lambda_{s}=1.0,\lambda_{n}=1.0,\lambda_{e}=0.25,\lambda_{f}=0.001$, and set the clamp threshold $\tau=0.2$. We train our model using Adam optimizer with an initial learning rate of 0.001. For the volume rendering optimization, we empirically set the weights of normal to be 0.05. We select $B=9$ images and sample $M=512$ pixels on each of them as a mini-batch, and sample $N_{c}=16$ points for coarse sampling and $N_{i}=16$ points for importance sampling on each ray and $T=100$ for one loop in our online system. The radiance field $f_{\omega}$ is initialized randomly from the zero-mean Guassian distribution. Moreover, we set $N_{c}=N_{i}=32$ and optimize the scene in $T=5,000$ iterations for further per-scene fine-tuning.

Datasets. We select the training subset of ScanNet [31] as supervision to train our NISR. ScanNet [31] is a popular real-scan RGB-D dataset, containing 2.5 million views with ground-truth 3D camera poses and surface reconstruction in more than 1,500 scans. For evaluation, we first evaluate our approach on the testing subset (100 scenes) of ScanNet, but using the monocular RGB sequences only. Additionally, to evaluate the generalization ability of our approach, we also perform evaluation on other datasets including TUM RGB-D dataset [32] (with 10 monocular RGB sequences) and Replica [33] (with the same 8 monocular sequences by [10]), using the pre-trained NISR from ScanNet without further fine-tuning on these two datasets. Since the TUM RGB-D dataset does not provide ground-truth 3D surface meshes, we apply TSDF fusion [65] to generate 3D surface meshes at a resolution of 2cm for the subsequent evaluation.

Methods for Comparison. We compare our method with state-of-the-art online monocular reconstruction approaches, including GPMVS [38], DeepVideoMVS [39], NeuralRecon [22], and TransformerFusion [23], which are most relevant to ours. Additionally, to evaluate the final reconstruction quality of our approach, we also compare our approach with some previous offline monocular reconstruction approaches, such as Atlas [44], Point-MVSNet [66], and FastMVSNet [67]. During the comparison, since Atlas, NeuralRecon, TransformerFusion and our approach directly generate a surface mesh as output while others only perform multi-view depth estimation, we fuse the multi-view depth maps into the final surface mesh using TSDF fusion [65], thus enabling surface quality comparison for all the compared approaches. Besides, we fine-tune those pre-trained models which are not trained on the ScanNet dataset for a fair comparison.

Metrics. To evaluate the surface reconstruction quality, we adopt several popular 3D surface quality metrics, including accuracy, completion, chamfer distance, F-score (with both precision and recall) [44], and normal consistency [48]. For a comprehensive comparison, we additionally measure the multi-view depth estimation quality, using the widely used depth map accuracy metrics such as Abs Rel, Abs Diff, Seq Rel, RMSE, $\delta$ and Comp [68]. For approaches that do not directly generate multi-view depth estimation like ours, we choose to render the depth maps based on the final surface mesh using pyrender¹¹1https://github.com/mmatl/pyrender.

Since TransformerFusion only provides its evaluation script without releasing the test code or resulting reconstruction meshes, for a fair comparison we conduct comparison using its evaluation script for 3D metrics on the ScanNet dataset. For the other datasets and 2D metrics evaluation, we use the evaluation script from NeuralRecon, which is different from TransformerFusion’s in mesh point sampling. More details can be found in the supplementary materials.

Fig. 5 shows visual comparison results of all the compared online methods and two representative offline methods, including Atlas for a volume-based method and PointMVS for a depth-based method. It demonstrates that the final mesh results of the surface reconstruction approaches are consistently more coherent than multi-view depth estimation methods even in offline configurations. Although NeuralRecon improves details compared with Atlas, it is still far away from fine-grained surface reconstruction. Benefiting from the proposed geometric prior guided NISR with volume rendering, our method not only successfully recovers thin parts or small objects, but also achieves sharper geometry (e.g., sofa in the fourth row), significantly improving the detail of reconstruction.

Evaluation on TUM RGB-D. We also conduct an evaluation on the TUM RGB-D dataset to show the generalization ability of our model. Table III shows the results on some major metrics compared with the other online methods. It can be observed that our method achieves the best performance, in all the surface reconstruction accuracy metrics and depth estimation accuracy metrics except F-score. The main reason of our method achieving a less F-score than GPMVS and DVMVS is the drop on recall (with ours (0.323), GPMVS (0.458), and DVMVS (0.507)). However, our method outperforms them on precision (with ours (0.464), GPMVS (0.343), and DVMVS (0.417)) and also outperforms NeuralRecon both on precison (0.464) and recall (0.258). For qualitative results, we can see that our method still achieves the high-fidelity results with the richest details (e.g., small objects on the table, chairs and monitors) among the compared methods, as shown in Fig. 6. Note that since the multi-view depth maps estimation would not be always robust for globally coherent surface reconstruction, we only show the visual effect of DeepVideoMVS as the best representative. This evaluation demonstrates that our geometric prior guided geometry learning for surface reconstruction can be generalized well into a new dataset, despite only pre-trained on the ScanNet dataset.

Evaluation on Replica. Similarly, we also evaluate our results qualitatively in comparison with NeuralRecon on the Replica dataset. Fig. 7 shows some comparisons of visual effects. We can see that our method recovers more complete surfaces as well as fine-grained details, such as the pillow on the sofa. Especially, our method successfully reconstructs TV cabinet and garbage can with high-quality while NeuralRecon fails.

We evaluate the runtime of our system on a platform with an Nvidia RTX 3090 GPU and Intel Xeon(R) Gold 5218R CPU. Table V provides the detailed timing of each main component among all the test scenes in ScanNet. Our system performs normal prediction for every key-frame in 61.48ms and performs image encoding and feature volume construction for every local fragment (with 9 key-frames) in 34.35ms and 257.13ms, respectively. For volume rendering optimization, the timing of each iteration (including rendering and backward propagation) is 79.29ms. A final mesh is extracted in 295.79ms when our NISR is updated. Since the mapping thread and optimization thread run in parallel, the mapping thread can run at up to 2.83 fragments (about 25.5 key-frames) per second.

We additionally evaluate the influence of results at different interactive frame rates of input stream and plot the curve of quality results on the Chamfer distance, F-score, and normal consistency in Fig. 8. We can observe that the more time for optimization with a lower frame rate, the better results it can achieve. Furthermore, it also demonstrates that our method outperforms the NeuralRecon (with the Chamfer distance 0.080, F-score 0.608, and normal consistency 0.816) and TransformerFusion (with the Chamfer distance 0.069 and F-score 0.655), regardless of the frame rates.

To demonstrate the effectiveness of each main component of our full system, we conduct an ablation study experiment. Table IV lists the quantitative results under different settings. (a)-(g) in the following paragraph denotes the experiment of different settings as illustrated in Table IV with their quantitative results. Fig. 9 and 10 show their qualitative results corresponding to (a)-(g) in Table IV.

Surface Normal Loss. We conduct an experiment of not using surface normal (b) and using surface normal (c) to show the effect of the geometric prior in the training stage. Table IV shows when removing the surface normal loss, F-score and normal consistency (N.C.) of (b) show a significant decrease. Fig. 9 shows the visual effect without (b) or with (c) this geometric prior. We can observe that the reconstruction surface becomes rough and loses geometric details when removing it. In conclusion, the surface normal as the geometric prior could improve details and regularize the NISR for smoother surfaces, playing an important role in the training stage for high-fidelity reconstruction results.

Multi-view Feature Fusion. We evaluate the effect of different types of multi-view feature fusion with channel-wise average (a) or average from the output of transformer blocks (c) . Although we apply the transformer blocks in local fragments instead of global regions used in [45], Table IV(c) shows its effectiveness in feature fusion for improving the final result. From Fig. 9, we can see that the improvement of multi-view feature fusion cannot directly influence the qualitative results of surface reconstruction, where the reconstructed surface (see Fig. 9(b)) is still rough and lacks details. Thus, surface normal is a more relevant factor for fine-grained quality surface reconstruction, while the transformer module is more about improving sparse voxel prediction, leading to a higher recall.

Voxel Size. Although we can obtain a continuous SDF from our NISR and the final mesh can be extracted at an arbitrary resolution, the voxel size of the sparse feature volume also influences the quality of reconstruction. (c), (d), (e) in Table IV and Fig. 9 show the influence of different voxel sizes for the fine level (using 4cm, 8cm and 16cm respectively). It demonstrates that a smaller voxel size will lead to higher quality and more details of reconstruction. Theoretically, further a smaller voxel size can obtain even better results, but it also causes more timing and memory consumption. In our experiment, we find that setting voxel size as 4cm is fine for surface reconstruction in high quality with enough geometric details.

Volume Rendering Optimization. Volume rendering optimization provide a refinement for the sparse feature volume of scene to improve the quality of reconstruction. Compared with (c), using online optimization (f) would improve the recall (e.g. filling some missing regions of a computer display as shown in Fig. 10) and thereby increase the F-score value in Table IV. Besides, geometry cues from normal map prediction could add more details with a slight improvement of normal consistency and enhance the visual effects (e.g., adding sofa crevice in Fig. 10). Furthermore, we also provide a per-scene fine-tuning (g) with a whole scene optimization in only a few minutes (less than 9 minutes). This would further slightly improve the surface reconstruction results both quantitatively and qualitatively.

Color Representation. We evaluate the effect of color representation in the optimization step. Although the radiance field cannot be optimized well in such a short time, the color representation could provide some details which the normal map does not have, further improving the quality of surface reconstruction. Fig. 11 shows that the reconstruction result has more details when using the color representation.

Although we formulate the NISR as the sparse feature volume for a more flexible and effective geometry representation, this representation is heavily dependent on the coverage of predicted occupied voxels. Thus, our approach is limited in its ability to complete missing regions if some voxels cannot be predicted successfully. This missing region is hard to complete during online optimization, leading to a lower recall than the offline methods (see Fig. 12(a)). Second, the quality of normal prediction influences the online optimization and some inconsistent normal prediction would worsen the reconstruction results (see Fig. 12(b)). It might be improved by using multi-view normal consistency check to reduce the impact of inconsistent normals, or even treating normal vectors as optimization parameters to participate in the optimization. Third, although our method could achieve finer details of surface reconstruction, there is still a gap between the reconstructed models and depth scans data for complex thin parts (see Fig. 12(d)) and it is still challenging to reconstruct mirrors (Fig. 12(c)). Lastly, there is still room to improve the timing of our online optimization. The more iterations it optimizes, the better results it could achieve. Therefore, techniques to speed up the rendering optimization [54] can be further adopted to improve the results.