Neural Surface Reconstruction and Rendering
for LiDAR-Visual Systems

We would like to enable NeRF training for generalized LiDAR-visual systems. LiDAR point clouds provide strong and accurate information of the environment, such as occupied, free, and unknown spaces. An occupancy map can be easily attained through ray casting to represent such information [30], which can serve as an auxiliary acceleration data structure for volume rendering to skip free space. However, due to the different field of views between LiDAR and cameras and the sparsity of the LiDAR point cloud, much space is visible to the camera but not to the LiDAR. Only using occupied information from LiDAR for NeRF training may lead to imcomplete rendering results, therefore, we need to identify the space in need to be encoded for efficiency and complete rendering. To address this, we adopt an occupancy map and categorize the occupancy grid states as free, occupied, visible unknown, and invisible unknown, where the visible unknown and invisible unknown grids are separated from the unknown grids based on whether they are visible in the input image. As illustrated in Fig. 3, we cast images pixels’ rays in the built occupancy map and mark the traversed unknown grids before the first arrived occupied grids or map boundary as the visible unknown grids. We encode the neural fields within a visible-aware occupancy map that only includes the occupied and visible unknown grids to reduce the training workload.

The neural signed distance field represents a scene as a distance field that maps 3D points $\boldsymbol{x}\in\mathbb{R}^{3}$ to signed distance value $s\in\mathbb{R}$. We leverage hash encoding[31] and tiny multi-layer perceptions (MLPs) to form a geometry neural network to encode a scalable signed distance field: $(\hat{s},\hat{\beta})=f_{\mathcal{S}}(\boldsymbol{x})$, where $\hat{\beta}\in\mathbb{R}$ is our proposed spatial-varying scale (Sec. III-C1). Most existing volume rendering methods for SDFs [16, 17] train the NDF via pure images. In contrast, we supervise the NDF using direct 3D LiDAR points.

Given a LiDAR point ${}^{L}\boldsymbol{x}={}^{L}\boldsymbol{o}+t\ {}^{L}\boldsymbol{d}$, where ${}^{L}\boldsymbol{o}$ is the LiDAR origin and ${}^{L}\boldsymbol{d}$ is the LiDAR point’s ray direction, we uniformly sample points from the LiDAR origin to the point along the ray. For one sample point ${}^{L}\boldsymbol{x}_{k}={}^{L}\boldsymbol{o}+t_{k}{}^{L}\boldsymbol{d}$, its ray distance value $s_{k}=t-t_{k}$ as the supervision ground truth value is transformed to the occupancy $o_{k}=\Phi(-s_{k},\hat{\beta}_{k})$, where $\Phi(v,h)=\left(1+\exp\left(-v/h\right)\right)^{-1}$ is the sigmoid function, and $\hat{\beta}_{k}$ is the predicted scale factor, which is indicates the standard deviation in the Logistic function controlling the sharpness of the transition. The binary cross-entropy loss [13] is employed to approximate a signed distance field:

where $\hat{o}_{k}=\Phi(-\hat{s}_{k},\hat{\beta}_{k})$ is the geometry neural network predicted occupancy value, $(\hat{s}_{k},\hat{\beta}_{k})=f_{\mathcal{S}}({}^{L}\boldsymbol{x}_{k})$, and $N_{L}$ is total number of points sampled on a ray.

The neural radiance field maps 3D points $\boldsymbol{x}\in\mathbb{R}^{3}$ and viewing directions $\boldsymbol{d}\in\mathbb{R}^{3}$ to view-dependent colors $\boldsymbol{c}\in\mathbb{R}^{3}$. We leverage hash encoding [31] for multi-resolution feature encoding and spherical harmonics encoding[32] for directional encoding, and these two encodings are concatenated and fed into a radiance MLP to predict view-dependent color: $\hat{\boldsymbol{c}}(\boldsymbol{x},\boldsymbol{d})=f_{\mathcal{C}}(\boldsymbol{x},\boldsymbol{d})$.

Given a pixel ray direction ${}^{C}\boldsymbol{d}$, we sample points $\{{}^{C}\boldsymbol{x}_{k}={}^{C}\boldsymbol{o}+t_{k}{}^{C}\boldsymbol{d}\}$ from the camera origin ${}^{C}\boldsymbol{o}$ along the ray with our proposed structure-aware sampling (Sec. I-A) and the volume rendering gets a pixel color $\hat{\boldsymbol{C}}$ via blending the samples’ colors and densities:

where $\delta_{k}=t_{k+1}-t_{k}$ is the distance between two adjacent samples along the ray, and the density $\hat{\sigma}_{k}$ is transformed from the signed distance value $(\hat{s}_{k},\hat{\beta}_{k})=f_{\mathcal{S}}(\boldsymbol{x}_{k})$ to enable volume rendering of SDF[33]:

where $\hat{M}_{k}=\frac{\partial\hat{s}_{k}}{\partial{}^{C}\boldsymbol{x}_{k}}\cdot{}^{C}\boldsymbol{d}$ is the gradient of the signed distance value along the ray direction, so-called slope. The volume rendering formulation (Eq. 2) can be further abbreviated to $\hat{\boldsymbol{C}}=\sum_{k}T_{k}\alpha_{k}\hat{\boldsymbol{c}}_{k}$, where $\alpha_{k}=1-\exp\left(-\hat{\sigma}_{k}\delta_{k}\right)$ is the opacity and $T_{k}=\prod_{j<k}\left(\exp\left(-\hat{\sigma}_{j}\delta_{j}\right)\right)$ is the transmittance.

To encode the scene’s appearance, a photometric loss between the source pixel color $\boldsymbol{C}_{i}$ and rendered pixel color $\hat{\boldsymbol{C}}_{i}$ is applied:

$N_{C}$ is total number of pixels sampled during the training. The photometric error indirectly adjusts the neural distance field by backpropagation via volume rendering (Eq. 2) and SDF-to-density transformation (Eq. 3).

We further regularize the NDF with Eikonal[14] and curvature [35] loss, and the details can be found in supplementary materials. The overall training loss is defined as follows:

where $\lambda_{rgb},\lambda_{eik}$ and $\lambda_{curv}$ are the weights for the photometric, Eikonal, and curvature losses, respectively.

The Replica dataset [6] provides room-scale indoor simulation RGBD sensor data. We emphasize the issues of extrapolation rendering consistency by uniformly sampling positions and orientations in each scene to generate the extrapolation dataset from Replica. The compared surface reconstruction, interpolation, and extrapolation rendering results are shown in Tab. I. As illustrated in Fig. 4 (Left), we capture more precise geometric details in slim objects while maintaining smoothness. For novel-view synthesis, the proposed structure-aware rendering method yields competitive results in interpolation rendering and achieves a significant improvement in extrapolation rendering compared to other methods. As depicted in Fig. 4 (Right), our method retains both the structure and texture in views of the ceiling, which were not seen with a vertical view in the training dataset.

For real-world datasets, we evaluate three types of trajectory datasets collected with a camera and a LiDAR from the FAST-LIVO2 datasets [5]: forward-facing (Campus), object-centric (Sculpture), and free-view (Culture, Drive) trajectories, as shown in Fig. 1. We use the localization results from FAST-LIVO2 as ground truth poses and use a train-test split for dataset interpolation evaluation, where every 8th photo is used for the test set and the rest for the training set [27]. The quantitative results are shown in Tab. I. RGBD-based methods [24, 28] are not included as they are not compatible with LiDAR-visual systems. Our method demonstrates competitive rendering performance with the state-of-the-art method 3DGS [27] and outperforms others in complex free-view trajectory scenes. The interpolation qualitative results are shown in Fig. 3 (Left). Our method captures precise geometric details in fuzzy objects (leaf) and avoids overfitting occurring in InstantNGP and 3DGS^† (middle wall’s depth). We further present our complete and detailed surface reconstruction results and extrapolation rendering results on the FAST-LIVO2 datasets, as shown in Fig 1 and Fig. 2. In extrapolating views, all methods show some degree of degradation in rendering quality, where our method renders more consistent results than other methods, especially in the free-trajectory-type scene, where the camera moves freely, and lacks co-visible views.

The overall algorithm is shown in Alg. 1 and Fig. 1. Given any desired rendering direction $\boldsymbol{d}$, the algorithm starts from the camera origin (Alg. 1-1), and iteratively steps forward (Alg. 1-1) along the ray direction according to its signed distance value. The gradient of the signed distance field along the ray direction ${M}$ is estimated using linear interpolation (Alg.1-1). A filtered gradient $m$ is updated with a relaxation coefficient $\gamma=0.7$ (Alg.1-1), which adaptively determines the next step size $\delta_{i}$ (Alg. 1-1) for efficiency. At every step, we ensure that the space between two adjacent steps is intersected (Alg. 1-1), wherein we take predicted scale $\beta_{i}$ into account to avoid triggering false revert steps in unconverged NDF. To avoid poor local minima, we keep marching even behind surfaces (Alg.1-1), and until a ray’s transmittance (Alg. 1-1) falls below a threshold $\epsilon_{T}=0.001$. The filtered slope $m$ is verified by sphere tracing, and smaller step sizes near surfaces yield a higher sampling frequency for more accurate slope estimations. Therefore, we apply the estimated filtered slope $M$ in the SDF-to-density transition to avoid expensive analytical or numerical gradient calculations. We further utilize the visible-aware occupancy map to skip the free space for efficiency, as shown in Fig. 1’s skip step.

The Eikonal loss [14] and curvature loss [35] are further employed for both NeRF and NSDF samples to prevent the zero-everywhere and overfitting solutions for the SDF:

where $\nabla f_{\mathcal{S}}(\boldsymbol{x}i)$ and $\nabla^{2}f_{\mathcal{S}}(\boldsymbol{x}_{i})$ are the gradient and Hessian of the SDF at $\boldsymbol{x}_{i}$ calculated by numerical differentiation with a progressively smaller step size [37].

The overall training loss is defined as follows:

where $\lambda_{eik}=0.1$ and $\lambda_{curv}=5\times 10^{-4}$ are the weights for the Eikonal, and curvature losses, respectively. We schedule the $\lambda_{rgb}$ linearly increases from $10^{-4}$ to $10$ during the training to ensure that the NeRF learns on a well-structured NSDF to avoid local minima.

We represent our neural fields following a similar architecture to InstantNGP [31], utilizing a combination of multi-resolution hash encoding and a tiny MLP decoder. The hash encoding resolution spans from $2^{5}$ to $2^{21}$ with 16 levels, and each level contains $2^{19}$ two-dimensional feature vectors. Given any position $x$, the hash encoding concatenates each level’s interpolation features to form a feature vector of size 32. One encoding feature is fed to a geometry MLP with 64-width and 3 hidden layers to obtain the SDF value and scale. Another encoding feature is concatenated with the spherical harmonics encoding of view directions and fed to an appearance MLP with 64-width and 3 hidden layers to obtain the view-dependent color. We sample 8192 rays for NDF training and follow InstantNGP[31] to fix the batch size of point samples as 256000 while the batch size of rays is dynamic per iteration. We take 20000 iterations for training, with outlier removal performed every 2000 iterations. We implement our method using LibTorch and CUDA. All experiments are conducted on a platform equipped with an Intel i7-13700K CPU and an NVIDIA RTX 4090 GPU.

We present more surface reconstruction results and extrapolation rendering results on the FAST-LIVO2 datasets, as shown in Fig 2. We further dive into the performance differences between volume-rendering-based methods (InstantNGP and Ours) and rasterization-based methods (3DGS), as shown in Fig. 4. InstantNGP and Our methods show more reasonable rendering results and fewer artifacts overall, such as the ground in the down-left image. While 3DGS shows powerful capability to capture high-frequency textures, such as the carved wall in the middle image.

So far the bottleneck of the efficiency is greatly impeded by the unbalanced sphere tracing’s steps, once there is a ray that does not converge to the surface, the other converged rays need to wait until it is finished or reach the maximum steps. The training time for a Replica scene takes about 20 minutes and the rendering time for a 1200x680 image takes about 0.07 seconds (13Hz). In the future, it could be solved by conducting a ray-oriented rendering for ray rendering instead of batch rendering.