PC-NeRF: Parent-Child Neural Radiance Fields under Partial Sensor Data Loss in Autonomous Driving Environments

Motivated by NeRF’s capability to render photo-realistic novel image views, several works have explored the potential of NeRF on LiDAR point cloud data for novel view synthesis [32, 33, 14] and robot navigation [4, 31, 39].

The goal of novel view synthesis is to generate a view of a 3D scene from a viewpoint where no real sensor image has been captured, allowing the opportunity to observe real scenes from a virtual perspective. Neural Fields for LiDAR (NFL) [14] combines the rendering power of neural fields with a detailed, physically motivated model of the LiDAR sensing process, thus enabling it to accurately reproduce key sensor behaviors like beam divergence, secondary returns, and ray dropping. LiDAR-NeRF [32] converts the 3D point cloud into the range pseudo image in 2D coordinates and then optimizes three losses, including absolute geometric error, point distribution similarity, and realism of point attributes. Similar to LiDAR-NeRF, NeRF-LiDAR [33] has also employed the spherical projection strategy and consists of three key components: NeRF reconstruction of the driving scenes, realistic LiDAR point clouds generation, and point-wise semantic label generation. However, if more than one laser point is projected onto the same pseudo-pixel, only the one with the smallest distance is retained [32]. Besides, it is particularly noticeable when using small resolution range pseudo images, as spherical projections can lead to significant information loss [40].

In robotics, LiDAR-based NeRF is usually proposed for localization and mapping. IR-MCL [31] focuses on the problem of estimating the robot’s pose in an indoor environment using 2D LiDAR data. With the pre-trained network, IR-MCL can synthesize 2D LiDAR scans for an arbitrary robot pose through volume rendering. However, the error between the synthesized and real scans is relatively large. NeRF-LOAM [4] presents a novel approach for simultaneous odometry and mapping using neural implicit representation with 3D LiDAR data. NeRF-LOAM employs sparse octree-based voxels combined with neural implicit embeddings, decoded into a continuous signed distance function (SDF) by a neural implicit decoder. However, NeRF-LOAM cannot currently operate in real-time with its unoptimized Python implementation. LocNDF [39] utilizes neural distance fields (NDFs) for robot localization, demonstrating the direct learning of NDFs from range sensor observations. LocNDF has raised the challenge of addressing real-time constraints, and our work endeavors to investigate this challenge.

In contrast to projecting the LiDAR point cloud onto a range pseudo-image, our proposed PC-NeRF handles 3D LiDAR point cloud data directly. Besides learning the LiDAR beam emitting process, our proposed PC-NeRF explores the deployment performance of NeRF-based methods.

When large-scale scenes such as where the autonomous vehicles drive need to be represented with high precision, the model capacity of a single NeRF is limited in capturing local details with acceptable computational complexity [5, 26, 27]. For large-scale 3D scene reconstruction tasks, Mega-NeRF [5], Block-NeRF [26], and Switch-NeRF [27] have adopted the NeRF sub-module solution. Mega-NeRF decomposes a scene into cells with centroids and initializes a corresponding set of model weights. At query time, Mega-NeRF produces opacity and color for a given position and direction using the model weights closest to the query point. Like Mega-NeRF, Block-NeRF [26] proposes dividing large environments into individually trained Block-NeRFs, which are then rendered and combined dynamically at inference time. For rendering a target view, a subset of the Block-NeRFs are rendered and then composited based on their geographic location compared to the camera. Switch-NeRF [27] proposes a novel end-to-end large-scale NeRF with learning-based scene decomposition and designs a gating network to dispatch 3D points to different NeRF sub-networks. The gating network can be optimized with the NeRF sub-networks for different scene partitions by design with the Sparsely Gated Mixture of Experts.

Regarding space-division-based NeRF on a smaller scale, further space division is employed for faster and higher-quality rendering [28, 29, 30]. DeRF [28] proposes Voronoi Diagrams spatial decomposition and provides more efficient inference (with the same rendering quality) and higher-quality rendering (for the same inference cost) than the original NeRF [21]. KiloNeRF [29] demonstrates that real-time rendering is possible using thousands of tiny MLPs instead of one extensive multilayer perceptron network (MLP). Rather than representing the entire scene with a single, high-capacity MLP, KiloNeRF represents the scene with thousands of small MLPs. Neural Sparse Voxel Fields (NSVF) [30] define a set of voxel-bounded implicit fields organized in a sparse voxel octree to model local properties in each cell. Specifically, NSVF assigns a voxel embedding at each vertex of the voxel and obtains the representation of a query point inside the voxel by aggregating the voxel embeddings at the eight vertices of the corresponding voxel. The aggregating vertice voxel embedding is further passed through an MLP to predict the geometry and appearance of that query point.

Our work also draws on the ideas of spatial division on both large-scale and small-scale for autonomous driving scene representation. The proposed PC-NeRF divides the holistic driving spaces into multiple large blocks and further extracts the geometric segments in each block for detailed local representation.

To efficiently represent the autonomous vehicle’s traversing environment, we construct multiple rectangular-shaped parent NeRFs (as Fig. 2(b) illustrates) one after the other along the vehicle’s trajectory, creating a new parent NeRF when the autonomous vehicle orientation variation exceeds a given threshold. Constructing parent NeRF requires three considerations: the effective LiDAR point cloud utilization, the driving environment representation, and the NeRF near/far bounds calculation. Since the point clouds become sparser the further away from the LiDAR origin while the NeRF model itself is a dense volumetric representation, laser points within a certain distance from the LiDAR origin rather than the holistic point clouds are chosen to train our proposed PC-NeRF model. Besides, the driving environments featured with roads, walls, and vehicles can be tightly enclosed by bounding boxes. Therefore, each parent NeRF’s space is represented as a large Axis-Aligned Bounding Box (AABB) in our work, which also makes it easier to calculate the related near and far bounds for rendering.

To further represent the environment at the segment and point levels and efficiently capture the approximate environmental distributions under partial sensor data loss, we divide the point clouds in a parent NeRF into multiple child NeRFs that enclose each environmental geometric segment. Since the geometric segment quantity is limited and the child NeRFs’ space can also be represented by AABBs, building child NeRFs from the raw point cloud is a fast way to generate detailed environmental representations. With less space volume, child NeRFs can represent a larger environment with the same model capacity by learning more detailed volumetric densities of segments. In addition, segment-level child NeRFs can compensate for the inadequate parts of the environmental representation caused by the sparsity of LiDAR point clouds.

The point cloud allocation for child NeRF can be divided into the following three steps, and the results are shown in Fig. 2(a) and Fig. 2(c). Step 1: Extract the ground point clouds from the fused point clouds related to road ground, roadside sidewalk, etc. The ground plane is one significant geometric segment in the driving environment and is spatially adjacent to the other individual geometric segments. Therefore, distinguishing it helps to extract other geometric segments further accurately. Step 2: Cluster the remaining non-ground point clouds into various segments using region-growing clustering. The advantages of the utilized clustering method are excellent scalability for operation on large-scale point clouds and the adaptive ability to cluster different shaped and sized objects. Step 3: Construct child NeRF AABBs. The AABBs of the segmented point clouds obtained from step 1 and step 2 can be used as child NeRFs’ spatial extent. After the division process above, the fused point clouds are distributed into a limited quantity of child NeRFs.

In NeRF volume rendering, the depth value of a ray $\mathbf{r}$ is synthesized from the weighted sampling depth values $t$ between the near and far bounds $t_{\mathrm{n}}$ and $t_{\mathrm{f}}$ by:

where $\mathbf{r}(t)=\mathbf{o}+t\mathbf{d}$ represents a ray with camera or LiDAR origin $\mathbf{o}$ oriented as $\mathbf{d}$, and the volume rendering integration weights $w(t)$ is calculated by:

where $\exp(-\int_{t_{n}}^{t}\sigma(s)ds)$ is the visibility of $\mathbf{r}(t)$ from origin $\mathbf{o}$, and $\sigma(t)$ is the volumetric density at $\mathbf{r}(t)$.

We apply LiDAR point cloud data to train our proposed PC-NeRF with model parameters $\bm{\theta}$. In the global coordinate system, the LiDAR origin position of the $i$-th frame is represented as $\mathbf{o}_{i}=(x_{i},y_{i},z_{i})$. The depth, direction, and the corresponding child NeRF order number of the $j$-th laser point $\mathbf{p}_{ij}=(x_{ij},y_{ij},z_{ij})$ in the $i$-th point cloud are $d_{ij}=\|(\mathbf{p}_{ij}-\mathbf{o}_{i})\|_{2}$, $\mathbf{d}_{ij}=(\mathbf{p}_{ij}-\mathbf{o}_{i})/d_{ij}$ and $k_{ij}$ respectively. Then, we can get one LiDAR ray $\mathbf{r}_{ij}=[\mathbf{o}_{i},\mathbf{p}_{ij},d_{ij},\mathbf{d}_{ij},k_{ij}]$. We further design three LiDAR losses and child NeRF segmented sampling for our proposed PC-NeRF model, illustrated in Fig. 3.

Parent NeRF depth loss: In Sec. III-A, the autonomous vehicle driving environment is represented efficiently using parent NeRF. We use the intersection of the LiDAR ray $\mathbf{r}_{ij}$ with its corresponding parent NeRF surface as the far bound $f_{ij}^{p}$. To adequately represent the volumetric distribution of the whole driving environment, we set parent NeRF depth loss as:

where the integration lower limit $t_{0}$ is set to 0. It is also recommended to set $t_{0}$ as $0.5\,m$ or $1\,m$ considering the space occupied by the LiDAR or the autonomous vehicle. And $\mathcal{L}_{\mathrm{L1}}^{{}^{\prime}}(x,y)=0.1\cdot\text{SmoothL1Loss}(10\cdot x,10\cdot y)$ is an extension of SmoothL1Loss that shifts the $|x-y|$ turning point from $1\,m$ to $0.1\,m$ to improve model sensitivity.

The commonly used depth inference approach uses the synthetic depth between the scene’s near and far bounds as the inference depth [21, 31, 37, 23, 25], which is used in Eq. 3 and calculated as follows:

Segment-to-point hierarchical representation strategy: To find object surface points effectively and address the reduction of point cloud data, we further propose a segment-to-point hierarchical representation strategy. The depth difference $f_{ij}^{p}-t_{0}$ between parent NeRF near and far bounds in large-scale outdoor scenes can be about ten times greater than that in indoor scenes, making it more difficult for the NeRF model to find the object surface through sampling. Given that it is much easier to find segment-level child NeRF space than to find point-level object surface points between the parent NeRF near and far bounds, we propose a segment-to-point hierarchical representation strategy, which first finds the child NeRFs that intersect the LiDAR ray and then finds object surface points inside the child NeRFs that intersect the LiDAR ray.

We use the intersections of the LiDAR ray $\mathbf{r}_{ij}$ with its corresponding child NeRF surface as the child NeRF near and far bounds $[n_{ij}^{c},f_{ij}^{c}]$. It is pretty evident that the depth difference between child NeRF near and far bounds $(f_{ij}^{c}-n_{ij}^{c})$ is much smaller than that between parent NeRF near and far bounds. Considering that the object surface has a certain thickness and the object surface may appear at the child NeRF bounds, we slightly inflate the child NeRF near and far bounds as $[n_{ij}^{c}-\varepsilon,f_{ij}^{c}+\varepsilon]$, where $\varepsilon$ is a small inflation coefficient.

Child NeRF free loss: To make the process of finding the segment-level child NeRF’s corresponding space faster, we propose the child NeRF free loss in Eq. 5 as no opaque objects exist in $(t_{0},n_{ij}^{c}-\varepsilon)$ and no objects can be observed in $(f_{ij}^{c}+\varepsilon,f_{ij}^{p})$ in a large number of cases.

Child NeRF depth loss: Based on child NeRF free loss, we propose child NeRF depth loss to find object surface points inside the child NeRF, which is calculated by:

To let the child NeRF free loss and the child NeRF depth loss have a smooth transition at the child NeRF bounds, $\mathcal{L}_{ij}^{\mathrm{cd}}(\bm{\theta})$ is further modified to:

where $\gamma$ is a constant designed to represent the smooth transition interval on a LiDAR ray between the child NeRF free loss and the child NeRF depth loss.

To sum up, the total training loss from one LiDAR ray $\mathbf{r}_{ij}$ contains all the losses mentioned above:

where $\lambda_{\mathrm{pd}}$, $\lambda_{\mathrm{cf}}$ and $\lambda_{\mathrm{cd}}$ are the parameter to jointly optimize different losses.

Child NeRF segmented sampling: To efficiently find the objects in large-scale scenes even under partial sensor data loss, we propose a child NeRF segmented sampling method for objects more likely to be found in and around the child NeRFs. Assuming that $N$ points are sampled uniformly along the LiDAR rays, the child NeRF segmented sampling is sampling $\lambda_{\mathrm{in}}\cdot N$ points in $[n_{ij}^{c}-\varepsilon,f_{ij}^{c}+\varepsilon]$ and sampling $(1-\lambda_{\mathrm{in}})\cdot N$ in $[t_{0},f_{ij}^{p}]$, as shown in Fig. 3. Therefore, child NeRF segmented sampling guarantees that at least $\lambda_{\mathrm{in}}\cdot N$ points are sampled inside child NeRF, which means that at least $\lambda_{\mathrm{in}}\cdot N$ sampling points are sampled near the real object.

Most current depth inference methods are one-step depth inference methods, similar to Eq. 4. In contrast, we provide a two-step depth inference method to infer more accurately based on the remaining sensor data. We first search in the parent NeRF AABB to acquire the child NeRF AABBs potentially intersecting the LiDAR ray $\mathbf{r}_{ij}$ and then conduct further inference in the child NeRF AABB’s near and far bounds $[\hat{n_{ij}^{c}},\hat{f_{ij}^{c}}]$ intersecting with the LiDAR ray, as shown in Fig. 4(a). To this end, we first select the child NeRF whose AABB outer sphere intersects the ray and then use the Axis Aligned Bounding Box intersection test proposed by Haines [41], which can readily process millions of voxels or AABBs in real time [30]. If the LiDAR ray does not intersect any child NeRF AABB, the space extent of all child NeRF AABBs for this LiDAR ray should be slightly inflated in incremental steps, and the above inference should be performed again.

To reduce the potentially severe impact of mistaking free space for object space, we keep the inferred depth values within one of the child NeRF near and far bounds, as shown in the first subfigure of Fig. 4(b). In real-world applications, the LiDAR ray usually intersects multiple child NeRF AABBs. In that case, we select the child NeRF AABB whose space area includes the weight (as Eq. 2 calculates) peak value, as the LiDAR ray cannot penetrate opaque objects when it emits, as shown in the first and third subfigure of Fig. 4(b). Furthermore, suppose the peak weight value is not in any child NeRF AABB. In that case, the child NeRF with the maximum weight integration $W$, i.e., the most likely existence range of an object on the LiDAR ray, is selected, as shown in the second subfigure of Fig. 4(b). Moreover, the weight integration $W$ is calculated by Eq. 9. When the LiDAR ray only intersects one single child NeRF AABB, we can directly select it as the object area, as shown in the last two subfigures of Fig. 4(b). Note that the child NeRF weight integration $W$ may be minimal, in which case the LiDAR ray does not intersect any child NeRF and no depth value needs to be inferred. The inferred depth value of each LiDAR ray is calculated by Eq. 10.

Datasets: We evaluate our approach using two publicly available outdoor datasets, including the MaiCity [17] and KITTI odometry datasets [42]. The MaiCity dataset contains 64-beam noise-free synthetic LiDAR scans in virtual urban-like environments. KITTI odometry dataset also contains 64-beam LiDAR data collected by the vehicle in real-world urban environments and provides a localization benchmark. Moreover, we further use semantic labels from SemanticKITTI [43] to filter out movable objects. In Sec. IV-B and Sec. IV-D, we sample one test scan out of every five scans and use the other scans for training, the same as the dataset split in some previous works, such as NeRF-LOAM [4] and NFL [14].

Metrics: We evaluate the novel LiDAR view synthesis and the 3D Reconstruction performance of our method. For the novel LiDAR view synthesis, We compare the reconstructed depth value with the real depth of each LiDAR ray on the test set and report the mean average absolute error (Avg. Error [$\mathrm{m}$]) and mean accuracy at two thresholds ([email protected]$\mathrm{m}$ and Acc@1$\mathrm{m}$ [%]). Moreover, for 3D reconstruction, we report the chamfer distance (CD [$\mathrm{m}$]) and F-score at two thresholds ([email protected]$\mathrm{m}$ and F-score@1$\mathrm{m}$).

Baselines: A standard pipeline for generating new LiDAR views is constructing a 3D point cloud map and then using the ray-casting approach [44] to query new point clouds from the map. We implement this pipeline, which voxelizes the 3D point cloud map into a 3D voxel map to speed up the query but may slightly reduce accuracy, as a baseline method named MapRayCasting. In addition, we also extend the original NeRF model proposed by Mildenhall [21] by replacing a camera ray with a LiDAR ray as IR-MCL [31], named OriginalNeRF.

Training details: We train our proposed PC-NeRF and all the baselines on an NVIDIA GeForce RTX 3090 and use Adam [45] as the training optimizer. The initial learning rate is set to $4\times 10^{-5}$ and is adjusted using Pytorch’s MultiStepLR strategy, where the adjustment milestones are $[5,10,20]$ and the adjustment factor is $0.1$. In Sec. IV-B and Sec. IV-C, $\lambda_{\mathrm{pd}}$, $\lambda_{\mathrm{cf}}$, $\lambda_{\mathrm{cd}}$, $\lambda_{\mathrm{in}}$, and $\gamma$ of our proposed PC-NeRF are set to $1$, $10^{6}$, $10^{5}$, $0.1$, and $2.0\,m$, respectively. This group of parameters is not specifically tuned for a single scene but is valid for all experimental scenes. Except as specifically noted, our proposed PC-NeRF uses only one training epoch to achieve the demonstrated accuracy. Both our PC-NeRF and OriginalNeRF use the hierarchical volume sampling strategy along the LiDAR ray proposed in NeRF [21], where the points number $N_{c}$ and $N_{f}$ for coarse and fine sampling along the ray are 768 and 1536, respectively. However, for coarse sampling, our PC-NeRF uses the Child NeRF segmented sampling proposed in Sec. III-C, while OriginalNeRF samples uniformly along the LiDAR rays.

We evaluate our proposed PC-NeRF’s inference accuracy and deployment potential in different scales qualitatively and quantitatively. We conduct experiments on the MaiCity and KITTI datasets using 50 consecutive scans as a scene, corresponding to the small-scale evaluation since each NeRF model is trained and evaluated only by one scene. As shown in Fig. 5, MapRayCasting and OriginalNeRF can roughly reconstruct the environment while losing many environmental details. Especially when the input sensor data is distributed centrally over large spatial segments such as the ground, OriginalNeRF tends to be trained overfitting to the representation of larger objects and ignoring smaller objects, as shown in rows 2 and 3 of column 6 of Fig. 5. In contrast, our proposed PC-NeRF achieves better results compared to the baselines by optimizing the scene-level, segment-level, and point-level environmental representations concurrently, as shown in the last column of Fig. 5. We also report the quantitative results for small-scale scenes in Tab. I. As shown in Tab. I, our proposed PC-NeRF performs much better than MapRayCasting and OriginalNeRF and achieves a high novel LiDAR view synthesis accuracy (e.g. Avg. Error $<$ 0.50$\,\mathrm{m}$) and 3D reconstruction accuracy (e.g. CD $<$ 0.23$\,\mathrm{m}$). Moreover, our proposed two-step depth inference has much better inference accuracy than the one-step depth inference and is highly stable on our proposed PC-NeRF. Therefore, for the subsequent experiments in Sec. IV-C and Sec. IV-D, we use the one-step depth inference and the two-step depth inference for OriginalNeRF and our proposed PC-NeRF, respectively. Compared to OriginalNeRF, which uses ten epochs of training, our proposed PC-NeRF demonstrates good deployment potential because our proposed PC-NeRF yields better results using only one training and does not change any model parameters for different scenes.

To further explore the deployment potential, we test our proposed PC-NeRF on large-scale scenes. According to Sec. III-A, we divide the KITTI 03 sequence (800 scans, $555\times 120\times 7.2\,{\mathrm{m}}^{3}$) into 32 sequential scenes and each PC-NeRF model is trained and evaluated on one scene. The spatial extent of each scene is about $61.5\times 42.5\times 3\,{\mathrm{m}}^{3}$, overlapping with that of neighbouring scenes. In the 30th scene, our proposed PC-NeRF trains three epochs because one or two epochs of training result in the failure of two-step depth inference. Additionally, our proposed PC-NeRF trains only one epoch on all the remaining 31 scenes. As a comparison, OriginalNeRF trains three epochs on all 32 scenes. From Tab. II and Fig. 6, it can be seen that our proposed PC-NeRF is trained to achieve a high novel LiDAR view synthesis accuracy (e.g., Avg. Error $<$ 0.66$\,\mathrm{m}$) and 3D reconstruction accuracy (e.g., CD $<$ 0.28$\,\mathrm{m}$), further demonstrating its promising deployment potential in large-scale environments.

We evaluate our proposed PC-NeRF’s 3D reconstruction under sensor data loss on the MaiCity 00 and KITTI 00 sequences. In real autonomous vehicle applications, some unfavorable conditions, such as hardware failures, adverse weather factors, and unstable communication in remote control tasks, may result in different degrees of sensor data loss. To simulate this challenge, we control the sensor data loss rate by setting the proportion of training point cloud data and using all the other point cloud data as the test set.

Let us first concentrate on the number of training epochs since the fewer the number of training epochs used to meet the required 3D reconstruction accuracy, the faster our proposed PC-NeRF can be deployed. As seen in Tab. III and Fig. 1, with only one epoch training, our proposed PC-NeRF can reconstruct urban and rural road environments very well until the sensor data loss rate reaches 67$\%$. However, due to too little sensor data input for model training when increasing the sensor data loss rate from 67$\%$, our proposed PC-NeRF that continues to be trained using only one epoch fails to infer using the two-step depth inference. Therefore, we increase the number of training epochs until successful inference is achieved using the two-step depth inference. At the sensor data loss rate of 90$\%$, the lowest number of epochs required for the two-step depth inference to be effective is five for scenes from the MaiCity dataset and ten for scenes from the KITTI dataset. This difference in epoch number is because the scenes in the KITTI and MaiCity datasets are rural and urban road scenes, respectively. Urban road scenes, with more straightforward environmental geometric segments, require fewer training epochs for efficient learning.

Let us then focus on the 3D reconstruction accuracy of our proposed PC-NeRF. As Tab. III shows, using several training epochs that are no more or even much fewer than that of OriginalNeRF, our proposed PC-NeRF achieves much higher accuracy than OriginalNeRF and also higher accuracy than MapRayCasting. Surprisingly, in Tab. III, we can find that as the sensor data loss rate increases, the 3D reconstruction accuracy of our proposed PC-NeRF does not decrease but instead increases or remains relatively stable. This accuracy does not decrease because our proposed PC-NeRF can rapidly learn an approximate scene distribution at the segment level. Moreover, if the input sensor data is too much or too little during training, more training epoch is needed to fit the model. Additionally, the 3D reconstruction accuracy is higher on the MaiCity dataset than on the KITTI dataset because urban roads have fewer environmental geometric segments than rural roads, contributing more to segment-level 3D reconstruction and, in turn, to scene-level and point-level 3D reconstruction.

In summary, our proposed PC-NeRF can effectively address the challenge of sensor data loss. Even when up to 67$\%$ of the sensor data is lost, our proposed PC-NeRF can still achieve a high novel LiDAR view synthesis accuracy (e.g., Avg. Error $<$ 0.44$\,\mathrm{m}$) and 3D reconstruction accuracy (e.g., CD $<$ 0.20$\,\mathrm{m}$) using only one training epoch, presenting a good potential for deployment and reconstruction performance even under partial sensor data loss.

To demonstrate the effectiveness of our method components, we conducted experiments on different components of our proposed PC-NeRF.

Effect of child NeRF free loss: In our proposed PC-NeRF, the child NeRF free loss controlled by $\lambda_{\mathrm{cf}}$ optimizes the scene-level and segment-level environmental representation. As seen in Tab. IV, the child NeRF free loss helps improve 3D reconstruction accuracy within a specific range of variation, but too high or too low reduces the 3D reconstruction accuracy.

Effect of balancing child NeRF free loss and child NeRF depth loss: Based on child NeRF free loss, child NeRF depth loss is used to optimize the point-level and segment-level environmental representations further, making the balance between child NeRF free loss and child NeRF depth loss a critical point. As shown in Tab. V, enlarging the smooth transition interval $\gamma$ between child NeRF free loss and child NeRF depth loss can improve the 3D reconstruction accuracy. At the same time, too large a smooth transition interval will decrease the 3D reconstruction accuracy. This accuracy decrease is because when the smooth transition interval is too large, child NeRF depth loss almost needs to supervise the sampling points on the entire ray and cannot effectively supervise the sampling points around the near and far bounds of child NeRF.

Effect of two-step depth inference: In Tab. I, it can be seen that two-step depth inference outperforms one-step depth inference in many cases, but inference errors also occur for OriginalNeRF. This inference error is because the validity of two-step depth inference is based on the model output weight distribution near the geometric segments’ distribution in the environment, as shown in Fig. 4(b). The extensive experiments in Sec. IV-B, Sec. IV-C, and Sec. IV-D demonstrate that our proposed PC-NeRF training method and two-step depth inference are highly compatible and consistently robust.