ActiveNeRF: Learning Accurate 3D Geometry by Active Pattern Projection

Given a set of input images from several viewing directions of a scene, multi-view stereo aims to reconstruct a 3D representation of the scene. Typically, this is done in four stages: cost computation or feature extraction, cost aggregation or matching features, optimization, and, finally, disparity refinement [19]. In this setup, the intrinsic and extrinsic camera parameters for the images are assumed to be known or can be computed from the images themselves. The traditional multi-view 3D reconstruction pipeline estimates depth maps for each view which can then be fused into representations such as meshes, volumes, or point clouds.

Most methods that achieve state of the art results today are learning-based stereo methods that leverage large-scale benchmarks and higher computational ability to perform supervision [6, 7, 11]. Works such as PointMVSNet and MVSNet [24, 2, 14] introduced end-to-end frameworks, allowing for all four stages of MVS to be done within a network. Recent unsupervised and self-supervised methods have achieved comparable results while also not needing ground truth depth. One such direction is image reconstruction loss, where an image’s material and lighting are recovered along with its geometry [22, 9]. Another such direction is reprojection which projects an image from one viewing angle to another and computes the difference between the actual image and the reprojected one [13, 27, 5].

Implicit neural representations [15] are an alternative way of representing 3D objects that have seen significant advancement recently. NeRF and subsequent works use an MLP to represent a neural radiance field which takes in a 3D coordinate and view direction and outputs the volume density and radiance at that coordinate. Compared to traditional representations, implicit neural representations achieve better results on tasks such as novel view synthesis and are able to generate higher-fidelity renderings in general [12, 16, 26]. However, they struggle to predict geometry as there is no straightforward way to extract the scene surface from the output density. Dex-NeRF attempts to do this on transparent objects by truncating the weight along a ray with a pre-defined threshold and considering that as the surface point with limited success [8]. Recent works such as VolSDF and NeuS [25, 23] aim to address this challenge by redefining the volume density using a signed distance function representation. Geo-NeuS improves on this by integrating sparse geometry and photometric consistency commonly found in multi-view stereo [4]. NeRF2Mesh [21] takes a different approach to this task by decomposing environment light into specular and diffuse components before rendering, simplifying the information that the neural radiance field needs to learn.

Despite their success in predicting geometry, these methods assume environment lighting is static and has low spatial frequency such as a single circular light source projected onto the scene. This type of illumination imposes weak regularization on the implicit geometric information available for image synthesis. In contrast, we use active illumination, a setting where the active light pattern moves with the camera, resulting in the lighting of each image being different. Furthermore, this active light pattern has high spatial frequency, leading to stronger regularization overall.

We assume our environment only contains opaque objects such that the camera can capture the reflected active light in at least one view. And the object’s surface can reflect some of the active light instead of absorbing most of it. We setup the camera and active light projector system in such a way that there is always a constant relative pose between them. This retains image synthesis quality while allowing our method to better understand geometry. We initially follow the classic NeRF setup but propose to synthesize novel view images with active light pattern, $I_{act}$, as well as images without, $I_{env}$.

Our task is to reconstruct geometry using a NeRF-like method in a scene containing active light, where the active light emitter’s pose is fixed relative to the camera. We choose to use active light as opposed to other forms of illumination because the projected active light pattern will deform after hitting the object’s surface. Furthermore, although the active light projector is moved with the camera, the active light pattern is still consistent across views. In this way, we can jointly optimize the active light pattern and the object geometry by rendering novel view images with active light.

Similar to NeRF, we first query the network using the 3D position and viewing direction to get the environment radiance of the camera ray and a rough depth estimate. Then, we use the depth estimate to query the active light projector model for the active light radiance. Simultaneously, we initialize a 2D tensor to represent the active light pattern, $I_{pattern}$, mimicking an active light projector projecting the pattern onto an environment. We synthesize the final radiance by adding the environment radiance and the active light radiance together and then capping the value at 1. The whole process is differentiable, allowing the active light pattern to be updated throughout training. In this way, the model will learn to match the ground truth active light image while also implicitly regressing depth estimation more accurately. Intuitively this makes sense because the rendering process explicitly uses surface points to render active light: if the surface point is not accurate, the resulting rendered image’s active light pattern will be shifted compared to the ground truth image’s active light pattern. Our high-level framework is shown in Fig.1 and detailed description of the pipeline is shown in Fig.2

The volumetric rendering in NeRF assumes that environment light is static, so a point in 3D space and a 2D viewing direction can accurately determine the radiance. Thus, NeRF simplifies the volumetric rendering process to a setting where the objects themselves emit light. However, to render an image with an active light pattern coming from an active light projector moving with the camera, we need to trace the camera ray to the active light projector and compute the active light radiance.

Here, we use a pinhole camera model to represent the active light projector, where $\mathbf{K}_{act}$ is the intrinsic matrix and $\mathbf{[R,t]}$ are the extrinsic parameters. Given a 3D surface point $\mathbf{x}$ in world coordinates, the pixel coordinate $\mathbf{p}$ can be computed by the intrinsic and extrinsic matrix. Therefore, the radiance of a ray can be extracted from the active light pattern image. We then linearly interpolate the active light image to enable sub-pixel radiance computation.

In summary, for each sampled point along the ray, the active light radiance is computed and added to the environment radiance. The resulting radiance will be used in the volumetric rendering equation to compute color:

Surface properties also play an important role in rendering active light images. For example, some surfaces are not lambertian, causing the active light to not reflect back to the camera at some viewing directions. To tackle this problem, we define a surface BRDF represented as an additional neural implicit field. The input to this field is a 9-D vector containing a 3D position vector$\mathbf{x}$, a 3D incident direction vector $\mathbf{d}_{in}$ and a 3D reflected direction $\mathbf{d}_{out}$. The output is a scalar representing the surface BRDF value $s$. In this way, the MLP network with parameter $\Theta_{act}$ can be denoted as $G_{\Theta_{act}}:(\mathbf{x},\mathbf{d}_{in},\mathbf{d}_{out})\rightarrow s$.

The final volumetric rendering equation we use is:

After we get the final radiance, we truncate this radiance so that the intensity is between 0 and 1.

We supervise our method using images with, $C_{final}^{gt}$, and without, $C_{env}^{gt}$, active light pattern:

We find that simultaneously optimizing the model with both losses from the beginning leads to instability. This occurs when the predicted depths from NeRF in the early stage vary substantially across views, causing the gradients from pattern regularization to be ambiguous, resulting in a failure to converge. Therefore, when the coarse depth estimation converges, we start training both modules jointly to further refine the depth.

NeRF focuses on the rendering quality of novel view synthesis but does not explicitly regulate geometry. Although, the geometry can be implicitly estimated by the density from the neural radiance field:

where:

$T_{i}$ corresponds to the depth value of each sampled point along the ray. Ideally, the weight of each sampled point that is not on the surface should be 0 and the color of points on the surface should be the final rendered color on that camera ray. In general, volumetric rendering should start from the object surface towards the observer. Thus, the weight at the surface should be the highest and decrease along the ray. However, in practice, the weight is non-zero along the ray. Usually, the density value increases near the surface and keeps increasing beneath the surface because the area below the surface is also occupied, as shown in Fig.3. Therefore, the surface computed by weighted sum over the samples’ depth values will be below the ground truth object surface.

We propose two ways to tackle this problem. First, following Dex-NeRF  [8], we can get a rough depth estimate by truncating the occupancy along the camera ray with a pre-defined threshold. This method requires tuning the threshold per scene which is not ideal. Second, we can render the depth directly using the point whose weight is the largest along the ray during inference time. In Fig.3, we show that the second option dramatically decrease the biases in depth estimation and does not require manual tuning for different scenes.

The dataset we use is a derivative of the original NeRF dataset. For each scene, we synthesize 100 images from different views. We use Blender [3] to re-render the scene with a simulated active light projector whose position and orientation is constant relative to the camera. The resulting 800x800 image is 1-channel and gray-scale with values ranging from 0 to 1. The intrinsic and extrinsic matrices of the camera and active light projector are known for all the views. For each view, the dataset contains both the image with and without the active light projection.

The environment radiance module, $C_{env}$, is modeled by an 8-layer MLP with 128 hidden size. The BRDF distribution module, $G_{\Theta_{act}}$, is modeled by a 6-layer MLP with 128 hidden size. Following NeRF, we also use positional encoding to improve the high-frequency surface performance in both modules. The position vector uses a 10-frequency positional encoding and the direction vector uses a 4-frequency positional encoding. We downsample the images to half resolution as the input. We train our model for 250k iterations within 6 hours on an NVIDIA RTX 3090. Our model is trained in an end-to-end fashion so the radiance field, BRDF field, and active light pattern are all trained at the same time.

We use chamfer distance between our reconstruction and ground truth to evaluate our model performance. For both reconstruction and ground truth, we downsample the point clouds to a voxel size of 0.003 to ensure resolution consistency. To measure the reconstruction quality in more depth, we compute the percentage of chamfer distance that is lower than 0.01m and 0.05m.

Since our method requires re-rendering the scene with active light, related methods which are evaluated on the original NeRF dataset cannot be easily adapted to our setting. We first compare our method with NeRF2Mesh [21], which uses the same synthetic dataset as NeRF and directly outputs the geometry as a mesh. It is worth noting that NeRF2Mesh uses a 4-channel RGBA image as input where the exact foreground mask can be inferred directly from the alpha channel. Moreover, the meshing procedure removes outlier points. Although it is not a completely fair comparison, as shown in Tab.1, our method is superior to NeRF2Mesh in terms of the average chamfer distance and $<0.05$m inlier percentage. The results justify that our model can more accurately learn geometry by way of active pattern projection.

Furthermore, we also compare our method to OpenMVS  [1]. OpenMVS is a non-learning based algorithm, which reconstructs the geometry by detecting and matching correspondences across different views. As shown in Tab.1, our method also outperforms OpenMVS on most metrics.

In this section, we validate the effectiveness of each component and design choice through ablation experiments.

The baseline between the camera and the active light projector is crucial to the proposed method’s performance. Since we query the active light intensity by triangulating surface points, the depth error’s sensitivity is inversely proportional to the baseline. When the baseline is 0 and the camera and the active light projector are collocated with each other, the active light pattern on the object surface is not affected by geometry and the model cannot regulate the geometry by active light pattern. Therefore, we carry out more experiments in simulation on different baselines between camera and active light projector to verify this assumption.

As illustrated in Table 4, increasing the baseline can further improve our method’s performance. However, the marginal gain diminishes substantially with further increases. This observation aligns with the expected behavior, as once the ambiguity is largely resolved by increasing the baseline, additional increments yield diminishing performance improvement.

As our method leverages active pattern projection to improve geometry reconstruction, it can more effectively utilize the information contained within each view image. To validate the influence of view count on our approach, we evaluate our method on different numbers of views and compare it with NeRF.

We use a $0.4\textrm{m}$ baseline between the active light projector and the camera to achieve optimal active light regulation capacity. As illustrated in Table 5 and Fig. 8, decreasing the number of views degrades the performance of both methods. Nevertheless, the degradation observed in our method is notably slower and less pronounced compared to NeRF. Remarkably, even with only 5 views, our method retains the capability to reconstruct high-fidelity geometry.

We further evaluate the effectiveness of the proposed method on real-world captured images. As shown in Fig. 9, we employ a RealSense D415 to capture images with and without active pattern, such that the relative pose between the camera and projector remains constant during multi-view image capturing. Because the D415’s pattern projector consists of 2 projectors, we cover one of the projectors.

We capture 5 objects at a distance of about 300 mm. For each object, we acquire images with $1920\times 1080$ resolution from 24 views around it. At each viewpoint, images with and without active light are captured with the same exposure time. To obtain the extrinsic parameters for all the views, we place 8 ArUco markers around the object and utilize the marker detector in OpenCV to generate an initial estimation. Subsequently, we refine the extrinsic parameters by minimizing the reprojection loss. To obtain the relative position of the projector and the camera, we track the rays of some feature points in the pattern under the camera coordinate system and fit the intersection point as the projector position. For comparison, we also reconstruct the objects using OpenMVS [1], where we use the images without active pattern. We acquire the groundtruth shapes of objects using a handheld 3D laser scanner (CREAFORM HandySCAN 700), whose accuracy is 0.030 mm. We use chamfer distance between the reconstruction and the ground thruth to evaluate the performance. The results are shown in Table 6 and Fig.10. Compared with OpenMVS, the point clouds reconstructed by our method are more accurate and with better coverage. OpenMVS performs worse in areas with less feature, such as the bottle body and the hair of the statues.