Nerfels: Renderable Neural Codes for Improved Camera Pose Estimation

The choice of map representation used in Visual Simultaneous Localization and Mapping (Visual SLAM), Structure-from-Motion (SfM) and Visual Localization systems is paramount as it effects the accuracy and power consumption of the system [7]. Sparse feature-based approaches [35, 3, 32] detect feature points in images which are matched and triangulated across multiple images to form 3D maps. These sparse representations are lightweight and can be effectively used in low power systems like robotics and augmented reality. At the heart of sparse representations is a geometric (also known as indirect) reprojection error, used to optimise camera poses.

In contrast, direct methods work with the raw pixel information[34]. Dense-direct methods exploit all the information in the image, even from areas where gradients are small; thus, they can outperform feature-based methods in scenes with poor texture, defocus, and motion blur [7]. These methods use a photometric alignment objective function similar that used in the well-known Lucas Kanade optical flow algorithm [29]. Though dense-direct methods can add robustness where sparse feature-based methods struggle, they require good initial solution. This limits their performance in wide-baseline localization where a good initialization is difficult to obtain. Additionally, representing the scene densely results in a larger memory footprint of the map, which can lead to higher power consumption for embedded localization systems[33]. Thus deciding on the density of the map representation for SLAM and SfM systems can result in trading off various characteristics of accuracy versus power consumption.

We focus our work at the intersection of sparse feature based approaches and dense image alignment approaches by representing the scene in a globally sparse but locally dense manner. We aim to reap the benefits of a lightweight, sparse model that doesn’t require good pose initialisation, whilst using the power of generative models that leverage image measurements to constrain the pose estimation. Our approach is inspired by surface elements, also known as Surfels [38], which are point primitives that model scene attributes without explicit connectivity between elements. Surfels are used to efficiently render complex geometric shapes by modelling the scene locally as a small circular plane, and have been used in SLAM systems such as ElasticFusion [51]. In such works, the scene is modelled densely by many Surfels: each individual Surfel models a very small portion of the scene (i.e. one Surfel per pixel observed). An interesting approach might be to use a sparse set of Surfels with a larger extent, and integrate them into pose estimation. However, as the baseline between cameras increases, the local planarity assumption weakens for non-planar scenes, and a simple planar representation does not accurately explain the 3D structure (demonstrated experimentally in 6.3).

Alternatively, neural rendering offers the promise of novel view rendering for arbitrary 3D scenes. Methods such as NeRF [31] can generate high quality, high resolution images for a limited scale of scenes given enough compute and training views. Code conditioned neural rendering presents an attractive way to train on different scenes and learn generic representations that can work on related but unseen 3D data. Related approaches demonstrate the use of effective code-conditioning to learn re-usable priors of the geometry and appearance [44, 41]. Follow up work has shown that one can additionally invert such neural rendering approaches to optimise over the camera pose. However, rendering the scene densely for camera pose estimation can be computationally expensive. One such approach, iNeRF [27], shows that guiding the rendering to occur near local features can significantly improve run-time. This begs the question of whether it is necessary to keep a dense representation of the scene for pose estimation with neural renderers, or to use a more sparse representation with a lower memory footprint.

To this end, we propose Nerfels, a hybrid sparse and dense 3D representation designed for pose estimation that is inspired by neural rendering approaches like NeRF, and locally dense point primitives such as Surfels. Nerfels use a code conditioned neural rendering network to represent the local 3D sphere around a sparse 3D keypoint, resulting in a locally dense yet globally sparse 3D scene representation. The use of code conditioning allows each Nerfel to share memory and 3D priors though a single neural network. Each Nerfel has a local pose in the global coordinate frame, which can be explicitly optimized to result in the best rendering across different views. Nerfels are capable of rendering RGB patches around each 3D keypoint into arbitrary views of the 3D scene, which can be used as an additional constraint to a standard sparse 3D point cloud SLAM or SfM pipeline. We believe that the Nerfels representation is an effective compromise between sparse and dense representations for embedded SLAM and SfM applications that require high accuracy pose estimation with a small memory footprint for their 3D representation.

Contributions. Our contributions are twofold: (1) We present Nerfels, a novel representation of local shape and appearance of 3D sparse maps that enhances keypoints to be locally renderable; (2) An end-to-end camera pose estimation system using Nerfels through joint optimisation of reprojection error + photometric error, resulting in improvement of wide baseline pose estimation for both hand-crafted and learned local features.

Sparse SLAM and SfM There is a large body of work that tackle camera pose by simultaneously estimating a sparse scene structure from keypoints detected in 2D images. Well-known SfM and SLAM works [23, 9, 1, 32] rely on triangulated 3D interest points to build a sparse structure of the scene using handcrafted local features [16, 28]. Neural networks have been used to learn local features [10, 11, 42] that are more robust to challenging scene changes such as lighting and viewpoint change. Our work is heavily based on such pose estimation approaches. In challenging pose estimation scenarios however, the accuracy of such approaches can degrade when few matches or inliers exist between the map and the image to be localized. We argue that by leveraging more local image information around the matched keypoints, we can improve pose accuracy.

Semi-Dense and Dense SLAM Early image registration work [29] used image gradients to align multiple images. More recently, SLAM approaches use direct image alignment [34, 14, 13] to jointly estimate a dense or semi-dense scene representation in a real-time pose estimation framework. Such approaches [51, 19] incrementally build a map from multiple observations that is able to render physically predictive representations of the scene. More recently, dense neural pose estimation approaches learn a lower dimensional code [6], basis function [47], or cost volumes [53] for the depth of each keyframe and optimise the depth jointly with the camera poses.

Neural Scene Representation Structure-less approaches [21, 5, 49] use a neural network to directly estimate camera poses without modelling a map. The localization of cameras is performed through a directly learned mapping from images to poses. More recent works [18, 4] similarly do not model 3D geometry but instead model a map through neural embeddings. Recent advances in implicit representation learning have demonstrated the power of coordinate-based multilayer perceptrons (MLPs) to map known world coordinates to signed distance fields [36, 20], occupancy grids [30] or RGB values [31, 48]. Such systems are capable of modelling physically predictive representations of 3D scenes. In one such work, NeRF, the pose of each camera is computed offline using COLMAP [43]. Follow up works [27, 26, 50] relax this constraint by optimising both the implicit function scene structure and the camera pose. These works represent the full scene with a single implicit function. The work of iMAP [45] builds a real-time RGB-D slam system using an implicit neural renderer. We extend this work by applying a similar invertible NeRF framework to multiple smaller NeRF fields simultaneously. Rather than train independent weights for each NeRF field, we use a code conditioning mechanism in an auto-decoder framework, similar to works such as DeepSDF and others [36, 45, 44, 41]. This helps to minimise the memory cost of the neural scene representation.

We define a Nerfel as latent code $c\in\mathbb{R}^{d_{c}}$, to be a compact representation of a continuous radiance field contained in a 3D sphere with radius $r_{s}$. In Figure 2, we show an example of $4$ Nerfels mind from a synthetic scene. Obtaining a projection of an individual Nerfels’ radiance field to an image canvas, $\bar{I}\in\mathbb{R}^{h\times w\times 3}$, is obtained using a mapping function $\mathcal{D}_{NR}$ which we call neural rendering decoder and a given pose $P_{N}\in\mathbb{SE}(3)$ that the sphere is viewed from: $\bar{I}=\mathcal{D}_{NR}(P_{N},c)$. In addition, we define $M_{R}$ to be the set of image coordinates where the integrated rays over image $\bar{I}$ have an alpha $\alpha>0$. The orientation of a Nerfel is defined by its own canonical coordinate system and isn’t bound to any particular global coordinate frame. This coordinate system is chosen so the Nerfels’ center of mass is located at the origin, and the orientation is derived by taking the Nerfels’ average normal vector and arbitrarily choosing an $up=[0,1,0]^{T}$ vector. Given single 3D point at the Nerfels’ origin, the Nerfels’ derived coordinate system, the camera poses in which the Nerfel is observed and the camera intrinsics, we can compute a set of canonical poses for a single Nerfel.

For illustration, the shelf looking Nerfel (top left Nerfel) in Figure 2 was extracted from a specific shelf of a synthetic frame after aligning the Nerfels’ pose to the shelf in the image. However, by changing that Nerfels’ pose it can also align to any similar looking shelf which share geometry and appearance. And thus, a Nerfel can act as a reusable component for describing parts of a scene; whilst one can think of a scene being sparsely described by a collection of Nerfels with a specific pose per Nerfel which aligns each one to that scene.

For the case of rendering a single Nerfel code, the decoder $\mathcal{D}_{NR}$ follows the rendering procedure outlined in  [31]. In this procedure, a neural-network implicit function parameterised by $\Theta$ maps the viewing pose $P_{N}$ together with a 3D coordinate location $(x,y,z)$ to an RGB value $\hat{r}\in\mathbb{R}^{3}$ and density $\sigma\in\mathbb{R}^{+}$ and is followed by a differentiable volumetric function that outputs the integrated RGB value along the viewing ray from the image canvas. In addition to the viewing direction and 3D location, our decoder also takes a code $c$ which is concatenated to the location positional encoding $\mathcal{L}_{x}$, where the viewing positional encodings are defined by $\mathcal{L}_{\phi}$ [31, 46]. To ease notation we describe the process of rendering a single Nerfel code in a given pose by $\mathcal{D}_{NR}(P_{N},c)$. We also define the operation of rendering multiple Nerfels as: $\bar{I}=\mathcal{D}_{NR}(\textbf{P}_{N},\textbf{c})$ where each Nerfel $c_{i}\in\textbf{c}$ rendering provides an image coordinate set $M_{R,i}$; and the union of these sets is: $\boldsymbol{M}_{R}=\bigcup M_{R,i}$. To resolve overlapping Nerfels when obtaining the RGB value in $I$, one can compute the average RGB value of overlapping Nerfels, take a weighted average of Nerfels’ RGB and alpha projections or as done in this work arbitrarily select some Nerfels to proceed others (details in Section 6).

We detail the process of mining Nerfel examples from a collection of scenes and how these are used to train a neural rendering decoder and a set of Nerfel codes. Specifically, we denote a set $N_{c}$ of Nerfel codes as $\mathcal{C}\in\mathbb{R}^{N_{c}\times d_{c}}$ and the decoder as $\mathcal{D}_{NR}(P_{N},c)$. We note two important requirements on our set of codes and the coupled decoder. The first, every code $c$ should encode the associated 3D Nerfel shape, meaning the decoder should be able to render a Nerfel for any given pose, or conversely recover the code (Section 5.1) from any provided realisation of the code in an image. The latter, the given set of codes $\mathcal{C}$ should encompass a broad variety of general 3D shapes that are detected with high probability by a given keypoint detector.

Nerfel Examples For mining a set of Nerfel examples we assume to have access to a collection of different scenes, where each scene has a set of RGB-D frames, $I_{D}\in\mathbb{R}^{h\times w\times 4}$, along with their ground-truth poses $P_{gt}\in\mathbb{SE}(3)$. In addition, we assume to have the intrinsic matrix $K\in\mathbb{R}^{3\times 3}$ which is unique for every scene. The process of collecting a set of Nerfel codes from a given scene is performed by first extracting keypoints in every image within the scene and running a process of retaining the keypoints that can be matched and triangulated between the frames. An example of a method that can extract such matches over a set of frames is COLMAP[43]. These matches can also be obtained by using the ground-truth poses and depth, as done in this work. The result of this process is a set of 3D keypoints $\mathcal{K}\in\mathbb{R}^{N_{kp}\times 3}$ with an additional dictionary of the indices of each keypoint’s frame. This representation allows us to sub-select a number of keypoints which appear in more than $t_{f}$ frames and are viewed from a wide variety of angles (selection of $t_{f}$ is discussed in Section 6). By knowing the 3D location of the remaining keypoints, we crop 2D regions around the keypoint within its observed frames and obtain a collection of patches for each Nerfel in a sequence. For each mined Nerfel, we store a set of image crops $I_{s}\in\mathbb{R}^{h_{c}\times w_{c}\times 3}$, where each crop has fixed dimensions $h_{c}\times w_{c}$. For each Nerfel with a set of image crops, poses $\mathcal{P}_{N}\in\mathbb{SE}(3)$ are also extracted. Here, poses $\mathcal{P}_{N}$ are set to be in a canonical coordinate frame so that a Nerfel becomes scene-agnostic.

Neural Rendering Decoder For constructing the neural rendering decoder we make use of neural implicit functions and a volumetric rendering technique discussed in Section 3. The implicit function is implemented by a MLP and is parameterised by $\Theta$ while the Nerfel codes are initialised as $c_{i}\sim\mathcal{N}(0,\frac{1}{d_{c}})$. The implicit function parameters and the Nerfel codes are optimised as follows:

where $\mathcal{L}_{D}$ is the NeRF[31] objective function modified with a code-condition mechanism described in Section 3. The RGB observations that are used in the optimisation procedure are the mined Nerfel crops $\mathcal{I}_{s}$ together with the poses $\mathcal{P}_{N}$. We note that in the optimisation of Equation 1 a coarse to fine strategy is used; we refer the reader to [31] for further details. While we follow the MLP architecture from [31], the input to the implicit function is of dimension $\mathcal{L}_{x}+d_{c}$ where positional encodings and the Nerfel code are concatenated before being fed to the implicit function. We note in Equation 1 the joint optimisation of the implicit function parameters $\Theta$ and the set of codes $\mathcal{C}$. By allowing the Nerfel codes to be optimised jointly, similar Nerfels collected from the objects with similarities or partial spatial overlap are pulled together (see Figure 3). In the procedure of mining Nerfels, by adjusting the threshold $t_{f}$, we can control the minimal number of image patches per Nerfel.

The following sections detail how Nerfels are leveraged for camera pose estimation. Figure 4 provides a high-level diagram of our Nerfel-based camera estimation pipeline.

We evaluate our proposed method on a synthetic dataset using a simplified version of the method. This is followed by a real world dataset Scannet[8] evaluation with the full version of the method. When optimising the decoder, the available training data is RGB-D with ground truth poses. When evaluating, we use only RGB data with sparse depth for the reference image keypoints. For details regarding the network architecture used for the neural renderer decoder, training and evaluation settings refer to the supplementary material.

Hyperparameters For recovering the Nerfel codes when computing Equation 2 per Nerfel code, we sample $16$ pose initialisations on a sphere to maximise the probability of detecting the correct Nerfel pose $P_{N}$. For each reference image, we limit the number of Nerfel codes to $6\ll N_{m}$, due to the expensive nature of storing the decoder gradients for each Nerfel during the joint optimisation performed in Equation 4. For sub-selecting Nerfel codes, we use a similar strategy as done in [32] where we take a grid of $8\times 6$, select the highest scoring keypoint in a grid cell and finally take the top scoring keypoints according to the amount of Nerfel limit. This procedure ensures Nerfels are well spaced out and land on a surface with a high chance of good fidelity when rendering, while avoiding the case where a Nerfel might occlude another Nerfel. For the real-world dataset, we set $\lambda_{Photo}=1000$ and $\lambda_{Photo}=10000$ for the synthetic one.

Runtime We consider our method to be model-based approach meaning we assume the Nerfel pose recovery discussed in Section 5.1 can be performed offline and we report the run-time of performing the joint optimisation (Section 5.2) to be $0.137\frac{iters}{sec}$ (which is directly affected by the number of Nerfels used during the optimisation procedure). Note that due to the least square objective function, run-time can be further optimized using a second order solver such as Levenberg-Marquardt using packages such as Ceres Solver [2]. Additionally, because rendering occupies the majority of the compute time for our method, additional NeRF speed-up approaches such as [40, 15, 17] should significantly reduce run-time. We leave additional run-time optimisation for future work.

Error Metrics The errors we inspect are the translation error in metric scale and rotation error in degrees. For both of these error metrics, we examine the error in different cut-off thresholds, specifically: 0.25m, 0.5m and 1m of translation error.

Our optimisation procedure on average runs at: $0.137\frac{iters}{sec}$. While providing an improvement over iNeRF, our method does not run in real-time when using a standard SGD optimiser (further discussed in Section 6). When optimising Equation 4 we use a photometric loss which is sensitive to illumination changes. We leave improvements on this matter to future work.

In this paper, we presented a framework for combining traditional geometric feature matching for pose estimation with local photometric alignment. The photometric alignment is performed using Nerfels, which use an underlying code-conditioned neural rendering mechanism. We verified experimentally that the additional constraints from the local photometric alignment improve pose estimation, especially in wide baseline scenarios. One key characteristic of this formulation is that by using a code-conditioned NeRF, we maintain the advantage of low-memory footprint maps, which is critical for applications like AR and Robotics, while gaining the advantages of a partially generative model. Additionally, relative to full scene neural rendering approaches, we reduce the required expressively of the neural renderer to local parts of the scene, enabling improved generalisation of the approach to unseen parts of the scene.