Just Flip: Flipped Observation Generation and Optimization for Neural Radiance Fields to Cover Unobserved View

Simultaneous Localization and Mapping (SLAM) is one of the key methods both in the fields of robotics and computer vision. SLAM is the process of generating a 3D map of an unknown environment while simultaneously estimating the position and movement of a robot or camera. This technique is being actively researched due to its applications in various real-world scenarios such as autonomous vehicles, unmanned robots, etc [1, 2, 3, 4]. Recently, in computer graphics, a novel technology called Implicit Neural Representation (INR) has emerged that utilizes neural networks to parameterize continuous and differentiable signals. As one of the INR, Neural Radiance Field (NeRF) [5] is a deep learning-based approach for 3D scene representation. In this approach, network learns 3D space by projecting a set of 2D views of a scene into a continuous 3D space, such that the color and density information from each view, thus new views of the scene is interpolated for 2D rendering. This technique can be applied to 3D mapping for SLAM as it can be exploited as a 3D scene representation. It also has several advantages over existing SLAM systems in that it can represent space continuously and is more memory-efficient as it utilizes neural networks to represent space. Due to these features, many recent works [6, 7, 8, 9, 10] have applied NeRF to SLAM and 3D mapping.

However, despite many advantages of NeRF, similar to the previous 3D perception methods it also has limitations in synthesizing unobserved views; from here, we define an unobserved view as a view that the robot has not explored or cannot explore. We also define the unobserved view problem as the task of estimating the synthesized view of an unexplored area. For example, in Fig. 1, the robot explores the orange camera trajectory while observing the lego truck, and the green cameras and trajectory represent unobserved views and the unexplored trajectory respectively. The view from the green cameras is unobserved views because they have not been explored yet. These unobserved views cannot be obtained from the observation trajectory due to the 2.5D observation of the robot and the self-occlusions of the objects. Since it is challenging to obtain observations of the unexplored region, various approaches based on NeRF show poor rendering performance on the unexplored scene. Previous research such as [11] tried to overcome the occlusion problem by capturing visibility patterns of 3D voxel exemplars, but they are learning-based methods which require a massive training dataset. The robot needs to be adaptive in environments where it encounters various objects, so NeRF achieves some advantages compared to the method based on the object dataset [11] while it does not require pretraining in such challenging environments. However, as mentioned above, the robot is hard to achieve unobserved views while navigating and thus obtains limited input image data. Therefore, training NeRF with a few images becomes important and various data augmentation methods such as [12, 13] have been studied. Although these methods use geometric approaches to acquire views between or near explored views, they still have limitations in that they do not consider any information about unexplored areas, resulting in the unobserved view problem. Moreover, since they require model training for data augmentation, they may not be suitable for robotics applications that require real-time processing.

To address the unobserved view problem in NeRF, we propose a new unobserved view generation and optimization method. We assume that artificially created objects in indoor and outdoor environments mostly exhibit symmetrical 3D shapes [14, 15, 16]. Typical representative datasets also exhibit symmetric characteristics of 3D objects. Under this assumption, by flipping object observation we can generate additional significant information of unobserved views. Since training NeRF requires both the images and camera poses, estimating the camera position for the flipped observation image is also imperative. Fortunately, some works [17, 18, 19, 20, 21, 22] that try to train NeRF without camera poses have been introduced. However, these methods cannot guarantee finding the global minimum of the camera poses where the camera pose is completely randomly initialized without additional model. To relax this problem of our method, we propose a method to estimate the 6DOF camera pose of the flipped image using the camera pose of the input image. But, this flip and estimation method is not effective for complex objects. To adapt to objects of various shapes, we refine the flipped camera pose and employ a Bayesian approach that incorporates the uncertainty of the flipped image. Therefore we show that it is possible to exploit the symmetric parts of the given 3D object and ignore the unnecessary part according to the estimated uncertainty, thereby we have the robust unobserved view prediction approach.

In summary, we propose a flipped observation generation and training method by considering robot trajectory to improve understanding of unobserved views. Our contribution is as follows:

• •

  We introduce the flip method for predicting unobserved views and propose an approach for the initial 6DOF pose estimation of the flipped images.

• •

  Recognizing that the initialized flipped camera pose can be noisy depending on the object’s shape, we propose a learning approach that performs bundle adjustment on the flipped pose. This method simultaneously optimizes the camera pose and network parameters.

• •

  Flipped images also inherently have uncertainties depending on the object. To address this, we use Bayesian approach that integrates these uncertainties into the model’s loss during the learning process.
  

In the traditional SLAM techniques, 3D representation was achieved using methods such as mesh [23, 24], point clouds [25, 26], and depth maps [27, 28]. However, these approaches have the drawback of having memory limitations since they require storing discrete information, resulting in the map being represented in a sparse manner. To address these limitations, recent research [6, 7, 8, 9, 10] has applied NeRF to SLAM for 3D mapping. NeRF defined as a continuous function by a neural network that takes in a spatial location and viewing direction as inputs and outputs the RGB values and volume density. By continuous neural network function, their approach has the advantages of high resolution and low memory. NeRF has shown remarkable performance in synthesizing photorealistic novel views of real-world scenes or objects. Afterward, NeRF have been extended to achieve fast training [29, 30, 31], few input data [32, 33], large scale scene [34, 35, 36], dynamic scene capture [37, 38], scene editing [39, 40].

In neural radiance fields, we attempt to effectively predict unobserved views in a few-shot setting for robots by estimating the camera pose and uncertainty associated with flipped images. Since the flipped image does not have a camera pose, NeRF needs to be trained without camera pose or simultaneously estimate the camera pose and neural radiance fields. This challenge has been tackled in some works. [17] implemented pose estimation by inverting a neural radiance field, while [18] adopted a two-stage strategy optimizing both camera parameters and the radiance field. With the use of the SIREN layer, [21] introduced sampling methods to counter the joint optimization in NeRF. However, they can only optimize camera pose for relatively short camera trajectories. [19] can reconstruct neural radiance fields and estimate camera poses using generative model. [20] applies bundle adjustment, and [22] utilizes monocular depth estimation, have also been explored. Amidst this backdrop and given our approach to flipped images, the task of addressing inherent uncertainty becomes crucial. There are several prior studies that have incorporated uncertainty into NeRF. Specifically, [41] introduced the concept of uncertainty to render complex scenes from unstructured images. In this approach, they derive uncertainty from transient networks and consider it in the pixel levels of the images. [42] produce uncertainty estimates by modeling a distribution over all the possible radiance fields modeling the scene. They represent the distribution of all potential radiance fields for scene modeling using variational inference. [43] by the same authors address the limitations of [42] by integrating a conditional normalizing flow with latent variable modeling. Based on these methods, we introduce a flip data generation and training method within NeRF to tackle the unobserved view problem.

NeRF is a deep learning-based approach for 3D scene reconstruction from 2D images. It aims to model a 3D scene as a continuous function that maps a 3D point in space to its color and intensity in the image plane. This function is represented as a neural network, which is trained on a set of input images $\mathcal{I}=\{I_{1},I_{2},...,I_{N}\}$ of a scene, with their associated camera parameters $\Pi=\{\pi_{1},\pi_{2},...,\pi_{N}\}$. NeRF model is to learn a mapping of image radiance $\mathbf{c}=(r,g,b)$ and spatial intensity $\sigma$ for 3D coordinates $\mathcal{X}=\{\mathbf{x}\}$ at viewing direction $\mathbf{d}=(\theta,\phi)$. This mapping is represented by a neural network and the mapping function can be represented mathematically as $F_{\Theta}:(\mathbf{x},\mathbf{d})\rightarrow(\mathbf{c},\sigma)$, where $\Theta$ is network parameters.

In order to obtain the rendered image $\hat{I_{i}}$, the color at each pixel $\mathbf{p}=(u,v)$ is estimated by a rendering function $\mathcal{R}$. The expected image color $\hat{I_{i}}(\mathbf{p})$ of camera ray $\mathbf{r}(t)=\mathbf{o}+t\mathbf{d}$ that starts from camera origin $\mathbf{o}$ with near and far bound $t_{n}$ and $t_{f}$ can be written as:

	$\displaystyle\hat{I_{i}}(\mathbf{p})$	$\displaystyle=\mathcal{R}(\mathbf{p},\pi_{i}|\Theta)$
		$\displaystyle=\int_{t_{n}}^{t_{f}}T(t)\sigma(\mathbf{r}(t))\mathbf{c}(\mathbf{r}(t),\mathbf{d})dt\text{,}$		(1)

where

$\displaystyle T(t)=exp(-\int_{t_{n}}^{t}\sigma(r(s))ds)\text{.}$

(2)

$T(t)$ denotes the accumulated transmittance along the ray from $t_{n}$ to t, i.e., the probability that the ray travels from $t_{n}$ to $t$ without hitting any other particle. Therefore, the network can be trained by minimizing the difference between the predicted (or rendered) image $\hat{I}$ and the ground truth input image $I$. This difference can be defined as photometric loss $L_{p}$:

$\displaystyle L_{p}(\Theta,\Pi)=\sum_{i}^{N}\|I_{i}-\hat{I_{i}}\|_{2}^{2}\text{.}$

(3)

In this paper, we introduced a flipped observation generation method for NeRF to predict unobserved views. Most of the robot exploration is performed in real-time, thus it is challenging to observe all the possible viewpoints. To predict unobserved views, we propose a method that exploits the observed views by image flipping. Given that flipped images do not have camera poses, we also proposed a method for estimating the 6DOF pose of the flipped images. Furthermore, we applied bundle adjustment to optimize these estimated poses in conjunction with the model’s parameters. Since only the observed parts should be exploited in order to have robust unobserved view prediction, for the flipped images, we incorporated Bayesian approach by considering the uncertainty estimation. From experimental results we show that our approach markedly improves performance in predicting scenes from unobserved viewpoints and thus shows competitive results compared to the traditional approaches. Our flipped observation method is suitable for robotic applications such as robot exploration and navigation where real-time is important since our approach is simple but robust.