Casual 6-DoF: free-viewpoint panorama
using a handheld 360° camera

6-DoF methods have been attracting much attention in recent years due to the need for motion parallax in VR applications. Thatte et al. [40] introduced stacked omnistereo, which uses two camera rigs stacked on top of each other in order to capture 6-DoF content. Welcome to Light Field [27] used a spinning camera setup for capturing high-density images and providing reliable depth estimation. Facebook Manifold RED [31] is a camera system that has been specifically designed to capture 6-DoF video. More recently, Broxton et al. [6] proposed a system that uses deep-learning to convert videos captured by a spherical camera rig into a layered mesh, which can be viewed in 6-DoF. These state-of-the-art capture methods were restricted to professional cameras and sophisticated setups. The photogrammetry-based methods such as depth image-based rendering (DIBR) [50, 26, 22] only require a set of overlapping photos which are not necessarily from the same camera, thus being more suitable for casually captured videos.

Inside-Out [15] is a photogrammetry-based method that enables wide area, free-viewpoint rendering via tile selection. DeepBlending [14] uses a similar architecture to Inside-Out. One significant improvement of this method is that they use deep learning-based image blending to achieve higher visual fidelity. Recently, Xu et al. [44] further improved the work to have the ability to reconstruct reflection on a reflective surface. However, with the requirement of thousands of input images company with input depth from Kinect, thus, is not ideal for casual users such as tourists. Recent works such as NERF [25, 47], free-view synthesis [32], and stable view synthesis [33] used deep learning to render high fidelity novel views. However, these methods are slow and require significant running time to synthesize each frame.

The methods discussed above were built for perspective cameras, meaning direct application to 360° panoramas would perform poorly. In this paper, we present a method for creating high fidelity 6-DoF scenes with 360° cameras.

Huang et al. [17] presented an approach that used geometry estimated via Structure-from-Motion (SfM) to guide the vertex warping of a 360° video. This technique simulates the feel of a 6-DoF experience. However, the experience is somewhat lacking due to its inability to produce motion parallax. Cho et al [9] extended this method, allowing it to use multiple 360° panoramas as input. However, it still presents the same limitation. The lack of motion parallax, in either case, can lead to VR discomfort.

One of the early 6-DoF panorama applications which provided motion parallax was proposed by Serrano et al. [36]. In their system, they used deep-learning to predict the depth map of a given 360° panorama and render a mesh representation of the environment (inherently allowing for motion parallax). They designed a three-layered representation to handle occlusion. However, their method only took input from a single viewpoint, meaning the output quality and range of motion were limited. MatryODShka [1] adapted Multi-Plane Image (MPI) [24, 49] to 360° panoramas, creating a layered representation of Omnidirectional Stereo (ODS) images with deep learning, however, this approach also has a limitation on its synthesis quality and range of available motion.

In this paper, we present a novel method of performing wide-area free-viewpoint navigation using casually captured 360° panoramas.

Recovering depth information is important to enabling accurate 6-DoF images. Previous depth estimation methods for 360° images has attempted to recover the depth map via matched features [34], this type of method suffers from feature detection errors often producing an incomplete depth map. Most recent works on 360° images designed to perform depth estimation with single [42] or pair of 360° images[51, 43]. These works estimate depth without validating depth consistency across images. The inconsistency across depth images causes severe ghosting artifacts and is not suitable for view synthesis of large scenes where dozens to hundreds of images are often used for reconstruction. To reconstruct consistent depth across multiple images, the depth needs to be estimated with the spirit of Multi-View Stereo (MVS).

Similar to other areas in computer vision, research has attempted to approach MVS using CNNs [45, 46]. However, these methods often perform poorly on wildly captured images. Since their networks are trained by comparing the estimated depth with ground truth depth, it is limited to adapt the newly captured scenes when ground truth is not available. Neural radiance field (NeRF) [25] uses a Multi-Layer perceptron (MLP) to learn the presentation of the scene. Although this method requires training for every given new scene, the advantage is that it can train the network by directly comparing the estimated result with the capture RGB images, thus, more suitable for general scenes than methods trained with ground truth depth. However, we observed NeRF has difficulty in network converging when training with 360° panoramas. We demonstrate the limitation in Sect. 8. Since the learning-based methods were unable to provide acceptable depths from our survey and tests, we developed our depth estimation as in Sect. 5.

We implement the dense depth estimation as follows. Given a 360° panorama, we first generate sweep volumes directly using $N$ closest 360° panoramas (we use $N=4$ in our experiment), given that it has a full FoV that could help to effectively avoid the out-of-FoV issue in narrow FoV videos. We then compare the sweep volume with the given 360° panorama to generate cost volume. We compute the cost volume using the classical ad-census [23] method, because it is able to demonstrate good result on texture-less region. Similar to previous methods [16, 30] we adapt guided image filters to filter the matching cost. Guided image filters smooth the filtered cost volume with regards to the edge of the guiding image, which helps sharpen the edge of the resulting depth images. We then follow the classical winner-takes-all depth selection to produce the dense depth map.

Our dense depth panorama is estimated on a per-view basis, meaning that the estimated depth information is based on the parallax present in neighboring panoramas. Each panorama has different neighbors, meaning the recovered depth panoramas may not be consistent, as presented by the top row of Fig. 4. This will cause visual artifacts during view synthesis. We solve the depth inconsistency, with an iterative depth refinement process. Our depth refinement process consist of three steps, depth fusion, depth correction via raymarching, and forward depth projection.

During the first iteration, we perform depth fusion using the estimated sparse and dense depth panorama. Similar to Hedman et al. [14], our depth fusing allows us to adopt advantages of each depth type while avoiding their limitations. Nonetheless, the fused depth can still contain issues such as depth inconsistency that comes from the depth map estimated using epipolar approach. Thus, it needs further refinement to be used in view synthesis.

Although our raymarching process able to removes the outlier depths that were too close to the camera, it often erased part of object. This is because, due to the limitation of epipolar based depth estimation, not all neighboring views would be correctly estimated for thin objects, if the sparse depth map fails to address, the raymarching process would end up removing the depth information of that object. An example has been shown in the second row of Fig. 4, where the pillar of the pavilion has been partially erased. In such cases, the refined depth panorama will lose the depth information related to the occluding object, causing artifacts where geometric details are missing. We overcome this issue via our following forward depth projection process.

For each iteration of depth refinement, we increase the value of $K_{rm}$ and $K_{fp}$ slightly (we use 2 in our experiment). The slightly increased checking range helps to improve the consistency of the depth over more panoramas, eventually $K_{rm}$ and $K_{fp}$ will be equals to the number of input panoramas. However, in our experiments, we found that around 3 iterations was sufficient for most of our test scenes. We illustrate each step of our depth refinement in Fig. 4, while showing the result of each refinement step. We also illustrated a few synthesized views using the depth maps from each depth refinement step to demonstrate the quality improvement.

Image to 3D coordinates: We first aligned orientations of input panoramas to ensure they facing the same direction. Since we use the equirectangular representation of 360° panoramas as in the prior work [40], we first convert the pixels $(i,j,d)$ of the input equirectangular RGBD images into spherical polar coordinates $(\theta,\phi,d)$, where $\theta\in[-\pi,\pi]$ and $\phi\in[\frac{-\pi}{2},\frac{\pi}{2}]$ and $d$ is the corresponding depth value from input depth panorama $D_{k}$. We then project all the converted spherical coordinates of each input panorama into local 3D Cartesian coordinates $\vec{v}$ by Equation 1.

Forming novel panorama: The next step is to reproject $\vec{v}$ into the center of target novel view panorama in its local coordinates. If we define the center of target novel view panorama as $\mathbf{t_{new}}$, then the transformation between any other input panorama which centered at $\mathbf{t}_{k}$ can be calculated using $\vec{t}=\mathbf{t_{new}}-\mathbf{t_{k}}$. Then all 3D points $\vec{v}$ of input panoramas are reprojected into the target position $\mathbf{t}_{new}$ by $(\vec{v}-\vec{t})$. Then, the reprojected points of each input can subsequently be converted into spherical polar coordinates $(\theta_{rep},\phi_{rep},d_{rep})$ of $\mathbf{t}_{new}$ using Equation 2.

This spherical polar coordinates then converted into pixels of the equirectangular representation of the novel view panorama.

We use Equation 1 and 2 to extract the necessary data from the input panoramas, and synthesize the novel views in two steps: 1) Backwards warping with depth correction, and 2) image blending, described as follows.

Direct projection of the input panoramas into a novel view tends to produce many missing pixels. Simple color interpolation such as using linear or cubic function result in a blending of foreground and background.

To overcome this, we first synthesize a depth panorama at the target novel view, interpolate the depth values to avoid any missing values. In our approach we adapted a morphological closing [21] similar to the one used by thatte et al. [40] in order to achieve the desired result. We use the synthesized depth panorama to extract RGB pixel values from each input panorama using Equation 1 and 2, to synthesis the novel view panorama.

However, there may be disagreement between the input depth panoramas for what the correct depth value is at the target position. A depth value that refers to the correct pixel in one input panorama may point to a different location in another input panorama, leading to visual artifacts in the synthesized panorama. To correct this, we apply Algorithm 1 to the synthesized depth panorama, improving the overall visual consistency. We perform backwards warping with the corrected depth panorama in order to obtain the RGB pixels from the input panoramas with reduced visual artifacts.

To synthesis color values of our novel view panorama, we use the synthesized depth panorama to extract the RGB pixel values from the corresponding input panoramas. We extract the pixel values from the closest $K$ neighboring panoramas, where $K=4$. If a suitable pixel cannot be found from closest neighbors, progress further to find the corresponding pixels across other input panoramas.

After extracting corresponding pixel values, their RGB values are blended by our weighting formula. Our weighting formula is based on three components: the correctness of the estimated depth $w_{d}$, the distance between the position of target view and input panorama $w_{cam}$, and the angle between the viewpoints and the relevant point in the scene $w_{ang}$. Then, the final blending weight $W$ for each pixel is computed by:

The $w_{d}$ is computed by the difference between the estimated depth and the actual depth value as:

where $d_{rep}$ is the estimated depth by reprojection from the position of the novel view depth panorama to the position of input depth panorama, and $d$ is a depth value at the input depth panorama. If the difference between $d$ and $d_{rep}$ is large, it means an occlusion occurs, and the pixel from this input panorama should contribute less towards our output.

The camera weight $w_{cam}$ is used to measure the distance between the center of the novel view panorama and the center of the input panorama defined as:

where $\vec{t}$ is described in Sect. 6.1. The constant $s$ is the scale weight, and we set it as $10$ in our experiment.

Our angle weight $w_{ang}$ is to reproduce view dependent features in the synthesized image by favoring pixels with similar view angles as considered in [7, 4]. The $w_{ang}$ is calculated by the angle between the vector $\vec{t}$ and vector $\vec{v}$ , defined as:

Finally, each pixel value of the novel panorama $I_{n}(p)$ is synthesised by weighted blending of corresponding pixel values from $K$ neighboring input panoramas $I_{k}$ as:

where $q_{k}$ represents the corresponding pixel in the $k$-th input panorama. We illustrate the effect of different weighting formulation in Fig. 7. Our weighted blending successfully synthesis pixel values of the novel view panorama while reducing blending errors such as ghosting artifacts.

Fig. 6 shows motion parallax of the synthesis view in various directions. Fig. 12 shows examples of synthesised panoramas, which provide novel panoramic views away from the captured path.

Comparisons of the view synthesis of the indoor scenes are in Fig. 8. We noticed that learning-based methods such as DeepBlending efficiently reduce visual artifacts caused by the geometry estimation error, but it often blurs the scene. Both our method and Inside-Out preserve the sharp appearance of the scene, but, compared with Inside-Out, our method shows fewer visual artifacts on boundary regions.

Comparisons of view synthesis of the outdoor scenes are in Fig. 9, where our method outperforms the prior works. The geometry reconstruction of a large outdoor scene is very challenging. DeepBlending and Inside-Out are highly dependant on the mesh quality by the surface reconstruction. The sparse depths produced by COLMAP tend to contain large missing regions, and therefore may cause surface reconstruction errors. Furthermore, their depth refinement does not consider depth consistency across scenes, and therefore often produces different errors across input panoramas. When blending multiple inputs for novel view synthesis, errors in different scenes are accumulated, producing visual artifacts.

Our depth estimation and refinement method generates reliable and consistent depth information for synthesizing panoramas in different viewpoints. Our method recovers detailed geometry with reduced errors and improves results by fusing sparse depths. However, one side effect is that it may remove some of the finer details, causing ghosting artifacts in some cases. Even with this side effect, our method consistently produces higher quality results than prior methods in all our test scenes.

We have tested our method with scene data used in Omniphoto [4], and the results are shown in Fig. 11. Similar to MegaParallax [3] Omniphoto relies on optical flow to guide image warping. Their estimated optical flow describes the movement of pixels from one image to another, and thus, is only valid within the capture circle. When a user moves outside of the capture circle, optical flow will no longer be able to provide adequate guidance for image warping, and the method has to rely on the estimated geometry proxy. However, their geometry proxy only provides rough geometry information. Using it to perform image warping produces significant visual artifacts, such as twisted objects. Compared to Omniphoto, our method and DeepBlending both can synthesize reasonable results outside the captured regions. In our test, our method shows better visual quality than deep-blending.

In our test, NeRF struggles to generate novel views using our dataset. We hypothesize that our test scene attempts to cover a very large area with 360° panorama images, the scale of the scene exceeds the capability of NeRF’s MLP network, which leads to poor results. We further compared our method with NeRF, and the result is shown in Fig. 10.

Previous methods [15, 14] were designed with dense inputs, and therefore we have tested with twice as short intervals. However, their results were not improved in our experiment. We hypothesize that the distortion of 360° panorama produced by the current 360° camera model is bad for convention reconstruction methods, increasing the sample frames introduce more noise features, which do not help improve the quality of the reconstructed mesh. Similar to previous methods, the number of images also impacts the quality of our method. We demonstrate how our method performs the other methods with the different number of input images in Table I. The increase in the number of input images would improve the result, but would also require more time for it to be processed. Nonetheless as shown in the table, for our method an input of 20 images from a 14 second capture video sequence is sufficient for an acceptable reconstruction result.

We quantitatively evaluated our method by comparing it with three closely-related methods [7, 15, 14]. The evaluation method we use follows the approach used by Waechter et al. [41]. The idea is to render novel views images at the same viewpoint as each of the input images, then use the input images as the ground truth to compare with in terms of completeness and visual quality [13].

We tested each method on six scenes with three indoor (PathWay, Hallway, Office) and three outdoor scenes (Bridge, Botanic Garden, Car Park). We did not evaluate the top/bottom view since viewers tend to fixate on the equator when exploring the panorama [4]. We measure the visual quality of the synthesized view at each location quantitatively by comparing the synthesized view with the ground truth image in several metrics such as Multiscale Structural Similarity Index (MS-SSIM), peak signal-to-noise ratio (PSNR), and the perceptual similarity (LPIPS). The average scores of our tests are in Table II. As shown in the table, our method outperforms others in all metrics in both indoor and outdoor scenes.

We also performed a set of ablation studies and show the result in Table II. It shows that synthesis results with only sparse depth estimation are the worst, only dense depth estimation improves the results, and a combination of the two shows a better result than each method. Also, the final depth refinement step shows a clear improvement.

Our results were tested on a desktop PC with an Intel Xeon W-2133 CPU at 3.60GHz, 16GB of system RAM, and an Nvidia RTX 2080Ti GPU. The performance of novel panorama synthesis varies with the number of input panoramas. Throughout the experiment, we have used a set of the closest four input panoramas to synthesize a novel panorama. The data used for the view synthesis method is a set of 4K ($4094\times 2048$) equirectangular images and their corresponding 2K ($2048\times 1024$) depth panoramas. We are able to synthesize the panoramas at 30fps while rendering to the Oculus Rift headset. This frame rate increases to around 90fps when synthesizing a novel panorama at a resolution of $2048\times 1024$.

The time it takes for our captured panorama collection to process depends on the number of images, as shown in Table I. We report the time and the amount of storage it requires to process a set of 20 images using our method and prior methods in Table III.

The storage space needed for the final, processed data-set, of 20 source views, is about 47MB. Depth estimation requires a significant space for COLMAP to store their processing data, for instance, the 20 inputs scene required about 3GB of storage. However, once the sparse geometry reconstruction is completed, only the camera information and reconstructed sparse point cloud (sparse depths) are kept, and the rest of data can be discarded.