MMPI: a Flexible Radiance Field Representation by Multiple Multi-plane Images Blending

The representations for density and color features are essentially similar to the voxel grid mentioned previously. However, for obtaining density and feature by coordinates, interpolation is only carried out along the $x$ and $y$ axes, and not along the $z$-axis (when sampling points for each ray, only points located at the depth of the image planes are selected, and no other points are included).

Multi-Plane Images (MPI) is an efficient 3D scene representation format consisting of $L$ image planes, each located at a fixed depth $d_{i}$. Because of the additional depth constraint provided, MPI performs better for forward-facing scenes shot in nearly the same direction, and converges more easily than those data structures without a fixed depth, which allows a larger depth range to be modeled. However, the MPI data structure also has many inherent defects. First, MPI can only cover one direction of the scene; moreover, the depth of the image plane selected by MPI is discrete, but the depth of the scene, in reality, is continuous. When using MPI to model scenes with greater depth variation, no matter how the image planes are arranged, there will be a region with excessive depth gap, and the depth of the scene in this region cannot be correctly predicted.

Our MMPI method is based on a key observation that changing the MPI’s orientation can result in better modeling of a specific region of the scene. For instance, in the scene of KITTI dataset as shown in Fig. 2, when we use the MPI facing the front of the scene, we find that the cars on the right side are blurrier, while when we use the MPI facing the right side of the scene, the image quality of the cars on the right side becomes significantly better. This is because the depth change of the cars in the front and back direction is too big, and the cars fall into the sparse area of planes in the MPI facing the front of the scene; while the depth change in the left and right direction is smaller and many objects are closer, the cars fall into the dense area of planes in the MPI facing the right side of the scene. Similarly, small objects such as roadside poles and tree trunks are more easily modeled by MPI facing left or right sides; the surfaces of some buildings need to be predicted with continuously changing depths when using MPI facing the front of the scene, but only a constant depth needs to be predicted by MPI facing left or right sides, thus improving the quality of scene modeling.

Based on the above observation, a reasonable approach to improve the modeling quality of a scene is to leverage multiple MPIs facing different directions and render them jointly, which allows for the combination of their individual advantages in a cohesive manner. Thus we propose Multiple MPI Blending (MMPI), which supports any number of MPI grids with any orientation for mixed rendering.

We place $K$ MPIs with different orientations in 3D space, each with a grid in its own NDC space, modeling all objects from the near plane to infinity. For a particular training viewpoint, we first generate a set of rays $\{\mathbf{r}_{s}\}$ based on all image pixels in that viewpoint. Then, for each ray $\mathbf{r}$ in the set, we need to obtain its intersection points with all MPI planes and arrange them in the order of their distance from the camera. This process can be done using traditional planar intersecstion algorithms (e.g., DDA [30]), but it can significantly slow down the training and rendering process. Instead, we propose the “sample and merge” strategy (Fig. 3), which effectively improves the sampling speed.

“Sample and merge” strategy. For each ray $\mathbf{r}=(\mathbf{o},\mathbf{d})$, we first project it into each MPI’s NDC space and sample the corresponding MPI $\mathcal{M}_{i}$ to get a set of intersection points $\{p_{j}\}_{i}$. The projected ray direction is then denoted as $Proj_{i}(\mathbf{d})$. Due to the ambiguity of the z-axis direction in the NDC space transformation, we need to perform an additional forward-facing check to mask out the points with z-component less than zero (which are back to the current MPI $\mathcal{M}_{i}$) in the $Proj_{i}(\mathbf{d})$ orientation. Then we input $\{p_{j}\}_{i}$ and $Proj_{i}(\mathbf{d})$ into the density and feature grid of the corresponding MPI to obtain the density and color sets $\{\sigma_{j}\}_{i}$, $\{c_{j}\}_{i}$, and further calculate the alpha set $\{\alpha_{j}\}_{i}$ by:

	$\displaystyle\{\sigma_{j}\}_{i}$	$\displaystyle=interp(\{p_{j}\}_{i},\mathbf{V}_{\sigma,i}),$		(8)
	$\displaystyle\{\mathcal{F}_{j}\}_{i}$	$\displaystyle=interp(\{p_{j}\}_{i},\mathbf{V}_{feat,i}),$		(9)
	$\displaystyle\{c_{j}\}_{i}$	$\displaystyle=MLP^{(rgb)}_{\Theta}(\{\mathcal{F}_{j}\}_{i},\{p_{j}\}_{i},Proj_{i}(\mathbf{d})),$		(10)
	$\displaystyle\{\alpha_{j}\}_{i}$	$\displaystyle=\{1-\exp(-\sigma_{j}\delta_{j})\}_{i}.$		(11)

Then all point sets $\{p_{j}\}_{i}$ are projected to the original 3D space and sorted according to the distance from the camera to obtain an ordered set:

	$\displaystyle\{p_{k},\alpha_{k}\}_{k=1}^{\sum_{l}^{K}n(\{p_{j}\}_{l})},$		(12)
	$\displaystyle\text{where}\ ||Proj^{-1}_{a}(p_{k-1})-\mathbf{o}||_{2}<||Proj^{-1}_{b}(p_{k})-\mathbf{o}||_{2},$
	$\displaystyle\ p_{k-1}\in\{p_{j}\}_{a},\ p_{k}\in\{p_{j}\}_{b}.$		(13)

Then we render the pixel color by:

$\displaystyle\mathbf{\hat{C}}(\mathbf{r})=\sum_{k=1}^{\sum_{l}^{K}n(\{p_{j}\}_{l})}T_{k}\alpha_{k}c_{k},\quad\text{where}\ T_{k}=\prod_{j=1}^{k-1}(1-\alpha_{j}).$

(14)

The technique of Multi-MPI blending can be expanded to encompass unbounded 360-degree scenes like indoor scenes, where the entire 3D space can be modeled by setting up multiple MPIs and making their frustum range cover the entire 360-degree space.

When there is a large overlap among MPIs, training by directly blending them may result in some degree of degradation. One possible explanation for this is that MPIs with a sparse distribution of planes at certain locations tend to learn faster, whereas MPIs with a denser distribution of planes learn at a slower rate but have better modeling ability. When these MPIs are simultaneously trained, those that are learned first may impede the learning of other MPIs, preventing the finer-level learning of certain parts and leading to a decreased ability to model small objects.

To address the issue mentioned above, we propose a solution called per-voxel reliability learning. We learn the relative confidence of multiple MPIs at each location in an end-to-end manner, and utilize an adaptive blending operation to merge MPIs, which gives larger blending weights to those advantaged MPIs with better local representation abilities while giving lower weights to those with less representation abilities.

Reliability grid and adaptive blending. We begin by creating a 1-channel reliability grid $\mathbf{V}_{r,i}$ for each MPI $\mathcal{M}_{i}$, which has the same size and resolution as the 3D density voxel grid in the NDC space where it is situated. For the $i$-th MPI $\mathcal{M}_{i}$, we obtain the reliability $\mathcal{R}_{i}(p)$ at a point $p$ in its NDC through trilinear interpolation of 3D coordinates:

	$\displaystyle\mathcal{R}_{i}(p)$	$\displaystyle=interp(p,\mathbf{V}_{r,i}),$		(15)
	$\displaystyle interp(p,\mathbf{V}_{r,i})$	$\displaystyle:(R^{3},R^{1\times N_{x}\times N_{y}\times N_{z}})\to\mathbb{R}.$		(16)

For the intersection point $p$ sampled at $\mathcal{M}_{i}$, to obtain its relative confidence among other MPIs, we project it to the NDC coordinate system where another MPI $\mathcal{M}_{j}$ is located, and obtain the corresponding reliability $\mathcal{R}_{j}(p)$ in the $j$-th MPI by:

	$\displaystyle\mathcal{R}_{j}(p)=interp(Proj_{j}(Proj^{-1}_{i}(p)),\mathbf{V}_{r,j}),$
	$\displaystyle\text{where}\ j\not=i$		(17)

Note that here the point $p$ does not necessarily fall on the image plane of the target MPI $\mathcal{M}_{j}$ after two reprojections, but since reliability varies continuously, we use trilinear interpolation at this point, interpolating on all $x,y,z$ axes. After that, the relative confidence $P_{i}(p)$ of point $p$ over all reliabilities is calculated using the softmax function:

$\displaystyle P_{i}(p)=\frac{e^{\mathcal{R}_{i}(p)}}{\sum_{j=1}^{K}e^{\mathcal{R}_{j}(p)}}.$

(18)

Then the rendering formula Eq.(14) is updated to the following form:

	$\displaystyle\mathbf{\hat{C}}(\mathbf{r})=\sum_{k=1}^{\sum_{l}^{K}n(\{p_{j}\}_{l})}T_{k}P_{k}\alpha_{k}c_{k},$
	$\displaystyle\mbox{where}\ \ T_{k}=\prod_{j=1}^{k-1}(1-P_{j}\alpha_{j}).$		(19)

Two-stage reliability learning scheme. To learn the reliability grid, we propose a two-stage learning scheme. Specifically, in the first training stage, each MPI is trained individually without learning the reliability; after this stage, each MPI has received sufficient training and the expressive ability has been fully explored respectively. Then in the second stage, we jointly train all the MPIs and learn their reliability at each spatial position in an end-to-end manner.

The MPI data format has an inherent drawback: it cannot be effectively viewed from the opposite side and is subject to distortion when viewed from an angle nearly parallel to image planes. Consequently, regardless of the number of MPIs used for blending, it becomes challenging to model objects located in the center of the camera trajectory, thereby limiting the scope of application. To address this issue, we propose the extra use of a centered cube voxel grid $\mathcal{C}$ other than the existing MPIs, which represents the scene information in the area surrounding the camera trajectory and nearby regions. This cube grid is created directly in the world coordinate system and interpolates on the $x$, $y$, and $z$ axes, in contrast to the MPI grid, which only interpolates on the $x$ and $y$ axes when obtaining density and color features.

In the rendering process, for each ray $\mathbf{r}=(\mathbf{o},\mathbf{d})$, the point set $\{p_{j}\}_{\mathcal{C}}$ is obtained by sampling the cube grid $\mathcal{C}$ directly in the world coordinate system, then color and alpha set $\{c_{j}\}_{\mathcal{C}}$, $\{\alpha_{j}\}_{\mathcal{C}}$ are obtained by a process similar to that of MPI. Together with the other MPI sampled point sets, they are merged and rendered according to the distance from the camera. Note that since the centered cube grid can only represent part of the scene information in one direction, it is not rendered separately, but always in combination with other MPIs.

Our training loss is defined as:

	$\displaystyle\mathcal{L}$	$\displaystyle=\mathcal{L}_{pho}+\lambda_{pt\_rgb}\mathcal{L}_{pt\_rgb}+\lambda_{bg}\mathcal{L}_{bg}$
		$\displaystyle+\lambda_{dist}\mathcal{L}_{dist}+\lambda_{TV}\mathcal{L}_{TV},$		(20)

where $\mathcal{L}_{pho}$ is the photometric loss, $\mathcal{L}_{bg}$, $\mathcal{L}_{pt\_rgb}$, $\mathcal{L}_{dist}$, $\mathcal{L}_{TV}$ are background entropy loss, per-point color loss, distortion loss, and total variation loss, respectively.

We evaluate our method on two challenging unbounded datasets, KITTI [31] and ScanNet [32]. The KITTI dataset contains image sequences of car trajectories captured by the stereo camera facing forward, and we randomly select 5 near-static sequential segments of scenes for evaluation. We use COLMAP [33] to obtain the ground truth of camera poses, and evenly select 22 images for training, while the remaining 42 images are used for testing. ScanNet is a large dataset containing 1,613 indoor scenes. Following the experimental configurations of NeuRIS [34], we select eight scenes and evenly choose 1/6 of the images in each scene for training. For testing, we randomly choose 500 images except for the ones used for training. For image quality evaluation, we adopt the three widely used metrics PSNR, SSIM[35], and LPIPS_VGG[36]. We further conduct NVS of extremely long trajectories and provide the details in the supplementary video.

For the KITTI dataset, we use four MPI grids (towards front, left, right and below, respectively) for blending, each MPI has a resolution of 270$\times$270, and a total of 256 depths are selected. For the ScanNet dataset, we use five MPI grids (towards front, back, left, right and below, respectively) and one centered cube grid for blending, where each MPI grid has a resolution of 192$\times$192 and a total of 128 depths selected, and the centered cube grid has a resolution of 160$\times$160$\times$160. When training on the KITTI dataset, we first train each MPI grid individually for a total of 30k iterations, and then freeze all MPIs except the reliability grid for a total of 10k iterations of reliability blending learning. For training on ScanNet, we omit reliability learning due to the small overlap between each MPI and train a total of 30k iterations directly.

We chose several representative works that can synthesize new perspective images of unbounded scenes and conducted quantitative comparisons with our MMPI. The works include NeRF [1], NeRF++ [4], mip-NeRF-360 [5], Plenoxels [8], TensoRF [14], F²-NeRF [18], and DVGO [7]. The mip-NeRF-360 experiment was conducted on the A100 GPU due to excessive GPU memory, while other experiments and training time statistics were completed on the Nvidia 3090 GPU.

On the KITTI dataset, MMPI outperforms all baseline methods on metrics except LPIPS (Table I). As shown in Fig. 4, the original NeRF and NeRF++ give very vague prediction results, while Plenoxels, DVGO, and TensoRF show structural artifacts, indicating that simple parameterization cannot solve the long-range NVS problem. The perspective warping of F²-NeRF is helpful in solving the NVS of long-range scenes, but there are still many wrong textures. Mip-NeRF-360, due to its own parameterization, predicts some regions too smoothly, loses detailed information, and is also accompanied by artifacts. In contrast, our MMPI can better restore the overall information of the scene while maintaining details such as roadside poles and road shadows.

We also conducted an evaluation of our method using the challenging 360-degree dataset ScanNet. The results demonstrate that our MMPI approach achieved superior rendering performance in quantitative comparison to all other fast-training NeRF methods (Table II). As shown in Fig. 5, Plenoxels exhibited noticeable banding artifacts, while F²-NeRF presented a large area of false patches. Meanwhile, we observed that mip-NeRF-360 also produced satisfactory results on ScanNet after an extended training period. This is due to its utilization of a larger MLP that can recognize and refine finer textured regions over an extended period of training. However, its prolonged training time poses a limitation on its practical application. An early-stopped mip-NeRF-360 with a training time of 1.5 hours will result in a fuzzier outcome.

For our ablation studies, we selected a sequence (0096) from the KITTI dataset. We compared the use of a single MPI for training and novel view synthesis (equivalent to using DVGO) with our MMPI approach. To further illustrate the effectiveness of our proposed per-voxel reliability learning for scenes with large overlap among MPIs, we also compared the MMPI method with directly training multiple MPIs without incorporating reliability.

The results in Table III indicate that after blending multiple MPIs, the novel view synthesis quality has improved significantly compared to using only a single MPI. Moreover, the introduction of per-voxel reliability learning has further improved image quality. This demonstrates the scene modeling capability of our proposed Multiple MPI blending, as well as the effectiveness of the training pipeline. We provide more qualitative comparisons in Fig. 6.