MVLayoutNet: 3D layout reconstruction with multi-view panoramas

We treat each panorama $\mathbf{I}_{i}$ as the reference view with a set of associated source views $\mathbf{R}_{i}$. Details of view selection are described in Section 4.1. Standard cost volume is constructed by enumerating several depth planes orthogonal to the reference view, efficiently warping source views to the reference camera space via homography, and measuring differences based on local patches [8] or deep features [32]. However, it does not precisely model the physical world where the orientation of an ideally-aligned patch should be specified by the surface normal. Therefore, we determine the cost for each pixel $\mathbf{p}$ and a depth $d$ in the reference view using homography specified by more accurate layout element orientation instead of the vanilla optical axis, described in Equation 2. We slightly abuse the term “depth” here as the distance between the layout element plane and the camera location.

where $s$ is any source view in $\mathbf{R}_{i}$ related to reference view $i$. $\mathbf{o}_{i,j}$ is determined by the layout element from $\mathcal{L}_{i}$ whose region covers pixel $\mathbf{p}$.

Although local homography depends on pixel locations, layout regions are often continuous and clustered. Therefore, we need only enumerate all layout element orientation $\mathbf{o}_{i,j}$, apply homography transformation to regions in the source views which overlaps with the corresponding layout element in the reference view. The overall computation cost is almost the same as a single homography applied to the whole image. Figure 3(a) illustrates how we apply homography from different planes. Regions in reference views overlapping with different layout elements are highlighted with different colors and applied with separate layout homography given different orientations.

Since our goal is to describe the overall geometry of the layout, we compress the probability volume to describe the layout element feature. First, we build a deep feature volume in the reference camera space for each source view similar to MVSNet [32] but via our layout homography. Then, the cost volume passes through a 3D-UNet to derive a volume probability $\text{Prob}(\mathbf{p}_{k},d)$, which is further compressed into a one-dimensional probability map $\text{Prob}(\mathcal{E}_{i,j},d)$ for each layout element $\mathcal{E}_{i,j}$ by aggregating information at the same depth layer as shown in the Figure 3(b). Mathematical details are described in Equation 3.

$c_{k}$ is a learned confidence to adjust the contribution of pixel $\mathbf{p}_{k}$ to the layout depth estimation, which is discussed in Section 3.4.

We regress the layout depth by aggregating the one-dimensional probability map in Equation 4.

During training, the regressed depth is supervised by ground truth layout depth $\mathcal{D}^{*}$ using L1 norm (Equation 5).

Since our goal is to reconstruct 3D layouts of the scene, we aim to compare with the state-of-the-arts sharing the same task [25, 26]. To our knowledge, these works are the best candidates for comparison since there are no studies that combine MVS with network inference of the layout as we do. Since absolute depth is not available from these methods, we apply a standard scale prediction proposed by [30], denoted as “SP” in all following experiments. We notice that it is also possible to build 3D layouts as a postprocessing step after multi-view stereo, which we will discuss in Section 4.2.

We propose to use 2D-3D-S [1] and ZInD [4] as commonly used or recently released datasets for experiments. In 2D-3D-S, all panorama images from the same room are mutually visible and grouped together as related views. We set 112 groups of images in scene 5 as test data and all other 401 groups as training data. For ZInD, images are grouped based on their associated floor ID. As a result, we obtain 7536 groups of images for training and 945 groups of images for testing. Matterport3D [3] is another commonly used dataset. But it lacks camera poses for skybox images and thus is not appropriate for our multi-view setting.

We propose several metrics to evaluate the quality of the 3D layout. The depth RMSE is the rooted mean square error of the estimated layout depth map and the ground truth layout depth map. It measures the mean error between the predicted layout and the ground truth. The scale error is the absolute difference between the estimated and real camera heights. It serves as an overall metric reflecting how close the predicted scale of environment is to the ground truth. Finally, it is crucial that layouts from different views are consistent in the 3D world, which is measured by coherency as the mean distance of pairs of 3D points extracted from corresponding pixels in the reference and the source images.

Table 1 shows that our method outperforms the state-of-the-art methods in all metrics on the 2D-3D-S dataset [1]. Worth mentioning, we achieve an improvement of 21.7% in terms of depth RMSE, and 50.0% in the scale error. We observe that 2D-3D-S is captured at quite similar camera heights, which reduces the discrepancy of coherency scores among compared methods. As shown in the Figure 4, we can reconstruct more scenes in low depth RMSE which indicates the effectiveness of the proposed method. When the depth RMSE is larger than 25 cm, we have nearly 10% less scenes which suggests that the our method is more robust.

We visualize the reconstructed layouts and the depth error maps for 2D-3D-S in Figure 5. Figure 5(a) plots the layout boundaries predicted from different methods, where the ground truth, HoHoNet [26], HorizonNet [25] and ours are visualized with red, purple, blue, and green colors respectively. As expected, the quality of layouts projected into the image space is quite similar among different methods. However, our improvement of 3D accuracy is significant as shown in Figure 5(b-d).

For ZInD dataset, we show even more salient improvement compared to the state-of-the-art methods as shown in Table 2. This dataset contains more unfurnished and extremely texture-less regions. Such challenging environments highlight the robustness of our network against monocular prediction. We achieve an improvement of 20.6% in terms of depth RMSE, and 64.3% in the scale error. Compared to experiments on 2D-3D-S, our method shows a bigger improvement in terms of coherency on ZInD with a wider range of heights the images are captured. We visualize the reconstructed layouts and the depth error maps in Figure 6 similar to what we do for 2D-3D-S. Note that even when 3D layouts from existing methods are far from ground truth as shown in the first row, our method still derives accurate layout depth from multi-view information. Such a challenging dataset suggests that our method can reconstruct the 3D layouts with much higher quality.

While semantic information is important to layout estimation, there are potentially multiple solutions to introduce them into our system. The most straightforward way which we suggest is to directly specify the confidence of regions according to predicted semantics. Specifically, we treat ceilings, floors, and walls as layout structures and set their confidence $c^{s}$ as one and others zero. An alternative solution is to learn the confidence from semantic features, in which case we pass semantic features through an MLP layer to obtain $c^{s}$. We compare these two choices with a version where $c^{s}$ is always set to one. We set $c^{a}=1$ to focus on semantic confidence and keep other settings the same as discussed in Section 3. Table 3 shows evaluation of different choices of using semantics on 2D-3D-S dataset. We find both learned semantics or user-specified semantic confidence helps improve the accuracy, where our user-specified version performs the best. Another interesting finding is that coherency is nearly the same among all methods, suggesting that our MVS module robustly produces coherent geometry no matter whether cluttered regions are taken into consideration.

Besides semantics information, we add additional degrees of freedom for the network to adjust confidence with a self-attention module. We aim to study the effectiveness of the self-attention confidence $c^{a}$ and its relationship to semantic confidence $c^{s}$. In Table 4, we evaluate a vanilla version by setting both confidence to one, a version where $c^{s}=1$ and $c^{a}$ are learned by our network, and a full version where both $c^{s}$ and $c^{a}$ are activated. We find attention helps to improve the performance, and the full version achieves the best performance. This suggests that attention is beneficial and should be jointly considered with semantic information.

An important question is whether it is feasible to derive a clean and accurate 3D layout by post-processing the MVS results. We design an experiment where MVS is processed with an orientation-fixed plane fitting to obtain the layout, where orientation is produced from our 2D layout network. We evaluate results from MVSNet [32], post-processed by plane fitting (PF), and our network by comparing them with the ground truth depth map. Detailed statistics are shown in Table 5. While it is possible to obtain a clean 3D layout from MVS, the accuracy is not satisfying. Specifically, the depth RMSE is similar to the monocular prediction from a monocular view with a scale prediction (e.g. Table 1, HorizonNet+SP). In contrast, our direct layout-based MVS module can significantly improve the 3D accuracy.