V-FUSE: Volumetric Depth Map Fusion with Long-Range Constraints

Much like other areas of computer vision, Multi-View Stereo (MVS) has benefited from the advent of deep learning. Progress has been driven by the creation of end-to-end systems, unifying all aspects of the MVS pipeline, and by replacing heuristics in the components of the pipeline with optimized network modules. An aspect of MVS that requires further investigation is depth map fusion, which is still implemented as a sequence of heuristic operations.

Considering that the top performing MVS systems in terms of geometric accuracy¹¹1NeRF [27] has inspired a vastly expanding class of algorithms that produce superior results in view synthesis, but not in 3D reconstruction. We consider NeRF a separate line of work from MVS. use depth map collections as the representation, depth map fusion can be a crucial step for obtaining the final 3D reconstruction of the scene. As has been shown by conventional fusion research [24], fusing depth maps, guided by geometric constraints, improves the precision of correct depth estimates by blending them with supporting estimates for the same part of the surface, detects and removes outliers, and reduces redundancy in the final 3D model. Current deep MVS approaches [3, 11, 22, 26, 43, 41, 42], however, bypass depth map fusion and proceed directly to filtering fusion, which includes various heuristic post-processing steps to obtain a global point cloud by filtering the point cloud reconstructed from the set of depth maps. This approach has been successful; however, without depth map fusion, not all geometric information from the scene is utilized. Our motivation in this work is to build an end-to-end fusion network that can generate much more accurate depth and confidence maps.

Filtering fusion that operates on local 3D neighborhoods is unable to leverage relationships among distant surface primitives, such as a surface being occluded from a faraway object. Similarly, convolution networks have a limited receptive fields and can only reason about local interactions. We present V-FUSE, an approach that allows a 3D convolutional network to benefit from such geometric information, in a differentiable manner, controlled by learnable hyper-parameters.

V-FUSE considers three types of constraints, inspired by the work of Merrell et al. [24]: support among consistent depth estimates across multiple views, occlusions and free-space violations that provide evidence against depth estimates contradicting surfaces estimated in different depth maps. Free-space violations provide the added benefit of encoding conflicts with respect to surfaces that may be invisible in the frame of the reference camera. There are three substantial differences between our approach and that of Merrell et al.: (i) theirs operates in $2\frac{1}{2}$D while ours operates in a 3D volume, (ii) their algorithms make decisions per pixel without considering context, and (iii) all parameters in our approach are learned end-to-end. Specifying visibility constraints in the fusion volume allows V-FUSE to reason based on interactions among depth estimates along the rays, as well as spatially among neighboring voxels. In the absence of these constraints, only the latter would have been possible via 3D convolutions, which cannot reason about long-range conflicts.

Reducing the storage and computational requirements of deep MVS networks is a necessity for increasing the resolution and quality of 3D reconstruction. 3D convolutional networks operating on cost volumes are forced to downsample high resolution inputs. Since our framework is also volumetric, we propose a technique for achieving high resolution near the surfaces while keeping memory requirements manageable. Specifically, we learn to generate a per-pixel, narrow depth search window by examining the input depth and confidence estimates. Unlike previous networks that iteratively refine the depth search space, our framework leverages the availability of input depth and confidence estimates to determine a reduced search space in a single pass.

Our main contributions are:

• •

  An end-to-end learning-based method for the fusion of depth and confidence maps, leveraging long-range, volumetric visibility constraints encoded into a visibility constraint volume (VCV).

• •

  A pixel-wise search window estimation sub-network to refine the depth search space.

We provide extensive evaluation of V-FUSE on MVS benchmarks [1, 20, 44], using 2D and 3D error metrics.

In this section, we review related work on learning-based MVS, as well as conventional and learning-based depth map fusion.(Unfortunately, no recent surveys on these topics are available, to the best of our knowledge.)

The combination of deep learning and plane-sweeping stereo has inspired a new generation of MVS algorithms. The plane-sweeping volume (PSV) [10] allows the use of cost aggregation and disparity estimation techniques developed for binocular stereo [19] in multi-view settings. The first deep learning-based plane-sweeping algorithm was DeepStereo [8] that addresses view synthesis in a self-supervised manner. Supervised formulations targeting depth map estimation are largely influenced by MVSNet [43] and concurrent work [15, 16]. We also adopt the PSV structure in this work for our fusion volume.

Several methods [3, 11, 22, 26, 36, 41, 42, 45] aim to improve memory efficiency in deep MVS through multi-resolution, iterative schemes that refine the depth search space with each increase in resolution. This is achieved via regular incremental reductions in search range [11, 42], or with a range set using only confidence estimates [3]. We have developed a non-iterative method for estimating per-pixel depth search windows based on information extracted from the distribution of input depth and confidence maps.

Recent work has addressed MVS by: combined classification and regression for depth estimation [30, 35], sequential depth interval selection [33], an adaptation of RAFT (Recurrent All-Pairs Field Transforms) [23], operating over adaptive intervals along epipolar lines instead of discrete depths [22], and the use of a non-parametric depth distribution model to mitigate shortcomings of unimodal depth models [41]. Transformers for MVS [6, 12, 34, 37] leverage the intra- and inter-attention mechanisms to achieve more accurate feature matching than previous architectures.

A popular choice for fusing depth maps among deep MVS pipelines is the work of Galliani et al. [9]. It is based on the projection of depth estimates onto several supporting depth maps to accumulate consensus subject to criteria on reprojection error and surface normal inconsistency. The dense COLMAP pipeline [32] also includes a fusion module that rejects outliers based on lack of photometric and geometric support and clusters inliers. Both techniques require setting several thresholds and parameters, and are limited to filtering depth maps into a final 3D model without improving the underlying depth maps.

Some deep MVS systems introduce custom fusion and filtering steps which are not included in the end-to-end trainable pipeline. These include P-MVSNet [21] that considers pixel and depth reprojection errors, and D²HC-RMVSNet [40] that includes geometric consistency scores. Instead of relying on filtering and averaging depth estimates, our work aims to refine and fuse depth maps before they are filtered and projected into a point cloud.

Recent methods [4, 38, 39] propose fusing streams of input depth maps by learning TSDF volume updates [38], by fusing in the latent space and learning a translator to produce a final TSDF volume [39], or by learning pose invariant scene volumes jointly with a MVS sub-network [4]. Volumetric methods suffer from large storage requirements. Our method is volumetric but allows for a very thin volume in the direction of the optical axis.

Donné and Geiger [7] developed a non-volumetric data-driven approach for fusing depth maps, estimated conventionally or by a learning-based technique. DeFuSR filters out wrong depth estimates, but also improves correct ones via refinement and inpainting sub-networks. It operates on re-projected depth estimates and image features at high resolution, depending entirely on 3D convolutions to reason about consensus.

In this section, we introduce the architecture of V-FUSE (an overview can be found in Figure 2). Our network takes as input a reference depth map $D_{0}\in\mathbb{R}^{HxW}$ and the corresponding confidence map $C_{0}\in\mathbb{R}^{HxW}$, and $N-1$ source depth and confidence maps $\{D_{v}\}_{v=1}^{N-1}\in\mathbb{R}^{HxW}$ and $\{C_{v}\}_{v=1}^{N-1}\in\mathbb{R}^{HxW}$ respectively. We begin by rendering the input source maps into the reference view to obtain $\{D^{ref}_{v}\}_{v=1}^{N-1}$ and $\{C^{ref}_{v}\}_{v=1}^{N-1}$. With this set of rendered maps, we build a visibility constraint volume (VCV), whose structure follows that of the plane-sweeping volume. The VCV encodes long-range, volumetric, constraints at each voxel. We use a 3D convolutional network to regularize the VCV and regress the final fused depth map. As output, our network produces fused depth and confidence maps for the reference view, $D^{f}$ and $C^{f}$. The construction of the VCV and the 3D convolutional network are supervised using an $l_{1}$ loss between the estimated and ground truth depth maps. For memory and run-time efficiency, we introduce a novel search window estimation (SWE) sub-network in order to restrict the depth search space used as input in the construction of the VCV. This sub-network is supervised through a novel loss that we discuss in Section 4.2.

We train our network in a supervised manner on the output depth and confidence maps of MVS frameworks. We formulate two loss functions, one for each sub-network.

We have presented an end-to-end depth map fusion network that leverages long-range visibility constraints encoded into a learnable pipeline. Our method improves input depth and confidence maps generated by MVS networks, integrating multi-view consensus and inconsistency measures. We also present a novel depth search space refinement sub-network that estimates a narrow search window along each ray to increase memory and run-time efficiency, as well as allow for high resolution depth estimation near surfaces. The combination of these concepts is able to obtain fused depth maps that are quantitatively and qualitatively much better than the inputs. While the depth map fusion in our work is end-to-end, merging the depth estimates into a unified point cloud remains a heuristic-driven process. We aim to incorporate a more principled point cloud reconstruction procedure from a collection of depth maps in future work. We also aim to explore the generalization ability of learning-based fusion.

In this document, we provide additional implementation details (Section S.1), formulations for learnable geometric constraints (Section S.2), details on confidence computation and evaluation (Section S.3), ablation studies evaluating architecture contributions (Section S.4), further quantitative evaluations (Section S.5), and additional qualitative comparisons and examples (Section S.6).

During training and inference, we set the number of input views for fusion to $N=5$ and the number of depth planes to $M=8$ for input depth maps from all methods. For the DTU [1] dataset, we use the maximum output resolution of each method as the input for V-FUSE. Specifically, we use $400\times 296$ for MVSNet, $1600\times 1184$ for UCSNet, and $1600\times 1152$ for NP-CVP-MVSNet and GBi-Net. For training, we scale the input by a factor of $0.5$, with the exception of MVSNet, for which we train at the full resolution. For Tanks & Temples [20], we use input resolutions of $1920\times 1056$ for UCSNet and $1920\times 1024$ for both NP-CVP-MVSNet and GBi-Net. As the minimum and maximum allowable search window radii, we use $\psi_{min}=0.005$ and $\psi_{max}=0.50$. The terms of the loss are weighed by $\lambda_{d}=0.5$, $\lambda_{c}=20.0$, and $\lambda_{r}=0.5$.

We define two parameters used in the formulations of the geometric constraints. For support, we use $\sigma_{p}$ to determine the sharpness of the support response boundary.

Here, $\gamma_{\sigma}$ is a learned hyper-parameter, $B^{min}_{p}$ and $B^{max}_{p}$ are the minimum and maximum depth bounds per pixel, $M$ is the number of depth hypotheses, and $b_{min}$ and $b_{max}$ are the overall minimum and maximum depth bounds that are given as input for the current reference view. Lower values of $\gamma_{\sigma}$ correspond to a tighter support window. Since this is a function of the per-pixel depth bounds, support adapts to the confidence at each pixel.

For occlusions and free-space violations, we use $\lambda_{p}$ to determine the sharpness of the sigmoid response boundary.

Here, $\gamma_{\lambda}$ is a learned hyper-parameter. Lower values of $\gamma_{\lambda}$ correspond to a softer sigmoid response boundary. Occlusions and free-space violations also adapt to the confidence at each pixel. See Figure S.1 for a visualization of the response boundary.

To evaluate the quality of confidence estimates, we report the area under the curve (AUC). The AUC is the area under the ROC curve, which maps the error rate for the depth map as a function of density based on the sorted confidence values [13]. We first sort the estimated depths according to confidence, and form the sparsification curve of MAE vs. depth map density by progressively dropping the least confident depths [13] to obtain depth maps of lower density, and presumably lower error. Small area under the sparsification curve indicates that confidence has been estimated well and can be used to rank depth estimates accurately.

The fused confidence maps are directly computed from the estimated search window radius. After we normalize the per-pixel window radii from 0 to 1, the confidence value at each pixel is $C^{f}_{p}=1-R_{p}$. Intuitively, a larger estimated radius for a given pixel should indicate lower confidence in the final depth estimate. This relationship is also reflected and enforced in our loss function. To compare the quality of the confidence maps, we report the AUC for all methods in Table S.1. The output confidence values after fusion prove to be more reliable estimates of confidence. Qualitative confidence map results can be seen in Figure S.2.

We provide ablation studies evaluating the contributions of several aspects of the network architecture. In Table S.2, we evaluate the MAE isolating the SWE sub-network, as well as the different constraints. We show the contributions of using the brute-force search space approach, as well as the contributions of using only support and only occlusions and free-space violations. In Table S.3, we show the memory, run-time, and parameter difference between the brute-force approach and the SWE sub-network.

Most state-of-the-art Deep MVS architectures directly compute confidence estimates from the output probability volume of the network, using a small window around the estimated depth voxel. We explore using this method, incorporating the radius estimate from the SWE sub-network, as well as using only the radius to compute confidence. Specifically, following previous work in Deep MVS, we compute confidence maps from the output probability volume of the network by summing the probability values for the four surrounding voxels corresponding to the index of the selected depth. We then test adding the inverse of the radius value from the SWE sub-network as a weighting to this confidence. Finally, we test using only the inverse radius value to compute our confidence. We evaluate these different methods of producing confidence maps in Table S.4. In all instances, the confidence maps produced by V-FUSE are better indications of depth estimate quality, and can be used to more effectively rank depth estimates.

We provide additional results on the BlendedMVS [44] dataset. We report the MAE on the validation set between the GBi-Net [26] input depth maps and the V-FUSE output fused depth maps. We used the pre-trained GBi-Net model trained on the BlendedMVS training set and tested V-FUSE using the model trained on DTU without any fine-tuning. The quantitative results are presented in Table S.6. We show significant improvements in all scenes. Qualitative results can be viewed in Figure S.4.

We also provide the Precision and Recall, alongside the F-Score on the Tanks & Temples [20] intermediate test set, retrieved from the benchmark leaderboard. The Precision score is the percentage of points in the reconstructed point cloud that have a Chamfer distance to the closest point in the ground-truth point cloud below some threshold, $\tau$. The Recall score is the percentage of points in the ground truth-point cloud that have a Chamfer distance to the closest point in the reconstructed point cloud below the same threshold, $\tau$. The F-Score is then the harmonic mean of these two scores. Quantitative results can be found in Table S.5. We improve the Precision and F-Score using UCSNet as input, and improve the Recall using GBi-Net as input.

We show additional qualitative results for all baselines. Figure S.3 shows a comparison of depth and error maps between NP-CVP-MVSNet [41] and V-FUSE on several scenes from the DTU [1] evaluation set. In Figure S.5, we provide a comparison between the final 3D reconstruction results of NeuralFusion [39] and V-FUSE. We only provide a qualitative comparison, as NeuralFusion does not provide any quantitative results on any of the datasets used in our experiments. Figure S.6 shows a comparison of depth and confidence maps between GBi-Net [26] and V-FUSE on several scenes from the DTU evaluation set. In Figure S.7, we can observe the depth maps comparison of scenes from the Tanks & Temples [20] dataset between GBi-Net [26] and V-FUSE. We provide depth map comparisons from several scenes of the validation set from BlendedMVS [44] using GBi-Net [26] as input in Figure S.8. We also show final reconstructions from the DTU and Tanks & Temples datasets in Figure S.9 and Figure S.10, respectively.