MP-MVS: Multi-Scale Windows PatchMatch
and Planar Prior Multi-View Stereo

We start by introducing the basic PatchMatch framework (ACMH). PatchMatch converts the depth estimation into a planar estimation problem, where the planar hypothesis of each pixel in the image can be obtained by random generation. Assuming that the plane hypothesis of pixel $p$ in the reference image is $\theta$, for reference image patch $W_{j}$ which is centered on pixel $p$, we can obtain the corresponding source image patch $W_{i}$ by homography. If the planar hypothesis of pixel $p$ is correct, the texture of corresponding patches should be similar. Then the photometric consistency cost $m_{i,j}$ between corresponding patches can be calculated as

Where $cov(x,y)=E(x-E(x))E(y-E(y))$, and $E(.)$ is the expected value of pixel intensities within a patch.

Define weights with each source image as $\omega_{j}$. The multi-view aggregated photometric consistency cost $m_{p}$ for each pixel with plane hypothesis is defined as

PatchMatch samples the planar hypothesis of neighboring pixels and computes aggregation cost. If the aggregation cost of the plane hypothesis is smaller, it is propagated to the current pixel. The PatchMatch typically performs three iterations of propagation optimization with a fixed window size. Unlike the base PatchMatch, our proposed mPM is propagation optimized with multi-scale windows. To ensure that propagation optimized can be implemented consistently and reliably, The following principles should be observed:

• •

  A larger scale window implies more computational cost, and to reduce the computation time, the pixels within the window should be sparsely sampled with equal spacing(Fig. 2), and the photometric consistency metric is computed by sampling points only.

• •

  To ensure that the spatial propagation process is efficient, window size should be the same for all pixels in the same iteration. Similarly, to ensure that all planar hypotheses can be refined at finer scales, the number of sampling points should be the same at different scales.

• •

  Larger-scale windows have a larger receptive field and are therefore able to estimate planar hypotheses for untextured areas more reliably. However, due to the effect of sparse sampling, the depth estimation accuracy obtained from larger-scale windows is poor. Therefore, iterative propagation should be performed starting from a larger scale window and gradually decreasing the window size.

Based on the above criteria, we sample 36 points at equal intervals in a window (Fig. 2). We define the window edge size $L$ at each scale $S$ as

The maximum scale of windows is determined according to the resolution of images. Starting from the maximum scale window, three iterations of propagation optimization are executed at each scale, and then the scale is reduced step by step for a total of $3\times S$ iterations.

Note that at each scale we reset the cost threshold $\tau(t)$ for a good match. To ensure that points that did not obtain reliable planar hypotheses at the previous scale get a robust view selection at this scale. For the t-th iteration under each scale window, the good match cost threshold $\tau(t)$ can be calculated as

where $\tau_{0}$ is the initialized matching cost threshold and $\alpha$ is a constant.

ACMH[9] divides neighborhood pixels into eight regions and adaptively selects the optimal hypotheses of each region to propagate (Fig. 4a). Compared to propagation using only neighboring pixels in [23], this adaptive sampling scheme makes it more likely to sample good hypotheses. Since photometric consistency does not enable reliable depth estimation in untextured areas, only a few reliable planar hypotheses exist for untextured areas (Fig. 3c), and most of the reliable planar hypotheses correspond to texture-rich areas (Fig. 3b). As a result, closer ground sampling in the untextured areas often fails to obtain reliable planar hypotheses, thus hindering effective spatial propagation. This leads to a decrease in the accuracy of the depth estimation. On the other hand, it leads to an increase in the number of outliers. It is more severe by the increase in untextured areas of high-resolution images.

Therefore, we do not sample planar hypotheses in the region near the estimated pixel, but set the sampling region far from the estimated pixel(Fig. 4b). For the four regions of top, bottom, left, and right, we set 10 samples in each region. In the four diagonal regions, we set 12 samples in each region. Our proposed strategy can effectively improve the reliability of depth estimation in untextured areas while suppressing the generation of outliers. Experiments show that our sampling strategy does not affect the depth estimation of thin objects.

There are many planar structures in architectural scenes, that can provide effective planar priors information to assist depth estimation, and many Approaches [27, 10, 26, 30, 31] carry out research on structure priors.

We introduce the planar prior assisted PatchMatch of [27] to improve the completeness of the reconstruction of untextured areas. ACMP divides a whole image into multiple rectangular regions, and a point with the smallest photometric consistency cost is selected in each rectangular region for triangulation, thus the accuracy of the plane prior relies on the reliability of selected points. Due to the ambiguity of photometric consistency in untextured areas, the depths of some of the selected points are wrong (fig. 5c), which results in generating a wrong planar prior model. Prior to COLMAP’s integration of geometric consistency into PatchMatch inference, geometric consistency typically filtered out outliers only in the post-processing stage. A multi-view geometric consistency can effectively refine depth estimation results. We note that using multi-view geometric consistency can find accurate sparse points in textureless regions. A robust strategy for generating planar prior models is to choose triangulated points by geometric consistency. Therefore, instead of directly generating planar prior assisted PatchMatch after computing an initial depth map, we first perform geometric consistency optimization, and the geometric consistency cost $m_{g}$ is calculated as

where $\bigtriangleup e_{i,j}$ is the forward-backward reprojection error between the reference image $I_{j}$ and the source image $I_{i}$.

We select points that satisfy $m_{p}<0.2$ and $m_{g}<1.0$ within each region (Fig. 5d). Fig. 5e shows that geometric consistency can eliminate most of the outliers. Compared to refACMP, we generated a planar prior model that is more accurate.

We performed ablation experiments on the training set of ETH3D for validating the effects of our mPM, checkerboard sampling scheme, and geometric consistency generating plane prior model. We validate the effectiveness of our approach by using ACMH as our baseline method and adding our individual modules incrementally. For depth map fusion, we used the latest version scheme of ACMMP to ensure that the best results are obtained for each module. The results are shown in Table 1.

Compared to baseline, mPM significantly improves the completeness of scene reconstruction through multi-scale windows, while this coarse-to-fine approach has less impact on the loss of accuracy. Similarly, our proposed checkerboard sampling scheme is able to propagate correct hypotheses more efficiently in untextured regions, which is why the completeness of scene reconstruction is significantly improved. Generating a planar prior model from geometric consistency information can obtain more accurate results than using photometric consistency directly. The above experiments show that our proposed approach can well balance accuracy and completeness, and can significantly improve depth estimation of untextured regions with a small loss of accuracy.

We select some SOTA approaches for comparison with ours, and in Table 2 we list the training and test set results of these approaches for the ETH3D high-resolution multi-view benchmark, which are publicly available on the ETH3D benchmark server. Indoor scenes often have a large number of artificial objects, which results in more untextured areas. Since we use several methods to improve the completeness of untextured areas, our approach overall obtains optimal completeness and $F_{1}$-score in indoor scenes. In more complex outdoor scenes, our proposed approach also shows strong performance in terms of accuracy. Overall, our approach is highly adaptable to both indoor and outdoor scenes.

In addition to quantitative analysis, we have also selected some open-source approaches for visual comparison to show the performance of our approach more intuitively. We selected five scenes for depth map estimation: delivery_area, electro, pipes, playground, and terrains. The results of the depth map visualization obtained by several approaches are presented in Fig. 6. COLMAP eliminates unreliable depth estimates by a strict strategy but at the expense of depth map completeness. ACMM and ACMMP obtain good results in untextured areas, but in some challenging areas that are only observed by a few images, these approaches have difficulty in estimating correct depth. It can be seen that the ground in part of the scenes (electro, playground, terrains) only has the correct depth calculated by ours. The playground scene has many thin foregrounds, and our depth estimation results are significantly better than other approaches for thin foregrounds, which confirms that our proposed checkerboard sampling scheme has no significant effect on thin objects.

Comparing point cloud reconstruction results with those publicly available on the ETH3D benchmark server (Fig. 7). From point cloud reconstruction results, our approach can obtain the most complete results, especially for some untextured areas, and our approach is able to obtain more robust results.

We list some approaches and ours to estimate the average running time (per second) of each depth map in Table 3. Considering the variability of data sizes and scenarios, we detail runtimes for all scenes in the ETH3D training dataset, and all approaches running on our platform based on a single GPU. Where ACMM and ACMMP downsample all images 2 times, ACMM performs 7 iterations and ACMMP performs 10 iterations in each scale image. COLMAP performs 5 iterations in a single scale image and our approach performs 16 iterations in a fixed scale image.

Since our mPM simplifies the multi-scale depth estimation, our approach is close to ACMM in terms of running time in most scenes, and even faster in some scenes. Our approach is 17% faster than ACMMP which also uses a multi-scale approach and planar prior assisted PatchMatch, while the performance of our approach is stronger than the above approaches.