Non-local Recurrent Regularization Networks for Multi-view Stereo

Similar to previous works, we use the plane-sweeping algorithm (Collins 1996) to construct cost maps at different depth planes. Specifically, after extracting high-resolution deep image features $\{\boldsymbol{F}_{i}\}_{i=0}^{N-1}$ for all input images by stacked dilated convolutions (Yan et al. 2020), we compute the warped image features $\{\tilde{\boldsymbol{F}}_{i}(d)\}_{i=1}^{N-1}$ for $N-1$ source images at depth value $d$ by differentiable homography warping. Then the 2D cost map at depth value $d$ is calculated by:

$$\boldsymbol{C}(d)=\frac{\sum_{i=1}^{N-1}\big{(}1+{\boldsymbol{w}}_{i}(d)\big{)}\odot\big{(}\tilde{\boldsymbol{F}}_{i}(d)-\boldsymbol{F}_{0}\big{)}^{2}}{N-1},$$

(1)

where ${\boldsymbol{w}}_{i}(d)$ is the view weight of the $i$-th source image and ‘$\odot$’ represents element-wise multiplication. Herein, ${\boldsymbol{w}}_{i}(d)$ is computed by applying one convolution layer followed by group normalization and rectified linear units, a residual block (He et al. 2016) and one convolution layer followed by a sigmoid function on the feature difference, $\tilde{\boldsymbol{F}}_{i}(d)-\boldsymbol{F}_{0}$. In this way, a cost volume $\{\boldsymbol{C}(d)\}_{d=0}^{D-1}$ can be obtained by concatenating 2D cost maps of all depth planes in the depth direction, where $D$ is the total number of depth planes.

As the core of learning-based MVS methods, cost volume regularization is important to aggregate context information in both spatial and depth domain. Existing recurrent 2D cost map regularization methods cast the whole cost volume as a series of 2D cost maps in the depth direction and sequentially regularize 2D cost maps via stacked recurrent neural networks. Although such methods are memory-friendly, they only explicitly consider the local interactions between adjacent depth values.

Inspired by non-local methods (Wang et al. 2018; Fu et al. 2019), we design a non-local recurrent regularization network to model long-range dependencies along the depth dimension, which helps to regularize cost volumes by aggregating more context information. As illustrated in Fig. 2, we divide the whole cost volume into $T$ blocks, each block contains $s$ cost maps. Each cost map is regularized by stacked convolutional LSTM cells to incorporate context information in both the spatial and the depth domain. As depicted in Fig. 3(a), our stacked convolutional LSTM cells are built upon 2D U-Net architecture to aggregate multi-scale context information in the spatial domain. Moreover, these LSTM cells contain one non-local LSTM cell and four vanilla LSTM cells. This non-local LSTM cell is beneficial to incorporate more context information in the depth direction.

Next we elaborate on the mechanism of our non-local LSTM cell. As illustrated in Fig. 3(b), besides using the current cost map $\boldsymbol{C}(d)$, the previous regularized cost map $\boldsymbol{C}_{r}(d-1)$ and the previous cell state map $\boldsymbol{\mathcal{C}}(d-1)$ as input like the vanilla LSTM cells, the non-local LSTM cell also incorporates the cell state map of the previous block, $\boldsymbol{B}(t-1)$, to consider long-range dependencies in the depth direction. How to obtain $\boldsymbol{B}(t-1)$ through the depth attention module will be detailed in the next section. With these inputs, the forget gate map ${\bf F}(d)$, candidate state map $\hat{\boldsymbol{\mathcal{C}}}(d)$, input gate map ${\bf I}(d)$ and output gate map ${\bf O}(d)$ are respectively modeled as:

	$\displaystyle{\bf F}(d)$	$\displaystyle=$	$\displaystyle\operatorname{sigmoid}$	$\displaystyle({\bf W}_{f}\ast[\boldsymbol{C}(d),\boldsymbol{C}_{r}(d-1)]+{\bf b}_{f}),$		(2)
	$\displaystyle{\bf I}(d)$	$\displaystyle=$	$\displaystyle\operatorname{sigmoid}$	$\displaystyle({\bf W}_{i}\ast[\boldsymbol{C}(d),\boldsymbol{C}_{r}(d-1)]+{\bf b}_{i}),$		(3)
	$\displaystyle\hat{\boldsymbol{\mathcal{C}}}(d)$	$\displaystyle=$	$\displaystyle\tanh$	$\displaystyle({\bf W}_{c}\ast[\boldsymbol{C}(d),\boldsymbol{C}_{r}(d-1)]+{\bf b}_{c}),$		(4)
	$\displaystyle{\bf O}(d)$	$\displaystyle=$	$\displaystyle\operatorname{sigmoid}$	$\displaystyle({\bf W}_{o}\ast[\boldsymbol{C}(d),\boldsymbol{C}_{r}(d-1)]+{\bf b}_{o}),$		(5)

where ‘$\ast$’ means convolution, ‘$[]$’ means concatenation, ${\bf W}$ is a transformation matrix and ${\bf b}$ is a bias term. Then, the current cell state map $\boldsymbol{\mathcal{C}}(d)$ and the current regularized cost map $\boldsymbol{C}_{r}(d)$ are computed as,

$$\boldsymbol{\mathcal{C}}(d)={\bf F}(d)\odot\boldsymbol{\mathcal{C}}(d-1)+{\bf I}(d)\odot\hat{\boldsymbol{\mathcal{C}}}(d)+{\bf A}(d)\odot\boldsymbol{B}(t-1),$$

(6)

$$\boldsymbol{C}_{r}(d)={\bf O}(d)\odot\tanh(\boldsymbol{\mathcal{C}}(d)),$$

(7)

where ${\bf A}(d)$ is the depth attention gate map given by

$${\bf A}(d)=\operatorname{sigmoid}({\bf W}_{a}\ast[\boldsymbol{C}(d),\boldsymbol{B}(t-1)]+{\bf b}_{a}).$$

(8)

Note that, the last term in the right side of Eq. (6) reflects the non-local interactions in the depth direction. This is the difference between our non-local LSTM cell and the vanilla LSTM cell. Finally, the regularized cost maps $\{\boldsymbol{C}_{r}(d)\}_{d=0}^{D-1}$ pass a softmax layer to produce the probability volume $\boldsymbol{P}$. In this way, our regularization not only incorporates multi-scale context information in the spatial domain, but also aggregates long-range dependencies in the depth direction.

In the previous section, the depth attention module plays an important role in distilling latent high-level features within each block. It captures discriminative depth context information for each block. Then, the high-level features between blocks are interacted in a gated recurrent manner to capture global depth context information. Finally, we can use this information to impose explicit long-range regularization constraints for non-adjacent depth values as described above.

The structure of the depth attention module is illustrated in Fig. 4. For each block, we only use every other depth value in the block to sample raw cost features $\boldsymbol{F}_{\text{raw}}\in\mathbb{R}^{32\times\frac{s}{2}\times H\times W}$ and regularized cost features $\boldsymbol{F}_{\text{reg}}\in\mathbb{R}^{8\times\frac{s}{2}\times H\times W}$, where $H$ and $W$ denote the image height and width respectively. This enables the larger receptive fields in the depth direction. We first concatenate these two kinds of cost features into complex cost features $\boldsymbol{F}_{\text{comp}}\in\mathbb{R}^{40\times\frac{s}{2}\times H\times W}$. Such dense connection is good to explore their potential interactions and learn non-local information (Huang et al. 2017). Then we use max-pooling and average-pooling along the depth dimension to aggregate discriminative depth information, generating two different depth context features, $\boldsymbol{F}_{\text{max}}$ and $\boldsymbol{F}_{\text{avg}}$. As pointed out in (Woo et al. 2018), we think that max-pooling is good to gather distinctive clues while average-pooling is beneficial to learn the extent of depth space. Using both can highlight the informative depth values as much as possible. These two features are further concatenated and forwarded into one convolution layer and a residual block to produce attention features $\boldsymbol{F}_{\text{att}}\in\mathbb{R}^{16\times H\times W}$.

To further model the global context information in the depth direction, the attention features between blocks are interacted by another recurrent neural network. Specifically, we reshape raw cost features $\boldsymbol{F}_{\text{raw}}\in\mathbb{R}^{32\times\frac{s}{2}\times H\times W}$ to $\mathbb{R}^{C\times H\times W}$, where $C=32\times\frac{s}{2}$. The input gate map ${\bf G}_{i}(t)$ and the forget gate map ${\bf G}_{f}(t)$ are modeled as

	$\displaystyle\!\!\!\!\!{\bf G}_{i}(t)$	$\displaystyle=\operatorname{sigmoid}({\bf W}_{ia}\!\ast\![\boldsymbol{F}_{\text{raw}}(t),\boldsymbol{B}(t-1)]\!+{\bf b}_{ia}),\!\!$		(9)
	$\displaystyle\!\!\!\!\!{\bf G}_{f}(t)$	$\displaystyle=\operatorname{sigmoid}({\bf W}_{fa}\!\ast\![\boldsymbol{F}_{\text{raw}}(t),\boldsymbol{B}(t-1)]\!+{\bf b}_{fa}),\!\!$		(10)

where ${\bf W}$ is a transformation matrix and ${\bf b}$ is a bias term. Finally, we update the cell state map of the current block

$${\boldsymbol{B}}(t)={\bf G}_{i}(t)\odot\tanh(\boldsymbol{F}_{\text{att}}(t))+{\bf G}_{f}(t)\odot\boldsymbol{B}(t-1).$$

(11)

By updating the cell states of different blocks in a gated recurrent manner, our method is able to capture global context information in the depth direction. These cell states of blocks are in turn used to update the cell states of the LSTM cells.

Following previous practices (Yao et al. 2019; Yan et al. 2020), we cast the depth inference task as a multi-class classification problem. We use the following cross-entropy loss to train our network:

$${\cal{L}}=\sum_{\boldsymbol{p}\in\Phi}\sum_{d=0}^{D-1}-\boldsymbol{G}(d,\boldsymbol{p})\cdot\log(\boldsymbol{P}(d,\boldsymbol{p})),$$

(12)

where $\Phi$ denotes the set of valid ground truth pixels, $\boldsymbol{G}(d,{\bf p})$ is the one-hot vector generated according to the ground truth depth map at pixel ${\bf p}$, and $\boldsymbol{P}(d,{\bf p})$ is the predicted depth probability at pixel ${\bf p}$. Note that, we do not need to store the whole classification probability volume during testing. Moreover, since the cell states of blocks are also sequentially updated, our designed depth attention module will not occupy too much memory.

After estimating depth maps for all input images, it is necessary to filter out wrong depth estimates for each depth map and then fuse them into a consistent point cloud representation. There are two critical factors for filtering wrong depth predictions: depth probability and depth consistency. The depth probability is usually measured by the corresponding probability of the selected depth, i.e., $\theta$. The depth consistency is usually measured by the geometric constraint: reprojection errors and relative depth errors (Yao et al. 2018). For a pixel ${\bf p}$ in the reference image $\boldsymbol{I}_{0}$, we denote its reprojection error w.r.t the source image $\boldsymbol{I}_{i}$ by $\psi_{i}$ and its relative depth error w.r.t. $\boldsymbol{I}_{i}$ by $\phi_{i}$. Then, its consistency view set is defined as:

$$\mathcal{S}=\{\boldsymbol{I}_{i}|\psi_{i}<\epsilon,\phi_{i}<\eta\},$$

(13)

where $\epsilon$ and $\eta$ are related thresholds. If $|\mathcal{S}|>\mu$ and $\theta>\tau$, the depth estimate of ${\bf p}$ is reliable, where $\mu$ is the threshold of consistent view number and $\tau$ is the probability threshold. Previous methods (Yao et al. 2018, 2019; Gu et al. 2020) usually set fixed threshold parameters to remove unreliable depth estimates. However, these intuitively preset parameters make their fusion methods not robust for different scenes. Thus, $\text{D}^{2}$HC-RMVSNet (Yan et al. 2020) proposes a dynamic consistency checking algorithm to measure depth consistency. Its dynamic consistency view set is defined as:

$$\mathcal{S}_{d}=\{\boldsymbol{I}_{i}|\psi_{i}<\epsilon(\mu),\phi_{i}<\eta(\mu),\epsilon(\mu)=\frac{\mu}{4},\eta(\mu)\!=\!\frac{\mu}{1300}\},$$

(14)

The above definition makes the thresholds of geometric constraint be adaptively adjusted based on the consistent view number $\mu$. This means that the estimated depth value is accurate and reliable when satisfying strict depth consistency in a small number of views, or relaxed depth consistency in the majority of views. However, this method still adopts a fixed depth probability threshold to filter depths with different depth consistency. Due to the uncertainty of depth prediction networks (Kendall and Gal 2017), the depth probability will sometimes filter credible depth values that meet the strict depth consistency, making reconstructed point clouds less complete. Thus, the depth probability threshold should also be adjusted according to the level of depth consistency. Based on these observations and our experiments, we define the dynamic probability threshold as:

$$\tau(\mu)=0.6\cdot\exp\left[{\frac{(\mu-10)}{8}}\right].$$

(15)

By traversing different $\mu$ values, the estimated depth of ${\bf p}$ will be deemed accurate and reliable if $|\mathcal{S}_{d}|>\mu$ and $\theta>\tau(\mu)$. That is, when the reprojection error and relative depth error are strict, the number of consistent view and depth probability threshold are relaxed to judge if the depth prediction is reliable. Finally, the reliable depth estimates will be projected into 3D space to generate 3D point clouds.

In this section, we directly use our model trained on the DTU training set without any fine-tuning for benchmarking evaluations. We compare our method with other state-of-the-art MVS methods, including geometric and learning-based methods. The geometric methods include Furu (Furukawa and Ponce 2010), Tola (Tola, Strecha, and Fua 2012), Gipuma (Galliani, Lasinger, and Schindler 2015), COLMAP (Schönberger et al. 2016), ACMM (Xu and Tao 2019) and ACMP (Xu and Tao 2020b). For the learning-based methods, SurfaceNet (Ji et al. 2017), MVSNet (Yao et al. 2018), R-MVSNet (Yao et al. 2019), CasMVSNet (Gu et al. 2020), CVP-MVSNet (Yang et al. 2020), UCSNet (Cheng et al. 2020), AttMVS (Luo et al. 2020) and $\text{D}^{2}$HC-RMVSNet (Yan et al. 2020) are compared.

To further demonstrate the effect of the proposed non-local recurrent regularization, we try to visualize the cell states of different depth blocks. In fact, it is hard to give comprehensive visualization about these cell states. Instead, we show the probability distribution of these cell states to see whether they indicate the global scene context. To this end, we average the features of $\boldsymbol{B}(t)$ along the channel dimension and apply the softmax operation along the depth dimension for all depth blocks to obtain the probability distribution. Then, this probability distribution is aligned with the probability distribution of all depth planes by copying the value of each block to its covering $s$ depth planes. The visualization results are shown in Fig. 7. The circle region in Fig. 7 is challenging as it is close to the central object in the spatial domain but in fact it belongs to the background areas. Without non-local recurrent regularization, it is easy for the baseline model to propagate the depth information of the central object to the circle region, resulting in incorrect estimation for this region (cf. Fig. 7(a)). With our proposed non-local recurrent regularization, the cell states of depth blocks indicate the circle region belongs to the background areas and constrain the regularization of LSTM cells (cf. Fig. 7(b)). This demonstrates to some extent that the cell states of depth blocks capture the global scene context to help cost map regularization.