Dynamic Fusion Network For Light Field Depth Estimation

There is a wide range of methods for light field depth estimation. The field can be roughly divided into methods based on EPI analysis, multi-view stereo matching based approaches and focus-based approaches. [40] proposed a 4D light-field depth estimation method via epipolar plane image analysis and locally linear embedding. [15] employed a sparse decomposition design to leverage the depth-orientation relationship on its epipolar plane images. Another popular approach is using total variation regularization. [21] proposed a nonlinear Total Variation (TV) based method for recovering 3D shape of an object by diffusing several initial depth maps obtained through different focus measures. [24] proposed an non-convex minimization scheme to determine depth maps based on prior knowledge. [13] applied a preconditioned alternating direction method of multipliers (PADMM) with a new cost function to generate a noise-free depth map. Jeon, et al. proposed an algorithm to estimate the multi-view stereo correspondences with sub-pixel accuracy using the cost volume [14]. [35] developed an occlusion-aware depth estimation algorithm to deal with occlusion edges.

Although the above traditional methods can produce good depth maps, these methods require presetting for the depth estimation with careful parameter tuning. Furthermore, they tend to be difficult to generalize into all scenes.

Recently, deep learning, in particular Convolutional Neural Networks (CNNs), has successfully broken the bottleneck of traditional methods in a wide range of fields. Such as, super-resolution [39], novel view generation [16] and material recognition [36]. For the depth estimation, [30] introduced a fully-convolutional neural network for highly accurate depth estimation. [1] predicted depth from a single focal slice using deep neural networks by exploiting dense overlapping patches. [11] suggested the first end-to-end learning method to compute depth maps from the focal stack. [31] estimated scene depths from a single image, using the information provided by a camera’s aperture as supervision. They introduced two differentiable aperture rendering functions that use the input image and predicted depths to simulate the depth-of-field effects caused by real camera apertures. Then they trained a monocular depth estimation network end-to-end to conduct depth estimation from RGB images.

These methods demonstrate that focusness cues can greatly contribute to depth estimation, achieving superior results. However, there is still large room for further improvement in terms of spatial correlation excavation in the focal stack and multi-modal fusion between the RGB image and the focal stack. In this paper, we propose a deep learning based method that dynamically incorporates a RGB and a focal stack to confront these challenges.

In this paper, our goal is to excavate the spatial correlation among focal slices and dynamically fuse RGB information and focusness features for focus-based depth estimation. The overall framework is described in Figure 1. First, the RGB image is fed into the RGB stream which consists of a SeNet-154 network [12] pretrained on ImageNet [7], a decoder employs four upsampling layer to gradually up-scale the final feature from the encoder and a refinement module concatenate the feature from decoder and encoder in color channels with three convolutional layers. Second, all focal slices are fed into another focal stream to generate focusness information from focal slices, relying on the proposed spatial-correlation perception modelling (SCPM) designed to excavate the spatial correlation in focal slices. Last, we propose a multi-modal dynamic fusion module (MDFM) to dynamically fuse multi-modal information between RGB data and the focal stack in an adaptive way for providing good depth map. The details of our two modules will be discussed in the following two sections.

Considering the stack of focal slices in a scene possess focus-area of multiple scales and focused at different depth, we aim to correlate the depth information with multi-scale focusness features. To do this, we propose a spatial-correlation perception module (SCPM), in which the spatial correlation between different focal slices is excavated by a proposed pyramid ConvGRU. In this way, multi-scale focusness information in different slices can be transfered along the depth direction.

Specifically, we first feed 12 focal slices $I_{1}(x,y,s,t)$, $I_{2}(x,y,s,t)$…$I_{n}(x,y,s,t)$ to four 5$\times$5 convolutional layers to encode focusness features $x_{i}$. This procedure can be defined as:

$$\begin{array}[]{l}f_{i}\left({I_{i}(x,y,s,t);\theta_{i}}\right)\to x_{i},\end{array}$$

(1)

where $s\times t$ indicates the angular resolution, $x\times y$ indicates the spatial resolution and $i$ represents the i-th focus slice. $\theta$ denotes the parameters of the encoder layer and $f_{i}$ is a learning mapping function. Then, in order to excavate the spatial correlation between different focal slices, we draw ideas from recent works in [8, 38, 43]. Concretely, they use a typical ConvGRU to capture short and long term temporal dependencies. In our work, we consider the focusness features as a feature map sequence and design a pyramid ConvGRU. In order to pass multi-scale focusness features corresponding to focus area of multiple scales, the proposed pyramid ConvGRU use an atrous spatial pyramid pooling (ASPP) module [5] for each of the gates instead of convolution, which can encode multi-scale foucsness information by applying atrous convolution at multiple parallel filters with different rates and field-of-views. The dilation rates are 1, 3 and 5, respectively. The formulation of the proposed pyramid ConvGRU is:

$$r_{i}=\sigma\left[\begin{array}[]{l}W_{r1}*[x_{i},h_{i-1}],\\ W_{r3}*[x_{i},h_{i-1}],\\ W_{r5}*[x_{i},h_{i-1}]\\ \end{array}\right],$$

(2)

$$z_{i}=\sigma\left[\begin{array}[]{l}W_{z1}*[x_{i},h_{i-1}],\\ W_{z3}*[x_{i},h_{i-1}],\\ W_{z5}*[x_{i},h_{i-1}]\\ \end{array}\right],$$

(3)

$$\begin{array}[]{l}n_{i}=\tanh(x_{i}*W_{xn}+r_{i}\odot h_{i-1}*W_{hn}+b_{n}),\end{array}$$

(4)

$$\begin{array}[]{l}h_{i}=\left({1-z_{i}}\right)\odot h_{i-1}+z_{i}\odot n_{i},\end{array}$$

(5)

where all $W_{*}$ and $b_{*}$ are model parameters to be learned. $\sigma$ is sigmoid function. $\odot$ and $\ast$ are element-wise multiplication and convolution, respectively. The pyramid ConvGRU takes the i-th focusness feature $x_{i}$ and a previous output $h_{i-1}$ as input, outputs a new feature $h_{i}$ by combining $h_{i-1}$ and a candidate state $n_{i}$ weighted by an output of an update gate $z_{i}$. The update and reset gates, $z_{i}$ and $r_{i}$, selectively update multi-scale focusness information from the input focusness feature $x_{i}$ and the previous output $h_{i-1}$, automatically learn the spatial correlation from neighboring slices. Note that the updated focusness feature $h_{i}$ is not disappeared with information transfering, but passed to the next slice.

As RGB images and focus slices imply different depth information, we consider that fusing different multi-modal information is important for depth estimation task. However, existing fusion schemes usually employ some manually set, including sum fusion, weighted fusion, concatenate fusion. These static fusion schemes not suitable for our task because the RGB and focusness feature are not equivalent quantities, is prone to information loss. As shown in the Fig.4(a), these static fusion methods processes the entire image. When the network parameters are fixed, the convolution kernel does not change with the input features, ignoring the relationship between multi-modal features. Therefore, a more proper and effective strategy should be considered. To this end, we introduce a multi-modal dynamic fusion module (MDFM) to dynamically fuse the RGB features and focusness features in an adaptive manner. In this module, the filter varies with focusness features is used to convolve with RGB information , thereby avoiding information loss.

Specifically, the process of our module consists of two steps: 1) we choose a adaptive kernel [33] instead of the spatially invariant convolution. In our work, the standard spatial convolution W is adapted at each pixel using focuesness features f via the adaptive kernel. Therefore, when the focusness features change, the parameters of the convolution kernel also change; 2) we apply the generated adaptive convolution kernel to RGB features f, making the whole network dynamically fuse multi-modal information to get an accurate depth map.

For the output from the spatial-correlation perception module h, we have:

$$\begin{array}[]{l}d_{i}=\sum\limits_{j\in\Omega(i)}{\exp(-\frac{1}{2}(h_{i}-h_{j})}^{T}(h_{i}-h_{j}))W\left[{p_{i}-p_{j}}\right]f_{j}+b\end{array}$$

(6)

where $[pi-pj]$ is index of the spatial dimensions of an array with 2D spatial offsets, i and j represent the pixel coordinates, $W$ is a standard spatial convolution W, $f_{j}$ represents the output from the RGB stream, $b$ is bias term, $d$ represents the final depth prediction map. Before we perform filtering operation, the filter has a pre-defined form and depend on the content of focusness features. It is hoped that the prediction map depends on both the RGB features and the reliable focusness information. Finally, we refine the depth map by adding two consecutive convolutional layers. As shown in the Fig.4(b), in test stage, compared to traditional methods of fusing multi-modal information which parameters do not change with input features, our model dynamically selects filter parameters for convolution based on the content of the focusness features and adaptively selects multimodal features for fusion, avoiding information loss.

In this section, we calculate the reconstruction error between the predicted images and the ground truth. We define the loss by:

$$\begin{array}[]{l}L=l_{depth}+\lambda l_{grad}+\mu l_{normal}\end{array}$$

(7)

where $\lambda$, $\mu$ are weighting coefficients. Here we set as $\lambda=1$, $\mu=1$. We use a logarithm of depth errors as $l_{depth}$ to minimize the difference between the depth estimate map $d_{i}$ and its ground truth $g_{i}$:

$$\begin{array}[]{l}l_{depth}=\frac{1}{n}\sum\limits_{i=1}^{n}{\ln\left({\left\|{d_{i}-g_{i}}\right\|_{1}+\alpha}\right)}\end{array}$$

(8)

where $\alpha(>0)$ is a parameter we set $\alpha=0.5$. In order to deal with edge distortion problems caused by CNNs training, we consider the following loss function of the gradients of depth:

$$\begin{array}[]{l}l_{grad}=\frac{1}{n}\sum\limits_{i=1}^{n}{\left({F\left({\nabla_{x}\left({\left\|{d_{i}-g_{i}}\right\|_{1}}\right)}\right)+F\left({\nabla_{y}\left({\left\|{d_{i}-g_{i}}\right\|_{1}}\right)}\right)}\right)}\end{array}$$

(9)

where $\bigtriangledown x(*)$ is the spatial derivative of $\|d_{i}-g_{i}\|$ computed at the ith pixel with respect to $x$, and so on. To deal with such small depth structures and further improve fine details of depth maps, we consider yet another loss for training, which measures accuracy of the normal to the surface of an estimated depth map with respect to its ground truth:

$$\begin{array}[]{l}l_{normal}=\frac{1}{n}\sum\limits_{i=1}^{n}{\left({1-\frac{{\left\langle{n_{i}^{d},n_{i}^{g}}\right\rangle}}{{\sqrt{\left\langle{n_{i}^{d},n_{i}^{d}}\right\rangle},\sqrt{\left\langle{n_{i}^{g},n_{i}^{g}}\right\rangle}}}}\right)}\end{array}$$

(10)

where $<\textperiodcentered,\textperiodcentered>$ denotes the inner product of vectors. Denoting the surface normal of an estimated depth map and its ground truth by $n_{i}^{d}=\left[{-\nabla_{x}\left({d_{{}_{i}}}\right),{\rm{}}-\nabla_{y}\left({d_{{}_{i}}}\right),1}\right]^{{}^{T}}$, $n_{i}^{g}=\left[{-\nabla_{x}\left({g_{{}_{i}}}\right),{\rm{}}-\nabla_{y}\left({g_{{}_{i}}}\right),1}\right]^{{}^{T}}$

We compare our method with 6 other state-of-the-arts light field depth estimation method, containing both deep-learning-based methods and traditional methods (marked with $*$). Deep-learning-based method is DDFF [11], EPINET [30], traditional methods are VDFF^∗ [24], PADMM^∗ [13], LF^∗[14], LF$\_$OCC^∗ [35]. For fair comparisons, the results from competing methods are generated by authorized codes. To verify the generalization and applicabilityof our network, we conduct experiments on three dataset.

The common consumer level cameras also can capture focal stack. To expand the practical application of our network, we test our network on the focal stack captured captured by a Samsung Galaxy S3 phone, provided by the authors of [32]. For ease of testing, we choose 12 focal slices and a RGB image from each scene. Figure 9 provides a qualitative comparison of different deep learning-based methods. It can be seen that the proposed method allows to recover high quality depth maps with less noise and sharper object boundaries, although the model was not trained on this specific dataset. The impressive results show that the proposed method can be used for further applications in our daily life. Further, our model is not limited by the light field camera because the smart-phone camera can capture focal stacks by manipulating the focus distance programmatically and it represents the dominant modality of image capture.