View-consistent 4D Light Field
Depth Estimation

To begin, we estimate depth labels at all edges in the light field using the EPI edge detection algorithm proposed by Khan et al. [Khan et al.(2019)Khan, Zhang, Kasser, Stone, Kim, and Tompkin]. This filters each EPI with a set of large Prewitt filters, and then estimates a representation of each EPI edge as a parametric line. Lines in EPI space correspond to scene points, with the slope of the line being proportional to the depth of the point. Therefore, this line representation captures both position and disparity information of scene edges.

However, the parametric definition implies the line exists in all views, and, hence, the representation does not contain any visibility information for the point across light field views. We can address this by looking at local structure around the edge for point samples along the line: the point is visible if the local gradient matches the global line direction. We define a gradient-alignment-based visibility score for each sample, then threshold this score to decide which part of the line is actually visible in a given view [Khan et al.(2020)Khan, Kim, and Tompkin].

Given an EPI line $l$, we sample it at $n$ locations to obtain a set of point samples $S_{l}=\{x_{i},y_{i}\}$ on the EPI $I$. Each sample $s_{i}$ corresponds to a projection of the original point in light field view $i$, whose visibility can be determined as:

$\displaystyle v_{l}(i)=\mathbbm{1}\left(\frac{\nabla I(s_{i})(\nabla l)^{T}}{\lVert\nabla I(s_{i})\rVert\lVert\nabla l\rVert}>cos(\tau_{v})\right),$

(1)

where $\mathbbm{1}$ denotes the characteristic function of a set beyond the visibility threshold, $\nabla I$ is the image gradient, $\nabla l$ is the direction perpendicular to the line $l$, and $\tau=\pi/13$.

The characteristic function was proposed by [Khan et al.(2020)Khan, Kim, and Tompkin] to generate a central view disparity map, and so only central view points were kept. By extending this idea to the entire light field, we retain all points along with their depth and visibility information, and use them to guide the disparity propagation into disoccluded and occluded regions. Since disparity directly at edges can often be ambiguous due to image resolution limits, the points are offset along the image gradient so that they lie on an actual surface. This is completed using the two-way propagation scheme of [Khan et al.(2020)Khan, Kim, and Tompkin], which also allows a disparity map for the central view to be generated using dense diffusion of the sparse point depths.

The EPI line-fitting algorithm works on EPIs in the central cross-hair views—that is, the central row and column of light field images. While it is possible to run it on other rows and columns, this can become expensive, and the central set is usually sufficient to detect visible surfaces in the light field [Wanner and Goldluecke(2013)]. Hence, we project the estimated disparity map from the center view into all views along the cross-hair. Since gradients at depth edges in the estimated disparity map are not completely sharp, this leads to some edges being projected onto multiple pixels in the target view. We deal with this by sharpening the edges of the disparity map before projection, as shown in Shih et al. [Shih et al.(2020)Shih, Su, Kopf, and Huang], using a weighted median filter [Ma et al.(2013)Ma, He, Wei, Sun, and Wu] with parameters $r=7$ and $\epsilon=10^{-6}$. Omitting this step can cause inaccurate estimates around strong depth edges. The result is not very sensitive to parameters $r$ and $\epsilon$ since most parameter settings will target the error-prone strong edges.

After depth reprojection, we must deal with the two problems highlighted in the overview: inpainting holes, and accounting for occluding surfaces in off-center sub-aperture views. We tackle this by using the edges from Section 3 to guide a dense diffusion process. Moreover, we ensure view consistency by performing diffusion in EPI space.

The EPI lines from the first stage constitute a set $L$ of cross-view edge features. $L$ is robust to occlusions in a single view as it exists in EPI space. As such, $L$ provides occlusion-aware sparse depth labels to guide dense diffusion in EPI space. Diffusion in EPI space has the added advantage of ensuring view consistency.

Let $D_{o}$ represent an angular slice of the disparity maps with values reprojected from the center view and with propagation guides (Figure 1). Then, we formulate diffusion as a constrained quadratic optimization problem:

$\displaystyle\hat{D}=\underset{D}{\mathrm{argmin}}\sum_{p\in D}E_{d}(p)+\sum_{(p,q)\in\mathcal{S}}E_{s}(p,q),$

(2)

where $\hat{D}$ is the optimal depth labeling of the EPI, and $\mathcal{S}$ is the set of four-connected neighboring pixels. The data $E_{d}(p)$ and smoothness terms $E_{s}(p,q)$ are defined as:

	$\displaystyle E_{d}(p)$	$\displaystyle=\lambda_{d}(p)\big{\lVert}D(p)-D_{o}(p)\big{\rVert}_{2}^{2},$		(3)
	$\displaystyle E_{s}(p,q)$	$\displaystyle=\lambda_{s}(p,q)\big{\lVert}D(p)-D(q)\big{\lVert}_{2}^{2},$		(4)

We take the weight for the smoothness term from the EPI intensity image $I$:

$\displaystyle\lambda_{s}(p,q)=\frac{c}{\lVert\nabla I(p)\rVert+\epsilon},$

(5)

where $c=0.1$. We define the weight for the data term as:

$\displaystyle\lambda_{d}(p)=\left\{\begin{array}[]{ll}15&\mbox{if $p\in\mathcal{C}$,}\\ \omega_{e}(p)&\mbox{if $p\in\mathcal{L}$},\\ 0&\mbox{otherwise,}\end{array}\right.$

(9)

where $\omega_{e}(p)$ is the edge-importance weight proposed by [Khan et al.(2020)Khan, Kim, and Tompkin], and $\mathcal{C}$ and $\mathcal{L}$ are the set of pixels coming from the reprojected center view disparity map and EPI line guides, respectively.

Equation (2) defines the optimal disparity map $\hat{D}$ as one that minimizes divergence from the labeled data (Eq. (3)) while being as smooth as possible. Equation (4) measures smoothness as the similarity between disparities of neighboring pixels. We wish to relax the smoothness constraint for edges, so smoothness weight is chosen as the inverse of the image gradient (Eq. (5)). This allows pixels across edges to have a disparity difference without being penalized. The data weight (Eq. (9)) is determined empirically and works for all datasets.

Optimizing Equation (2) is a standard Poisson optimization problem. We solve this using the Locally Adaptive Hierarchical Basis Preconditioning Conjugate Gradient (LAHBPCG) solver [Szeliski(2006)] by posing the data and smoothness constraints in the gradient domain [Bhat et al.(2009)Bhat, Zitnick, Cohen, and Curless].

We now have view-consistent disparity estimates for every pixel in the central cross-hair of light field views: $(u_{c},\cdot)$, and $(\cdot,v_{c})$. As noted, this set is usually large enough to cover every visible surface in the scene. Hence, all target views $(u_{i},v_{i})$ outside the cross-hair can be simply computed as the mean of the reprojection of the closest horizontal and vertical cross-hair view ($(u_{c},v_{i})$ and $(u_{i},v_{c})$, respectively).

We compare our results to the state-of-the-art depth estimation methods of Jiang et al. [Jiang et al.(2018)Jiang, Le Pendu, and Guillemot] and Shi et al. [Shi et al.(2019)Shi, Jiang, and Guillemot]. Both methods use the deep-learning-based Flownet2.0 [Ilg et al.(2017)Ilg, Mayer, Saikia, Keuper, Dosovitskiy, and Brox] network to estimate optical flow between the four corner views of a light field, then use the result to warp a set of anchor views. In addition, Shi et al. further refine the edges of their depth maps using a second neural network trained on synthetic light fields. While Shi et al.’s method generates high-quality depth maps for each sub-aperture view, they do not have any explicit cross-view consistency constraint (unlike Jiang et al.).

For our evaluation, we used both synthetic and real world light fields with a variety of disparity ranges. For the synthetic light fields, we used the HCI Light Field Benchmark Dataset [Honauer et al.(2016)Honauer, Johannsen, Kondermann, and Goldluecke]. This dataset consists of a set of four 9 $\times$ 9, 512 $\times$ 512 pixels light fields: Dino, Sideboard, Cotton, and Boxes. Each has a high-resolution ground-truth disparity map for the central view only. As such, we use this dataset to evaluate the accuracy of depth maps generated by our method and the baseline methods.

For real-world light field data, we use the EPFL MMSPG Light-Field Image Dataset [Rerabek and Ebrahimi(2016)] and the New Stanford Light Field Archive [sta(2008)]. The EPFL light fields are captured with a Lytro Illum and consist of $15\times 15$ views of $434\times 625$ pixels each. However, as the edge views tend to be noisy, we only use the central $7\times 7$ views in our experiments. We show results for the Bikes and Sphynx scenes. The light fields in the Stanford Archive are captured with a moving camera and have a larger baseline than the Lytro and synthetic scenes. Each scene consists of $17\times 17$ views with varying spatial resolution. We use all views from the Lego and Bunny scenes, scaled down to a spatial resolution of $512\times 512$ pixels.

We evaluate both the accuracy and consistency of the depth maps. For accuracy, we use the mean-squared error (MSE) multiplied by a hundred, and the percentage of bad pixels. The latter metric represents the percentage of pixels with an error above a certain threshold. For our experiments we use the error thresholds 0.01, 0.03, and 0.07. The unavailability of ground truth depth maps for the EPFL and Stanford light fields prevents us from presenting accuracy metrics for the real world light fields.

To evaluate view consistency, we reproject the depth maps onto a reference view and compute the variance. Let $\rho_{0},\rho_{1},\dots,\rho_{n}$ represent the depth maps for the $n$ light field views warped onto a target view $(u,v)$. The view consistency at pixel $s$ in view $(u,v)$ is given by:

$\displaystyle\mathcal{C}_{(u,v)}(s)$

$\displaystyle=\frac{1}{n}\sum^{n}_{i=0}(\rho_{i}(s)-\mu_{s})^{2},\hskip 28.45274pt\textrm{where }\mu_{s}=\frac{1}{n}\sum^{n}_{i=0}\rho_{i}(s),$

(10)

and overall light field consistency is given as the mean over all pixels $s$ in the target view $S$:

$\displaystyle\mathcal{C}_{(u,v)}=\frac{1}{S}\sum^{S}_{s=0}\mathcal{C}_{(u,v)}(s).$

(11)

This formulation allows for the consistency to be evaluated quantitatively for both the synthetic and real world light fields.

We also compare computational running time. Our method is implemented in MATLAB except for the C++ Poisson solver. Both baseline methods use the authors’ implementations. The learning-based components of both baselines uses TensorFlow, with other components of Jiang et al. in MATLAB. All CPU code ran on an AMD Ryzen ThreadRipper 2950X 16-Core Processor, and GPU code ran on an NVIDIA GeForce RTX 2080Ti.