Self-Supervised Light Field Depth Estimation Using Epipolar Plane Images

Depending on the types and depth clues of the input data, current methods of light field depth estimation can be divided into sub-aperture image-based, EPI-based and refocusing cues-based depth estimation. Traditional methods  [18, 10, 19, 11, 12]focus on the structural characteristics of the light field and obtain the depth map from the hand-crafted features. These methods are generally accompanied by complex calculation and subsequent optimization, and need to be manually tuned for different scenarios. Recently, by building an end-to-end network, deep learning technology has made a breakthrough in the application of pixel level light field, such as depth estimation, saliency detection [20], super-resolution reconstruction [21] and so forth.

Methods based on Sub-aperture Images. Inspired by the multi-view and stereo algorithms, Jeon et al [22, 18] use the sub-aperture images to estimate the multi-view stereo correspondences with a sub-pixel accuracy shift. These methods show good performance in the real scene captured by the lenslet-based camera, but suffer from noise at the occlusion boundary. Considering the narrow baseline between sub-aperture images, some methods [4, 5, 17] use it as the input of CNN to estimate the disparity of the central view. On the basis of U-net [23], Heber et al [4] propose the concept of EPI-Volume, using stacked sub-aperture images as input, which are located in the same horizontal or vertical perspective. Shin et al [17] extract four directional sub-aperture images in the epipolar plane and stacked them into the network to achieve the state-of-the-art performance. Using the input method in [17], Leistner et al [5] shift the stack of sub-apertures to make the network better adapt to the small- and wide- baseline light field. However, it is difficult to estimate the detail features in scenarios with this method.

Methods based on EPI. Previous EPI-based depth estimation methods [11, 24, 25, 26] prefer to design complex operation algorithms to calculate slope according to the structure of epipolar line. Although high estimation accuracy can be obtained, especially for occlusions, hand-crafted parameters and optimization are unavoidable, like most traditional methods. To overcome this, recent works [27, 3, 28, 29, 6, 16, 30, 31] have treated EPI as the input of CNNs for end-to-end estimation. Luo et al [16] designed a two-stream CNNs to estimate the depth of a single pixel from each patch. Zhou et al [31] explore the estimation effect of different scale EPI patches and fused multi-directional features. In  [6], Li et alpropose a relation network to extract orientation relation features between EPI center view and neighboring pixels, which can further improve the estimation accuracy. However, the current CNN-based method uses a single spatial dimension of EPI image and only one common pixel point in different directions, which makes it sensitive to noise.

Methods based on Refocusing cues. Light field refocusing cues are widely used in traditional depth estimation methods. Most researches focus on exploiting different depth features, such as defocus, correspondence, shade and occlusion. Tao et al [9, 10, 32] propose a depth estimation that combines defocus and correspondence features. Based on their works, Wang et al [19] create occlusion-aware depth-maps via a modified angular photo-consistency. Lin et al [13] synthesize the symmetry of focal stack and data consistency measure, which is more robust to noise and under-sampling. William et al [12] also propose a data cost based on an angular entropy metric and adaptive defocus responses. These methods, which utilize the unique refocusing capability of the light field, can be adapted to complex and diverse real scenes, but also introduce a large number of hand-crafted features that are more complex. Current methods that combine refocusing with CNNs have rarely been studied, and the existing works usually only use the focused central view image as input, failing to learn the data changes before and after refocusing. For example, Zhou et al [33] estimate the disparity label of each pixel from a set of focal stack with the continuous disparity range.

Overall, compared with traditional depth estimation methods, CNN-based ones can significantly improve the estimation accuracy. However, refocused 2D slices (sub-aperture images or EPIs) of light field data in different depth planes provide abundant refocusing cues, which cannot be effectively used by the existing CNN-based methods. This also makes it perform poorly in complex scenes, especially real scenes. In order to effectively use the light field focusing clue, in this paper we focus on building a unified training framework to fuse CNN with refocusing.

In recent years, self-supervised learning has attracted extensive attention because it provides an alternative method to label existing datasets. By constructing auxiliary tasks, self-supervised learning extracts corresponding supervised information from a large number of unlabeled data. This enables better training of deep neural networks. Liu et al [14, 15] propose image ranking as an auxiliary task for some regression problems. Through the training of this task, the network can learn different semantic features. Then the network parameters are transferred to the original task for fine-tuning, which allows for deeper and wider training of the network. Other self-supervised tasks include predicting the relative position [34] of patches in images, restoring the entire image from surrounding pixels [35], and generating color images from gray-scale images [36]. In the field of monocular depth estimation, stereo matching clues are usually used to implement self-supervised training of network according to scene disparity and view position [37, 38, 39, 40]. This shows high estimation accuracy in both image and video sequences.

Inspired by these works, we propose a self-supervised depth estimation framework based on light field refocusing, which can be considered as the first work of light field depth estimation exploiting self-supervised learning. Different from the task of depth regression, this paper estimates the disparity shifts of scenes before and after focusing to get the refocusing cues, so as to learn the effective feature representation of 2D slices of light field data. Our approach not only improves network training, but also enables the network to better adapt to complex scenarios.

The EPI is the 2D slice from 4D light field. We use $(u,v,x,y)$ to represent the 4D light field coordinates shown in Figure 3, where $(u,v)$ represents the directional dimension and $(x,y)$ represents the spatial dimension. Here, by fixing $v$ and $y$, or $u$ and $x$, respectively, the EPI in the horizontal or vertical direction can be obtained. Due to the limitation of a single space dimension, it is easily affected by the local spatial position. Therefore, extracting depth from EPI is easily disturbed by noise in the scene, and it is difficult to extract effective linear structure features [27]. Existing methods generally use different directions of EPI as input to improve the robustness of depth estimation. However, due to the limitation of spatial dimension, EPI in different directions only has a single common spatial pixel point, which not only reduces the estimation efficiency (EPI in different directions can only estimate the depth of the central pixel point), but also makes it difficult for different directions of input to effectively complement each other, resulting in redundant information. Moreover, the estimation accuracy of noise scenes cannot be improved effectively. To solve these problems, we propose to increase the common spatial location points of EPI in different directions. Here we use the classical horizontal and vertical input modes. No matter horizontal or vertical EPI, there is only a sample of a single spatial location in its perspective dimension, so EPI in different directions has only a single common intersection. Therefore, our method is to increase the spatial resolution of horizontal and vertical EPI by increasing the sampling of spatial pixel points in the perspective dimension, so that the common area of EPI in both directions is maximized. As shown in Figure 3, in order to increase the spatial location points common to EPIs in different directions, EPIs with different spatial sampling points are stacked in the directional dimension, that is, from $(u,x)$ to $(u\cdot y,x)$, and from $(v,y)$ to $(v\cdot x,y)$. This representation is called EPI-Stack in this paper. In this way, we improve the spatial resolution of the polar plane image, and it provides more common space locations for the horizontal and vertical EPI in the perspective dimension, which increases the spatial constraint of EPI and is robust to noise.

For this stacked EPI image, the corresponding network structure is designed for feature extraction. As shown in Figure 4, the stacked EPI images in both horizontal and vertical directions are input into the Siamese network, where ’Conv-ReLU-Conv’ is used as the basic module for feature extraction. Since the spatial location sampling points of EPI-Stack are the same in both horizontal and vertical directions, the weight of the two-branch network is shared here. Compared with traditional EPI, the size of EPI-Stack proposed in this paper is too large. In order to quickly extract the spatial structure of EPI and learn the subtle changes of polar slope, we add feature extraction modules with different scales at the front of the network, using convolution layers with dimensions of $5\times 5$ and $3\times 3$ respectively, and set the convolution stride of stacked perspective dimension to 3 and the spatial dimension to 1. Several residual modules are used to extract the deep structure features of the scene, and then the disparity features of the scene are obtained by concatenating the outputs of the two-branch network. Here, except for the first basic module, all are convolution filters with size of $2\times 2$ and stride 1.

According to the self-supervised learning method in Section III, we use the Siamese network based on EPI-Stack to estimate the disparity values of the scene before and after focusing and compute the disparity shifts. Using the loss function in Eq. 2, we can achieve the self-supervised training of the network. Then we use mean absolute error as the loss function to fine tune the Siamese network, as shown in Eq. 3.

Given $M$ EPI-Stack pairs in mini-batch, we denote the ground truth disparity of the i-th image as $y_{i}$, and the predicted value from the network is ${{\hat{y}}_{i}}$. The above networks are trained on the 4D HCI dataset  [7], which provides 24 well-designed scenarios and corresponding disparity maps for quantitative evaluation. Each scene has $512\times 512$ spatial resolutions and $9\times 9$ angular resolutions. In this experiment, 16 scenarios were used for training and the rest for testing. Considering the limited number of training samples, we randomly extracted EPI-Stack samples of size $(9\cdot 25)\times 25\times 3$ from Horizontal and vertical views for training. We still use $(9\cdot 512)\times 512\times 3$ as the input size for testing. In this way, we not only get enough samples for training, but also use the whole scene for testing, which significantly improves the estimation efficiency compared to the existing EPI-based methods. In addition, we use the Keras framework to implement the network with a Nvidia 1080 Ti GPU. We set the batch size to 80, use the RMSprop optimizer, and set the weight decay rate to $10^{-5}$. Note that the proposed network is end-to-end trained and does not use subsequent complex processing measures.

We use the bad pixel ratio (BadPix), which denotes the percentage of pixels with 0.07 error value, the Mean Square Error (MSE), and Peak Signal to Noise Ratio (PSNR(dB)) to evaluate the performance of our approach. For the predicted disparity map $d$, the ground truth $gt$, and the scene region $N$, its BadPix is defined as,

and MSE is defined as,

Lower scores are better for BadPix and MSE.

To evaluate the effectiveness of our approach in real-world scenarios, this paper conducts a qualitative experimental analysis using the Stanford Light Field Archive real-world dataset [41]. These data are captured with the Lytro Illum handheld camera and contain a variety of complex real scenes, which can be used to better evaluate the performance of depth estimation.

The effect of self-supervised learning on depth estimation. We illustrate the validation of the proposed self-supervised depth estimation by comparing the loss curves. From the validation of the loss curve in Fig 5, it can be seen that the self-supervised depth estimation method can accelerate the convergence of the original network compared with the randomly initialized network model. At the same time, we visualize the network convolution kernel before and after self-supervised learning, which is illustrated with a single EPI. Compared with the randomly initialized model (Fig 6(a), the self-supervised depth estimation method can compute the disparity shift according to the slight change of EPI slope before and after refocusing, which enables the network filter to learn a more efficient linear representation of the EPI (Fig 6(b)). These visual filters are extracted from the last layer of a single network branch.

We use the self-supervised method to estimate the disparity map in Stanford real scene. In the experiment, we select some complex occlusion scenes for qualitative comparison. As shown in Figure 7, it can be found from the first row that compared with random initialization, the self-supervised method can effectively estimate the depth discontinuities and obtain leaves with clear boundaries. Similarly, in the complex scene of the second row, our method is robust to the interference of complex background and recovers the branches in the foreground. The last row is a challenging scene with multiple occlusion. The wire meshes in the fore- and back-ground overlap each other and contain a lot of details at the intersection of the meshes. We can find that before using self-monitoring training, our network can recover the scene contour, but in the detail region, such as the region marked by yellow line, the fore- and back-ground interfere with each other. So it is difficult to extract effective depth features. After training with self-supervised learning, the details of the scene are restored. In addition, in the region marked by red line, our approach not only restores foreground information, but also enables more accurate estimates of backgrounds at different depths.

The effect of EPI-Stack input mode. To verify the advantage of EPI-Stack input mode in noisy scenes, we add extra Gaussian noise to the original scene and compare it with the disparity before adding noise. Here we compare the recent method EPI_ORM [6] based on traditional EPI input mode, which achieves the state-of-the-art in multiple scenarios through the relation network. In this experiment, to compare the effect of different input modes on performance, we implement end-to-end training on the original network without self-supervised training. By comparing the estimation results of two input modes in Fig 8, we find that the traditional EPI input mode can achieve high estimation accuracy in the original scene. However the disparity map estimated after adding noise is accompanied by obvious blurring, which is also the shortage of the current traditional EPI input mode. By contrast, our proposed approach with EPI-Stack still yields clear results. This is because the EPI-Stack input method increases the sampling of spatial pixel points, which enables EPI in different directions to have larger public areas and introduces stronger spatial constraints. Therefore, this input mode can improve the robustness of depth estimation for noise scenes.

We compare our approach with state-of-the-art methods: CAE [12], SPO [25], EPN [16], EPINET [17], EPI-Shift [5], and EPI_ORM [6]. We use the training scene (boxes, cotton, dino, and sideboard) from 4D HCI Benchmark for quantitative comparison. Table I shows quantitative results for these state-of-the-art methods averaged on synthetic scenes. It can be seen that our method achieves second best result with BadPix metric and the best performance with MSE and PSNR metrics. This is because our method can not only get effective refocusing cues, but also this special EPI-Stack input mode increases the number of common spatial sampling points, improves the robustness of depth estimation, and makes the disparity map smoother. In addition, we provide a comparison of average runtimes and an indication of whether the GPU implementation is used. Compared with other methods, our approach achieves higher estimation accuracy and efficiency.

Figure 9 provides a qualitative comparison of the current method on Stanford light field data. Note that we extract the light field data of $9\times 9$ viewing angle from the real data to test. We set the parameter label of the traditional method CAE [12] to 75. The real scene parallax range is [-1, 1] by default. Qualitative comparison shows that the traditional CAE method is susceptible to noise interference in the real scene, which reduces the estimation accuracy. This method also takes a long time to compute. Methods EPINET [17] and EPI-Shift [5] both use sub-aperture images as input. Although the scene contour is recovered, the local details are not well estimated and are obviously blurred. EPI_ORM [6] method uses the traditional EPI input mode and restores the details of the scene better. However, due to the limitations of synthetic scene training, this method does not adapt to complex disparity changes in real scenes. Therefore the occlusion boundaries of scenes cannot be restored in Figure 9. Our method is not trained in a real scene and still obtains high disparity map. Through self-supervised learning based on refocusing, our approach extends the disparity range of EPI, extracts effective refocusing cues, and achieves higher estimation accuracy. Especially in discontinuous and multi-occluded regions, our method obtains clear contours and preserves the foreground and background boundaries.