Learning Depth from Focus in the Wild

Field of view. Scene FoVs are calculated by work distances, focus distances, and the focal length of cameras in Eq.(1). Since the work distances are fixed during a focal sweep, relative values of FoVs (Relative FoVs) are the same as the inverse distance between sensor and lens. We thus perform an initial alignment of a focal stack using these relative FoVs as follows:

where $s_{n}$ is the distance between the lens and the sensor in an $n$-$th$ focal slice. $A$ is the sensor size, and $W$ is the working distance. $f$ and $F_{n}$ are the focal length of the lens and a focus distance, respectively. $min$ denotes an index of a focal slice whose FoV is the smallest among focal slices. In this paper, we call this focal slice with the $min$ index as the target focal slice. We note that the values are available by accessing the metadata information in cameras without any user calibration.

Alignment network. As shown in Fig.3, our alignment network has 3-level encoder-decoder structures, similar to the previous optical flow network [15]. The encoder extracts multi-scale features, and multi-scale optical flow volumes are constructed by concatenating the features of a reference and a target focal slice. The decoder refines the multi-scale optical flow volumes in a coarse-to-fine manner using feature warping (F-warp). However, we cannot directly use the existing optical flow framework for alignment because defocus blur breaks the brightness constancy assumption [34].

Simulator. Because public datasets do not describe changes in FoVs or hardware limitations in off-the-shelf cameras, we propose a useful simulator to render realistic sequential defocus images for training our alignment network. Here, the most important part is to determine the error ranges of the intrinsic camera parameters, such as principal points and focal distances. We estimate them as the following process in Fig.4: (1) We capture circle patterns on a flat surface using various smartphone models by changing focus distances. (2) We initially align focal stacks with the recorded focus distances. (3) After the initial alignment, we decompose the residual motions of the captured circles using 3 basis vectors, image crop and $x-$ and $y-$axis translations. (4) We statistically calculate the error ranges of the principal points and focus distances from the three parameters of the basis vectors. Given metadata of cameras used, our simulator renders focal stacks induced from blur scales based on the focus distance and the error ranges of the basis vector.

Initial feature extraction. In an initial feature extraction step, we utilize a dilated convolution to extract focus features. After the dilated convolution, we extract feature pyramids to refine the focal volumes in the refinement step. Given an input focal stack $S\in R^{H*W*N*3}$ where $H$, $W$ and $N$ denote the height, width and the number of focal slices respectively, we extract three pyramidal feature volumes whose size is $H/2^{L}\times W/2^{L}\times N\times C*2^{L}$ where $L\in\{0,1,2\}$ and $C$ is the number of channels in the focal volume. This pyramidal feature extraction consists of three structures in which SRD and EFD are iteratively performed, as described in Fig.5. Each pyramidal feature volume is then used as the input to the next EFD module. The last one is utilized as an input of the multi-scale feature aggregation step in Sec.3.3.

Sharp Region Detector. The initial feature of each focal slice is needed to communicate with other neighboring focal slices, to measure the focus of the pixel of interest. A work in [21] extracts focus features using a global pooling layer as a communication tool across a stack dimension. However, we observe that the global pooling layer causes a loss of important information due to its inherent limitation that all values across focal slices become single values.

Multi-scale feature aggregation. The receptive field of our feature extraction module might be too small to learn non-local features. Therefore, we propose a multi-scale feature aggregation module using one hour-glass module to expand the receptive field, which is similar to the stereo matching network in [32]. At an initial step, we use three different sizes of kernels (2$\times$2, 4$\times$4, 8$\times$8) in the average pooling layer. Unlike [32], the reason for using average pooling is to avoid a memory consumption issue because DfF requires more input images. We then apply a ResBlock on each output of average pooling in order to extract multi-scale features. These features are embedded into the encoder and aggregated by the decoder of the hour-glass module. The aggregated feature volume is utilized as an input in the refinement step.

Refinement and Regression. The refinement module has three hour-glass modules with skip-connections like [3]. Here, we add transposed convolutions to resize the output of each hourglass whose size is the same as each level of a pyramidal feature volume from the feature extraction module. We construct an input focal volume of each hourglass by concatenating pyramidal feature volumes of the feature extraction module with the output focal volume of the previous hourglass. As each hourglass handles increasingly higher resolutions with pyramidal feature volumes, the focal volumes are refined in a coarse to fine manner. To obtain a depth map from the output focal volumes, we multiply a focus distance value and the probability of each focus distance leading to maximal sharpness. The probability is computed by applying a normalized soft-plus in the output focal volumes in a manner similar to [37]. The whole depth prediction network is optimized using a weighted loss function $L_{depth}$ from scratch as follows:

where $||\cdot||_{2}$ means a $l_{2}$ loss and $D_{gt}$ indicates a ground truth depth map. $i\in\{1,2,3,4\}$ means the scale level of the hour-glass module. In our implementation, we set $w_{i}$ to 0.3, 0.5, 0.7 and 1.0, respectively.

Implementation details. We train our network using the following strategy: (1) We first train the alignment network in Sec.3.1 during 100 epochs using the alignment loss in Eq.(2). (2) We freeze the alignment network and merge it with the depth prediction network. (3) We train the merged network during 1500 epochs with the depth loss in Eq.(3). (4) In an inference step, we can estimate the depth map from the misaligned focal stack in an end-to-end manner. We note that our network is able to use an arbitrary number of images as input, like the previous CNN-based DfF/DfD [21, 37]. The number of parameters of our alignment network and feature extraction module is 0.195M and 0.067M, respectively. And, the multi-scale feature aggregation module and the refinement module have 2.883M and 1.067M learnable parameters, respectively. That’s, the total parameters of our network is 4.212M. We implement our network using a public PyTorch framework [25], and optimize it using Adam optimizer [18] ($\beta_{1}=0.9,\;\beta_{2}=0.99$) with a learning rate $10^{-3}$. Our model is trained on a single NVIDIA RTX 2080Ti GPU with 4 mini-batches, which usually takes three days. For data augmentation, we apply random spatial transforms (rotation, flipping and cropping) and color jittering (brightness, contrast and gamma correction).

DDFF 12-Scene [12]. DDFF 12-Scene dataset provides focal stack images and its ground truth depth maps captured by a light-field camera and a RGB-D sensor, respectively. The images have shallow DoFs and show texture-less regions. Our method shows the better performance than those of recent published works in Tab.1 and achieves the top rank in almost evaluation metrics of the benchmark site²²2https://competitions.codalab.org/competitions/17807#results.

DefocusNet Dataset [21]. This dataset is rendered in a virtual space and generated using Blender Cycles renderer [4]. Focal stack images consist of only five defocused images whose focus distances are randomly sampled in an inverse depth space. The quantitative results are shown in Tab.2. As shown in Fig.6, our method successfully reconstructs the smooth surface and the sharp depth discontinuity rather than previous methods.

4D Light Field Dataset [14]. This synthetic dataset has 10 focal slices with shallow DoFs for each focal stack. The number of focal stacks in training and test split is 20 and 4, respectively. For fair comparison on this dataset, we follow the evaluation protocol in the relevant work [37]. In qualitative comparisons Fig.7, our SRD and EFD enable to capture sharp object boundaries like the box and fine details like lines hanging from the ceiling. In quantitative evaluation of Tab.2 the MSE and RMSE are half of them from the comparison methods [1, 38].

Smartphone [13]. This dataset shows real-world scenes captured from Pixel 3 smartphones. Unlike previous datasets, ground truth depth maps are obtained by multiview stereo [30, 31] and its depth holes are not considered in the evaluation. As expected, our network achieves the promising performance over the state-of-the-art methods, whose results are reported in Tab.2 and Fig.8. We note that our method consistently yields the best quality depth maps from focal stack images regardless of dataset, thanks to our powerful defocused feature representations using both SRD and EFD.

Generalization across different datasets. Like [37], we demonstrate the generality of the proposed network. For this, we train our network on Flyingthings3D [22] which is a large-scale synthetic dataset, and test it on two datasets including Middlebury Stereo [29] and DefocusNet dataset [21]. As shown in Tab.3 and Fig.9, our network still shows impressive results on both datasets.

Alignment network. We first evaluate our alignment network. To do this, we render focal stacks using our simulator which generates defocused images based on a camera metadata. We test our alignment network in consideration of four cases: 1) without any warping, 2) with only initial FoVs in (1), 3) a classical homography method [5], 4) our alignment network with initial FoVs. The quantitative results are reported in Tab.4, whose example is displayed in Fig.10. The results demonstrate that our alignment network achieves much faster and competable performance with the classic homography-based method.

SRD and EFD. We compare our modules with other feature extraction modules depicted in Fig.11. We conduct this ablation study on DefocusNet dataset [21] because it has more diverse DoF values than other datasets. The quantitative result is reported in Tab.5.

Number of focal slices. Like previous DfF networks [37, 21], our network can handle an arbitrary number of focal slices by the virtue of 3D convolutions. Following the relevant work [37], we train our network from three different ways, whose result is reported in Fig.12: The ’5’ means a model trained using five focal slices; The ’Same’ denotes that the number of focal slices in training and test phase is same; The ’Random’ is a model trained using an arbitrary number of focal slices.