Learning to Estimate Kernel Scale and Orientation of Defocus Blur with Asymmetric Coded Aperture

The input of our pipeline is a spatially-varying defocused image. Spatially-varying indicates the blur at each pixel may differ from the neighbor in terms of scale and orientation. It has been shown in  [20] that performing a deblurring algorithm with the wrong kernel yields low-quality images while the correct one produces high-quality results. Using this insight, we formulate blur kernel estimation to separate a correct deblurred result from the wrong ones to teach the neural network how to choose the right kernel scale and orientation. Thus, in our framework, we first perform a deblurring algorithm with PSF hypotheses (varying in scale and orientation) on a defocused image. These results are then concatenated and fed into the network, which is trained to identify the optimal deblurred result index for each pixel. The PSF kernel associated with the optimal deblurred image quality reports the correct scale and orientation.

Our training target can be formulated as an optimization problem:

	$\displaystyle\min_{\theta}{\cal L}_{c}($	$\displaystyle pred_{\theta},RS(gt))+\lambda_{s}{\cal L}_{s}(GS(pred_{\theta})),$		(1)
		$\displaystyle pred_{\theta}=f_{\theta}(RS(d_{1},d_{2},\dots,d_{n},dI)),$

where a weighted sum of cross-entropy loss ${\cal L}_{c}$ and smoothness loss ${\cal L}_{s}$ is minimized. The network (denoted as $f_{\theta}(\cdot)$, with model parameters $\theta$) takes the deblurred results $d_{1},d_{2},\dots,d_{n}$ and the original defocused image $dI$ as input. We perform random shuffle (denoted as $RS(\cdot)$) on the input and the ground-truth label $gt$ to avoid overfitting and impose a smoothness loss on the network output $pred_{\theta}$ for better performance. In computing the smoothness loss, we apply the Gumbel-Softmax [21] technique (denoted as $GS(\cdot)$) in order to convert the 3D output $pred_{\theta}$ into 2D. The smoothness loss is then weighted by $\lambda_{s}$. We detailed the motivation to perform random shuffle and the Gumbel-softmax methods in Section 3.2.

The network output is a logit volume with a size of [height, width, depth]. Vectors along the depth dimension reveal the likelihood that slices in the input are sharp (high-quality) or not. The final blur scale and orientation for each pixel can be determined by looking up the blur kernel parameter associated with the highest logit value.

We expect the estimated blur map to be smooth as the blur scale and orientation share the same continuous character as scene depth does in most areas. To compute smoothness loss on our output, we need to convert the 3D logit volume into a 2D indexing map. As the $\arg\max$ operation to flat a 3D volume into 2D is not feasible here with its non-differentiable property, we apply Gumbel-Softmax [21] to fulfill this need. It takes a logit vector as input, and outputs the index of the element associated with the largest logit value. Then, we compute ${\cal L}_{s}$ through taking the absolute average of the first-order gradient of the 2D indexing map. The final loss ${\cal L}$ is a weighted sum of ${\cal L}_{c}$ and ${\cal L}_{s}$: ${\cal L}={\cal L}_{c}+\lambda_{s}{\cal L}_{s}$, where we weight ${\cal L}_{s}$ by $\lambda_{s}$.

Our training procedure needs a large portion of spatially-varying defocused images, each with a per-pixel ground-truth blur scale and orientation to supervise the network training. With very few previous works sharing coded aperture images, none of them involves scene objects resting on both sides of the focal plane. As we intend to address the focal plane ambiguity problem, we need to generate a new dataset to meet the need. Here we present an approach to acquire defocused images from CG generated all-in-focus images.

$$\centering c(x)=\alpha\frac{x-S_{1}}{x},\,\,with\,\,\alpha=\frac{f_{1}}{S_{1}}D\@add@centering$$

(2)

The sign and magnitude of $c(x)$ indicate the orientation and the scale of the blur kernel, respectively. With an all-in-focus image ${I}_{k}$ and coresponding depth map ${d}_{k}$, for each pixel ${p}_{i}\in{I}_{k}$ and its depth value ${x}_{{p}_{i}}$, we can compute the blur parameter $c({x}_{{p}_{i}})$ according to Eq. 2. We round $\{c({x}_{{p}_{i}})|\forall{p}_{i}\in{I}_{k}\}$ to obtain a set of $\{{c}_{{I}_{k}}\}$ where $[c({x}_{{p}_{i}})]\in\{{c}_{{I}_{k}}\},\forall{p}_{i}\in{I}_{k}$. Here $[\cdot]$ denotes the rounding operation. With ${c}_{i}\in\{{c}_{{I}_{k}}\}$ , we resize, flip, and normalize the coded aperture to form a PSF ${F}_{{c}_{i}}$ according to ${c}_{i}$. Convolution is then performed on each pixel ${p}_{i}$ in the all-in-focus image ${I}_{k}$ with kernel ${F}_{[c({x}_{{p}_{i}})]}$ to form the spatially-varying defocused image $\hat{{I}}_{k}$. In addition, we save $[c({x_{{p}_{i}}})],\forall{p}_{i}\in{I}_{k}$ as the per-pixel ground-truth label for the training process. To avoid producing unrealistic blurry images, we restrict the maximum blur size during defocus generation.

We capture 100 all-in-focus images from scenes provided in UnrealCV, and generate defocused images for each one with 12 different focal distances ($S_{1}$ in Fig. 1). Thus 1200 defocused images are acquired totally with each coded aperture we used. The maximum blur scale is restricted to be 7, 9, and 10 pixels to form the depth-13, depth-17, depth-19 dataset, respectively. We later use these datasets vary in coded apertures and maximum blur scales to verify our framework’s generalization performance.

In our experiments, we split the entire dataset for each coded aperture and maximum blur scale to 936 defocused images for training, 108 for validation, and 156 for evaluation. The split is implemented in such a manner that there are no overlapping scenes across the training/validation/evaluation subsets, which means all 12 defocused images generated from one particular scene are forced to belong to one of these three subsets.

We implement our proposed network architecture in Table. 1 using TensorFlow API [25]. In the training procedure, the network is optimized by the SGD optimizer with the random shuffle applied on the input stack and label. The learning rate, batch size, and smoothness weight $\lambda_{s}$ are 0.01, 1, and 0.1, respectively. The temperature argument for the Gumbel-softmax function is 0.5. We train the network on a single GPU (GeForce RTX 2080 Ti), and the network takes around five days to converge.

We formulate our blur kernel estimation as a classification problem. Logit volume from the network output indicates whether pixels in the deblurred stack are classified as optimal-quality or not. The predicted blur scale and orientation can be determined by indexing the highest value along the depth channel of the logit volume. We report the prediction N-1 accuracy and the N-3 accuracy in the following manner: N-1 accuracy indicates the percentage of pixels holds the same predicted blur scale and orientation as the ground-truth label does. The N-3 accuracy is computed similar to N-1 but also counting the nearby prediction (i.e., the pixel is predicted within the range of its ground-truth label $\pm 1$) as a correct prediction. For example, pixel with label +3 predicted as +2 (or +4) is not counted as a correct prediction in N-1 but in N-3. The intention behind reporting the N-3 accuracy is to compensate for the inaccuracy introduced during rounding COC diameter $c(x)$. The N-3 accuracy also registers how effective this framework will perform in a real-world application. Since even the prediction is off by 1 pixel, adjusting focus with it will still result in a much more in-focus image.

After further analyzing the incorrect prediction, which significantly differs from its neighbor region’s prediction and the ground-truth label, such as the bottom right part of the first row (dark area) in Fig. 3, we find that a large piece of them comes from the texture-less region in the all-in-focus image. We hold this is an ill-posed challenge as defocus texture-less images with various blur scale and orientation show a tiny variation in between. Also, in real-world applications, a texture-less region can hardly be a region of interest.