Efficient Light Field Reconstruction
via Spatio-Angular Dense Network

To process LF images, early methods [35, 36, 37, zhang_light_2015] explicitly estimate the disparity or depth and then obtain the reconstructed SAIs by warping the input SAIs and blending. Various depth estimation methods are proposed, e.g., phase-based estimation [zhang_light_2015] and EPI-based estimation [37]. Different blending methods are also introduced, e.g., soft-blending [38] and learning-based blending [39]. These methods are overly dependent on the quality of the disparity or depth maps which the existence of noise in them causes undesirable artifacts. Such approaches are also disadvantaged when handling occluded areas as the warping process has no information about the invisible parts.

In recent years, significant progress in computer vision has been achieved by the deep learning-based methods [28, 40, 41, 32], and this technique has been applied to LF reconstruction. The first deep learning-based LF reconstruction method has been proposed by Kalantari et al. [20] which designed a disparity network to estimate the disparity and reconstruct the intermediate SAIs by warping the input SAIs with the calculated disparity. After that, the input and intermediate SAIs are fed into a color network to obtain the refined SAIs. This method has achieved outstanding performance with the deep features extracted by the deep neural networks. However, the reconstruction quality is limited due to the artifacts brought by the explicit disparity estimation and warping process. Moreover, each SAI needs to be individually reconstructed, which leads to duplicated calculations. Similarly, Zhou et al. [42] proposed an encoder-decoder network to estimate the disparity to synthesize views by warping. With the help of its modified ResNet-50 to extract expressive representation and three sub-networks for disparity estimation, noise filtering, and view rendering respectively, robustness is gained against input noise.

Wang et al. proposed an EPI-based method to decompose the 4D task into 3D sub-tasks and train two networks separately for vertical and horizontal reconstruction [27]. Heber et al. [26, 10] designed a U-shaped network to extract shape information by EPIs while Shin et al. [25] proposed a EPI-based method to compute disparity maps from LF with a four-branch network. A common drawback of EPI-based methods is they ignore either one [27, 25, 26] or two dimensions [22, 10] of the 4D LF in their optimization and cannot be jointly optimized in all dimensions. Therefore, they cannot process the LF data in a global scope.

In [21], Yeung et al. proposed an end-to-end network that removes explicit disparity manipulation and reconstructs SAIs at a single forward propagation. This method has achieved groundbreaking performance with its SAS convolution that enables joint optimization in all dimensions. However, it has endured the same defect of stacking convolutional layers for obtaining deeper feature representation as other computer vision tasks [28, 43]. Different from decomposing 4D convolution [21], Meng et al. [44] pursued to fully exploit high-dimensional LF information and proposed aperture group batch normalization to ease the training of 4D convolutional network. In addition to pixel-wise loss function in the spatial space, they proposed an angular loss to optimize the error in the EPI space to preserve the correlation information of adjacent viewpoints. However, suffering from heavy 4D convolution operations, it was still running at a large cost of computation and memory, and the high-dimensional information was yet to be fully exploited as the performance gain was not proportional to its costs. Besides pure learning-based methods, Chandramouli et al. [45] proposed a generative model to tackle the issue that learning-based models are confined to the observation model they have been trained on. Despite its limited performance, the fashion is inspiring with possibilities of LF models with generalization abilities.

There are some other methods [46, 47] that perform LF reconstruction but in a compressive manner. Their compression performance is proved to be promising, but they are essentially a different task from regular LF reconstruction as the full LF image is involved to be encoded into a small volume of information instead of a limited set of SAIs. Although the above methods and the regular LF reconstruction are not working in exactly the same application scenarios, it would be interesting to see where their performance ceilings are located.

In this paper, our proposed SADenseNet has a similar pipeline as [21] that contains two parts, namely the feature extraction and the SAI reconstruction. However, we ditch the refinement work as SADenseNet’s feature extraction is so strong with the help of correlation blocks and spatio-angular dense skip connections that post-processing components are unnecessary. Rather than inefficient explicit warping process [20] and costly 4D convolution operations [44], we pursue the potential of decomposed convolution operations and information flow enhancement.

In order to acquire powerful feature representation, stacking convolutional layers is the most straightforward solution since more parameters lead to higher non-linearity and more complicated mapping functions. However, heavier computation costs and higher risks of over-fitting are also incurred. To this end, some studies have discovered more efficient network structures.

The residual connection is one of the widely used structures firstly proposed by He et al. [48] to force layers to learn residue of a mapping function instead of the mapping function itself. Such a technique eases the optimization and improves the feature representation, and can be applied to other computer vision tasks such as single image super-resolution (SISR) [29, 49, 50] and visual tracking [51, 52].

On the other hand, dense skip connections have been proposed to mitigate the gradient-vanishing problem while aggregating hierarchical features of low- and high-frequencies [30]. Tai et al. [31] proposed a dense connected network to aggregate short-term and long-term memories for image restoration. In SISR, Tong et al. designed local and global dense skip connections to aid the feature extraction [32]. Haris et al. [33] designed a projection unit to learn down-sampling as well as up-sampling. Li et al. [34] introduced feedback units containing projection groups to form a recurrent neural network (RNN) to enforce a curriculum learning strategy. The basic units of the latter two methods [33, 34] are reinforced with dense skip connections to gain richer feature representation. Among these, [31] and [32] have similar architectures with our SADenseNet as they perform dual dense skip connections. Nevertheless, the critical difference is that their duality is for extracting homogeneous features while ours are for spatio-angular features from the two distinct domains.

Video-related tasks share similarities with LF tasks as they are both extracting cross-domain representation and processing multiple images or frames that are highly correlated to each other. A powerful spatio-temporal feature representation is a key to success as well. In terms of decomposition of high dimensions, similar methods can be found in pseudo-3D-based P3D [53, 54] and R2D [55]. Different from the pseudo-4D convolution in LF, the pseudo-3D convolutions decompose a 3D convolution into two 2D convolutions operating in the spatial and temporal spaces separately. Convolution decomposition has been proved to achieve better performance than the original 3D convolution achieved by the extra non-linearity and the reduced risk of over-fitting.

Let us consider $\mathcal{F}(X)$ as the LF reconstruction function that maps the input SAIs $X$ to the reconstructed SAIs $\hat{Y}$, hence

The objective of this paper is to train an end-to-end network to learn $\mathcal{F}(X)$ in Eq. 1 for estimating $\hat{Y}$ that approximates the ground truth $Y$. The proposed SADenseNet comprises two components, correlation feature extraction, and SAI reconstruction, and an illustration is given in Fig. 2. The input SAIs $X$ are fed into correlation feature extraction containing a series of correlation blocks, which will be described in Section III-B. In addition to the correlation blocks, spatio-angular dense skip connections are exploited through the inter- and intra-correlation-blocks to enhance the information flow, which will be elaborated in Section III-C. With the extracted correlation features, SAI reconstruction will finally reconstruct SAIs $\hat{Y}$, which will be explained in Section III-D.

For addressing the aforementioned domain asymmetry problem in Section I, we propose correlation blocks that process the spatial and angular information asymmetrically. The structure is demonstrated in Fig. 3.

The component is based on the SAS module proposed in [21, 23]. However, motivated by the fact that the spatial space is usually substantially larger compared to the angular space, i.e. $U\times V\ll W\times H$, we presume that more spatial operations are needed than angular operations. Therefore, in each block, the 4D LF tensor of size $(U,V,W,H,C_{1})$, where $C_{1}$ is the number of input channels, is firstly reshaped into the spatial mode of size $(U\times V,W,H,C_{1})$ to flatten the angular dimensions. Subsequently, a series of spatial convolutional layers, denoted as $[\mathcal{C}^{\mathcal{S}}_{1},\mathcal{C}^{\mathcal{S}}_{2},\dotsb,\mathcal{C}^{\mathcal{S}}_{n_{\mathcal{S}}}]$, are operated on the reshaped tensor of kernel size $(1,k_{W},k_{H},C_{1},C_{1})$ to extract the spatial features, where $n_{\mathcal{S}}$ is the number of spatial convolutional layers¹¹1In this paper, the last two values of the size of 3D convolution kernels denote the input and output channels correspondingly.. Likewise, the output of spatial convolution series is reshaped into the angular mode of size $(U,V,W\times H,C_{1})$ and convolved by an angular convolutional layer of kernel size $(k_{U},k_{V},1,C_{1},C_{2})$ to extract angular features that model the correlation information, where $C_{2}$ is the number of output channels. Lastly, the tensor is reshaped back into the original shape with new channels $(U,V,W,H,C_{2})$ for the upcoming operation. Hence, a correlation block can be formulated as

Such an asymmetrical processing method suits LF image data compared with the existing 4D convolution alternatives because the spatial information is sufficiently convolved to extract informative features with an enlarged receptive field in the spatial domain before the subsequent angular convolution.

In order to acquire deeper correlation features, a series of correlation blocks are employed consecutively as

where $F_{i}$ denotes the output tensor of $i$-th correlation block.

The angular convolutions of correlation blocks are operated on the coarse-to-fine spatial features, thus they can extract correlation features hierarchically. The correlation extraction can be regarded as implicit disparity modeling but without explicit dependency on the accuracy of disparity. We denote the number of correlation blocks as $n_{CB}$ and set both $(k_{U},k_{V})$ and $(k_{W},k_{H})$ to $(3,3)$ across all the correlation blocks. Finally, the output of the last correlation block $F_{n_{CB}}$ will be fed to the following SAI reconstruction process.

For further enhancing the feature representation extraction process, we propose to improve the information flow with spatio-angular dense skip connections consisting of three types of connections, namely spatial and angular dense skip connections, and image skip connections.

The spatial dense skip connections are intra-correlation-block connections that concatenate the output of shallow spatial convolutional layers to the output of the subsequent layers within a correlation block, i.e. the spatial convolutional layers will receive the features from the preceding layers. Consequently, the spatial convolution function in Eq. 4 can be revised as

where $\tilde{\mathcal{C}^{\mathcal{S}}_{i}}(x)$ represents the $i$-th densely connected spatial convolutional layer and $[\cdot]$ indicates concatenation. The spatial dense skip connections are demonstrated in Fig. 2 and 3 as blue arrows connecting blue tensors under the spatial mode.

The densely connected structure comes with three critical benefits. Firstly, the feature tensors are reinforced with both shallow and deep information, forming hierarchical representations. This is contrary to the previous methods [21] where the shallow features are absent in high-level processing. Secondly, it facilitates the optimization of the network, especially for the early layers, since every layer will access the gradients directly in back-propagation and the gradient vanishment problem is also greatly alleviated. Last but not least, the high trainable parameter utilization leads to model efficiency and prevents over-fitting.

While the spatial dense skip connections strengthen the spatial representation within individual correlation blocks, we explore to bring this benefit to the angular domain by adopting angular dense skip connections that concatenate the output of the preceding correlation blocks $F_{i}$ to the latter ones. As angular convolutions are connected with their counterparts in the previous correlation blocks, the angular dense skip connections play a role as inter-correlation-block information flow as depicted in Fig. 2 as the red arrows.

Furthermore, inspired by the practice of appending raw input to the intermediate layers in video processing [41] and optical flow [56], we introduce image skip connections to provide input $X$ as primitive features for all the blocks. This design is very similar to [41, 56] as these two methods implicitly estimate the motion between frames and inject raw image information into the intermediate motion feature for further refinement, while in our case, the correlation blocks implicitly model the LF disparity and the raw image offers complementary information for the angular convolutions. The image skip connections are depicted as yellow arrows connecting the input LF and the correlation blocks in Fig. 2.

Accordingly, the function of correlation blocks in Eq. 5 are revised into

where $[\cdot]$ indicates concatenation. The output of the last correlation block $F_{n_{CB}}$ aggregates the feature maps of all the correlation blocks to form hierarchical spatio-angular feature representation. It will be fed to the following SAI reconstruction process.

In [30], the number of feature maps increased by dense skip connections is referred to as the growth rate. For simplicity, we keep the growth rates in both domain consistent across the network. Therefore,

where $r_{\mathcal{S}}$ and $r_{CB}$ are growth rates of the spatial and angular dense connections respectively. Consequently, the number of feature maps of the last correlation block $F_{n_{CB}}$ and the last spatial convolution $\tilde{\mathcal{C}}^{\mathcal{S}}_{n_{S}}(x)$ in each correlation block will grow to $r_{CB}\times n_{{CB}}$ and $r_{\mathcal{S}}\times n_{\mathcal{S}}$ respectively. In the experiments of Section IV, the growth rates are set to 32, which are only half of the feature map number used in [21], by virtue of the efficiency of densely connected architecture.

Before reconstruction using the extracted hierarchical correlation features, we follow the practice in [32] to employ a bottleneck layer to aggregate the hierarchical features and reduce the number of the feature maps to 96. Then, the reduced features are reshaped into the angular mode, zero-padded, and fed into an angular convolutional layer with a kernel size of $(U,V,1,96,\mathcal{N})$. It shrinks the angular resolution from $(U\times V)$ to $(1\times 1)$ while producing $\mathcal{N}$ channels, which correspond to the $\mathcal{N}$ reconstructed SAIs. In Fig. 2, as $(U\times V)=(2\times 2)$ and $\mathcal{N}=60$, the kernel size of the angular convolution is $(2,2,1,96,60)$.

The proposed SADenseNet is implemented using the deep learning library Keras [58] with Tensorflow [59] backend. The network is trained and tested on a PC with an Intel Core i7-6700K 8-core 4.00GHz CPU, an Nvidia GTX 1080 Ti GPU, and 32GB RAM. The source code and trained models are publicly available at https://huzexi.github.io.

In regard to training, to conduct a fair comparison, we follow the protocol of [21] to train the network with the training set proposed by Kalantari et al. [20], which contains 100 samples. The mini-batch size is set to 2 with a spatial size of $(128\times 128)$. The training process is iterated with an Adam optimizer [60] and the learning rate is set to $1e-4$. We follow the conventional practice to process the luminance channel of the YCbCr color space. The other two channels, namely Cb and Cr, are acquired by up-sampling the angular resolution using bicubic interpolation. During training, data augmentation is applied to improve the generalization of the network. We follow the strategies in [25] to randomly flip and rotate the spatial and angular dimensions simultaneously. As a result, the training data can be reused 8 times.

$n_{CB}$ and $n_{\mathcal{S}}$ are set to 6 and 5 correspondingly as the default values, and the network is trained by minimizing the Mean Squared Error (MSE) loss function as follows:

where $n$ iterates over the reconstructed SAIs and $(x,y)$ indicates a spatial location.

Firstly, we compare SADenseNet’s performance on real-world images with the state-of-the-art methods: Kalantari et al. [20], HDDRNet [44], NoisyLFRecon [42] and Yeung et al. [21].

Experiments are conducted on four real-world LF datasets, namely 30 Scenes [20], EPFL [61], Occlusions [62] and Reflective [62] which are captured by Lytro Illum cameras [18]. These four datasets contain 30, 118, 43, and 31 LF images correspondingly. The LF images that have appeared in the 100-sample training set are removed for fairness. In each LF image, the angular resolution is $(14\times 14)$ and the spatial resolution is $(376\times 541)$. During the evaluation, only the central $(8\times 8)$ SAIs are adopted as the rest are dark and noisy, 22 pixels of the four borders in the spatial resolution are also shaved as in [20]. The reconstruction is performed from $(2\times 2)$ SAIs to $(8\times 8)$ SAIs. The reconstruction quality is measured by Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) in RGB color space.

The results are shown in Table I. It can be observed that SADenseNet has outperformed the other methods observably. Compared with the best of them, Yeung et al. [21], SADenseNet outperforms by more than 1.00 dB PSNR in all datasets except the Reflective dataset where our proposed approach has a reconstruction advantage of 0.71 dB PSNR.

For a better understanding of the performance gains, selected visual results from the four test datasets are presented in Fig. 4. It is visible that our error maps are basically clearer than the other prior arts. Concretely, the reconstructed edges of the objects are more complete in IMG_1528, e.g., the lamp pole behind the leaves in the red bounding box and the sign above the vehicles in the blue bounding box. The lamp pole example poses a major challenge to LF reconstruction as it is occluded by the leaves in some SAIs. Such a phenomenon can also be observed at the fence border in the red bounding box of Occlusions_29 and the English letter behind the leaves in the blue bounding box of Reflective_12. This can be attributed to the abundant spatio-angular representation in handling the appearance that is visible in limited SAIs. Similar improvement can also be observed in the reconstruction of complicated details, e.g., the flower cores in the blue and red boxes of Mirabelle_Prune_Tree, and the windows in the blue bounding box of Occlusions_29, which are entirely blurred in the result of most methods. In Reflective_12, it is very obvious that the method of HDDRNet [44] and Yeung et al. [21] incorrectly reconstructs a large area of shadow at the bottom left corner while SADenseNet successfully reconstructs it with the long-term information supplied by the dense skip connections [31] and enlarged spatial receptive fields of the correlation blocks while HDDRNet [44] and Yeung et al. [21] confine the correlation information and falsely renders the shadow across all SAIs.

On the other hand, to demonstrate the model’s efficiency, we compare the memory cost in terms of the number of trainable parameters and the computation cost in terms of the testing speed, reported in Table I. The speed tests are operated in CPU-only mode. The results demonstrate that the method of Kalantari et al. [20] and NoisyLFRecon [42] are plagued by their explicit disparity estimation and warping process. Also, their SAI synthesis functions must be performed separately for each SAI as the intermediate information cannot be shared ending up with a very low processing speed. Regarding the memory cost, HDDRNet [44] and NoisyLFRecon [42] employ a relatively huge model as the former hires 4D convolution operators and the latter adopts a modified ResNet-50 which has taken more than 19 million parameters. These two means aim at extracting an expressive feature representation but lead to a huge model instead of improvement in reconstruction performance. On the other hand, the end-to-end separable-convolution-based methods, [21] and SADenseNet, reconstruct SAIs in one propagation at a substantially faster speed. Moreover, compared with [21], our SADenseNet requires only 3/4 of parameters and is nearly 3 times faster. The achievement can be attributed to the correlation blocks with efficient convolutions and spatio-angular dense skip connections that improve information flow.

The performance of SADenseNet is also evaluated on the synthetic HCI dataset [57]. We follow the practice of Wu et al. [24] and Yeung et al. [21] to calculate PSNR and SSIM on the Y channel only over the $(3\times 3)$ to $(9\times 9)$ reconstruction task. The results on Buddha and Mona are reported in Table II. SADenseNet has produced competitive results exceeding Yeung et al. [21] by 1.61 dB PSNR averagely. In regard to SSIM, SADenseNet has obtained the closest results to the ones reported by Wu et al. in [24] while outperforming its PSNR significantly likewise.

The reconstruction quality of the two testing images has been visualized in Fig. 5. The most significant difference in Buddha is located at the edges of the bricks on the floor where SADenseNet does not produce the blurring artifacts as [21]. Similar improvement has been observed in Mona where artifacts are produced around the leaf border in [21] but not seen in SADenseNet.

To further study the nature of SADenseNet, a series of ablation studies are conducted in this section. We keep the previous configuration in Section IV-B to conduct the studies on real-world datasets.

Depth estimation is one of the important LF measurement applications. As SAIs are reconstructed, we further investigate the depth maps estimated from the densely sampled LF to demonstrate SADenseNet’s reconstruction quality from another perspective. Fig. 8 gives some selected images’ depth maps generated from the ground truth, the densely-sampled LF reconstructed by Yeung et al. [21] and SADenseNet using the estimation method of [13]. It can be observed that SADenseNet has produced visibly better depth maps than Yeung et al. [21] with high-quality object edges in Chain-link_Fence_2. It is interesting that in Cars and IMG_1411, depth generated on SADenseNet’s reconstructed SAIs is more satisfactory than ground truth. In Cars, the car on the left should be located between the metal plate in the foreground and the building in the background. Both ground truth and Yeung et al. [21] misclassify it as having a similar depth to the metal plate. The misclassification mainly happens in textureless areas such as the sky in the top-left corner of Cars and to the left of the plant in IMG_1411. However, depth maps estimated with SADenseNet do not suffer from this problem.

The depth predictions in Fig. 8 suggests that the region of the textureless areas and the foreground objects become more distinguishable to the depth estimation algorithm [13] after being pre-processed by our proposed method. One plausible explanation for such improvement is that the prior learned by our proposed method is able to refine the input LF during reconstruction before feeding into the depth estimation algorithm. In conclusion, SADenseNet not only produces high reconstruction quality but also supplies complementary information to improve the accuracy of depth estimation and other measurement applications.

Even though SADenseNet has demonstrated superior performance in the task of LF reconstruction, it doesn’t necessarily outperform other methods across all samples. While there are 222 real-world samples tested in the previous evaluation, SADenseNet fails 14 of them, in most of which reflective surfaces are presented. Two of them are visualized in Fig. 9 with NoisyLFRecon [42] and Yeung et al. [21] as the representatives of disparity-based and separable-convolution-based methods, respectively.

In reflective_29, it is observed that SADenseNet is outperformed by Yeung et al. [21] by 0.5 db PSNR, and in the error maps the reflective surface of the steel pot is reconstructed with distorted artifacts in the red box which doesn’t happen in the results of the other two methods. In the other example, Water_Drops, in spite of the higher PSNR and SSIM achieved by Yeung et al. [21] and SADenseNet, the flaw can be easily spotted in their error maps that the reflective water drops are not correctly reconstructed, while NoisyLFRecon [42] is not confused by these reflections with a visibly clearer error map. Given these results, we presumed that disparity-based methods are more robust to the scenes with reflective surfaces, which remains a problem for our method to study in future research.