Spatial-Angular Attention Network for
Light Field Reconstruction

First, we briefly review the major works on light field view synthesis (or view synthesis) depending on whether the depth information is explicitly used.

Depth image-based view synthesis. Typically, these kind of approaches first estimate the depth of a scene, then warp and blend the input views to synthesize a novel view [20, 8]. Conventional light field depth estimation approaches follow the pipeline of stereo matching [21], i.e., cost computation, cost aggregation (or cost volume filtering), disparity regression and post refinement. The main difference is that light field converts disparity from discrete space into a continuous space [22], delivering various depth cues that enable light field depth estimation with different strategies, such as structure tensor-based local direction estimation [22], depth from correspondence [23], depth from defocus [24, 25] and depth from parallelogram cues [26]. Moreover, some learning-based approaches incorporate the aforementioned depth estimation pipeline with 2D convolution-based feature extraction, 3D convolution-based cost volume aggregation and depth regression [27]. For novel view synthesis, input views are warped to the novel viewpoints with sub-pixel accuracy using bilinear interpolation and blended in different manners, e.g., total variation optimization [22], soft blending [20] and learning-based synthesis [28].

Recently, researchers mainly focus on the studies for maximizing the quality of synthesized views based on the deep learning technique. Flynn et al. [13] proposed a learning-based method to synthesize novel views using the predicted probabilities and colors for each depth plane. Kalantari et al. [7] further employed a sequential network setting to infer depth (disparity) and color, and optimized the model via end-to-end training. Following the sequential network setting, Meng et al. [29] developed a confidence estimation network between depth and color networks to infer pixel-wise blending weights. Shi et al. [8] proposed to blend the warped views in both pixel level and feature level. Jin et al. proposed to use a regular sampling pattern (four corner views) [30] and a flexible sampling pattern [31] for light field depth estimation, and then perform the reconstruction using spatial-angular alternating refinement. Ko et al. [32] introduced a dynamic blending filter to generate the filter coefficients adaptively according to the warped views. Different from the sequential network settings mentioned above, Zhou et al. [33] proposed a novel learning-based Multi-Plane Image (MPI) representation that infers a novel view by alpha blending of different images. Mildenhall et al. [14] further proposed to use multiple MPIs to synthesize a local light field.

Depth image-based view synthesis approaches solve the problem of large correspondence gap in the sparsely-sampled light field by using depth estimation and warping. But the scene depth are based on the Lambertian assumption, and thus, these approaches will suffer from depth ambiguity when addressing the non-Lambertian effect, as demonstrated in Fig. 8 (second case). In this paper, we address the problem of large correspondence gap with a non-local attention mechanism to capture large gap correspondences. Since we do not rely on depth information, the proposed method shows higher reconstruction quality on the non-Lambertian cases.

Reconstruction without explicit depth. These kind of approaches treat light field reconstruction as the approximation of plenoptic function. In the Fourier domain, the sparse sampling in the angular dimension produces overlaps between the original spectrum and its replicas, leading to aliasing effect [10]. Classical approaches [34, 35] consider a reconstruction filter (usually in a wedge shape) to extract the original signal while filtering the aliasing high-frequency components. For instance, Vagharshakyan et al. [36] utilized an adapted discrete shearlet transform in the Fourier domain to remove the high-frequency spectra that introduce aliasing effects. Shi et al. [37] performed DSLF reconstruction as an optimization for sparsity in the continuous Fourier domain.

Inspired by the success of deep learning in computer vision, depth-independent plenoptic reconstruction approaches [9, 10] were widely investigated by using deep convolution networks. Specifically, Zhu et al. [38] proposed an auto-encoder that combines convolutional layers and convLSTM layers [39]. For explicitly addressing the aliasing effects, Wu et al. [10] took advantage of the clear texture structure of the EPI and proposed a “blur-restoration-deblur” framework. However, when applying a large blur kernel for large disparities, this approach fails to recover the high-frequency details, and thus leading to blur effect. Liu et al. [40] further applied a multi-stream network that takes 3D EPIs in different directions as input. In addition to extracting slices from the plenoptic function, Yeung et al. [11] directly fed the entire 4D light field into a pseudo 4D convolutional network, and proposed a novel spatial-angular alternating convolution to iteratively refine the angular dimensions of the light field. Jin et al. [41] further extended the spatial-angular alternating convolution to the problem of compressive light field reconstruction. Wang et al. [42] applied pseudo 4D convolution to reconstruct the two angular dimensions of the input light field sequentially. Wu et al. [43] proposed an evaluation network for EPIs with different shear amount, termed as sheared EPI structure, and further boosted its performance with an end-to-end optimization framework [44]. With this structure, the networks implicitly use depth information to select a well reconstructed EPI. However, the performances of these networks are limited by the finite perceptive field of the convolutional neurons, especially when handling the large disparity problem.

Attention was first built to imitate the mechanism of human perception that mainly focuses on the salient part [45, 46, 47]. Vaswani et al. [48] indicated that the attention mechanism is able to solve the long term dependency problem even without using convolution operation or recurrent neural cell. Attention mechanism is typically embedded within a conventional network backbone, such as a VGG-net [19], a ResNet [49, 50] or even a simple multi-layer perceptron (MLP) [51]. It encourages the network to focus on the salient parts by assigning an adaptive weight (attention map) to the extracted features.

Hu et al. [52] and Woo et al. [50] proposed to use a global pooling (max-pooling or average-pooling) followed by an MLP to aggregate the entire information in the spatial dimension. This attention mechanism enables the network to focus on certain channels in the feature maps. However, the global pooling operation will decimate the high-frequencies in the feature maps [53], which could be unacceptable, especially for a reconstruction task. Alternatively, Vaswani et al. [48] proposed to use a weighted average of the responses from all the positions with respect to a certain position for Natural Language Processing (NLP), which is called self-attention. Wang et al. [1] further bridged the self-attention for NLP to more general tasks in computer vision, such as video classification. More concretely, a high-dimensional feature map, e.g., a 3D tensor with spatial-temporal dimensions, is reshaped into its 2D form for the operation of matrix multiplication. Different from the convolution, the self-attention allows the element in the feature map to interact with any elements regardless of their distance (range), and thus, is also termed as non-local attention.

Rather than using the non-local attention mechanism, Wang et al. [54, 55] proposed a parallax attention module to calculate the correspondence between two stereo images along the epipolar line. Tsai et al. [56] introduced an attention module in the angular dimension to weight the contribution of each view in a light field. Guo et al. [57] applied a pixel-wise attention map using convolution layers for adaptively blending feature maps from different frequency components in a light field.

Compared with the attention modules in [54, 55, 56, 57], the major difference in this paper is that the proposed attention is calculated non-locally in the 2D epipolar plane for each pixel, enabling the network to capture spatial-angular correspondences. That also makes us different from existing non-local attention mechanisms [1, 2, 48], which perform non-local attention across the entire data dimensions. Another significant difference is that we embed a channel-to-angular pixel shuffling operation into the non-local attention mechanism to explicitly achieve the light field reconstruction task. To the best of our knowledge, our method is the first to apply non-local attention to the light field reconstruction task.

We propose a network with Multi-Scale Reconstruction (MSR) structure to maintain view consistency (i.e., continuity in the angular dimension) in the low spatial resolution feature space while preserving fine details in the high spatial resolution feature space. As shown in Fig. 3(a), the backbone of the proposed SAA-Net follows the encoder-decoder structure with skip connections (also known as U-net). But the proposed SAA-Net has two particular differences: 1) In each skip connection, we use a deconvolution layer along the angular dimension before feeding the feature maps to the decoder part; 2) In the encoder part, we use strided convolution with stride only in the spatial dimensions of the light field. Table I provides the detailed configuration of the proposed SAA-Net.

The encoder part of the SAA-Net generates multi-scale light field features and reduces the redundant information in the spatial dimension to save the computational and GPU memory costs for the non-local perception. We use two convolutional layers (3D) with stride $[2,2]$ and $[2,1]$ to downsample the spatial resolution of the light field feature with ratio 4 and 2 along the width and height dimension, respectively. Before each downsampling, two 3D convolutional layers with filter sizes $3\times 1\times 3$ and $1\times 3\times 3$ (width, height and angular) are employed to take place of a single convolutional layer with filter size $3\times 3\times 3$, reducing $1/3$ parameters without performance degradation [11].

The skip connections copy the feature layers before each downsampling layer in the encoder, as shown in Fig. 3(a). For each skip connection, a deconvolution layer (also known as transposed convolution layer) is applied to upsample the feature map in the angular dimension, followed by a $1\times 1\times 1$ convolution. Since the angular information mainly concentrates on the 2D EPI $E(x,s)$ for reconstructing a 3D light field $L(x,y,s)$, the filter size in each deconvolution layer in the skip connection is set to $3\times 1\times 7$.

The decoder part of the SAA-Net upsamples the feature map from the spatial-angular attention module (Sec. IV-B) by using two deconvolution layer with stride $[2,1]$ and $[2,2]$ in the spatial dimensions (width and height), respectively. The decoder also receives information from the skip connections by concatenating the features from corresponding levels along the channel dimension [60], as shown in Fig. 3(a). We then use two 3D convolutional layers with filter sizes $3\times 1\times 3$ and $1\times 3\times 3$ to compress the channel numbers in each level of the decoder. This can be considered as the blending of the light field features from different reconstruction scale. Note that all the reconstructions (upsampling operations) in the angular dimension are implemented in the skip connections and the spatial-angular attention module, where the latter will be introduced in the following subsection.

Inspired by the non-local attention mechanism in [1, 2], we propose a Spatial-Angular Attention Module (SAAM) to disentangle the disparity information in light field. The main differences between the proposed SAAM and the previous non-local attention [1, 2] are as follows: 1) Since the disparity information is encoded in the EPI, the non-local attention mechanism is performed in the 2D epipolar plane rather than the entire 3D space; 2) We model the light field reconstruction with pixel shuffling [61] that disentangles the reconstructed angular information from the channel dimension.

A straightforward choice to perform spatial-angular attention is to embed the attention module in each resolution scale of the U-net. However, implementing non-local perception in the full resolution light field (feature map) is intractable due to the high computation complexity and GPU memory cost. Alternatively, we insert the proposed SAAM between the encoder and decoder, i.e., to perform non-local operation in the low-resolution feature space, as shown in Fig. 3.

Since features in a 3D CNN will be a 5D tensor $\phi\in\mathbb{R}^{B\times W\times H\times A\times C}$ (i.e., batch, width, hight, angular and channel), we first apply two convolution layers with kernel size $1\times 1\times 1$ to produce two feature layers $\phi_{q}$ and $\phi_{k}$ with size of $B\times W\times H\times A\times C^{\prime}$. The channel number $C^{\prime}$ is set to be $\frac{C}{8}$ (i.e., $C^{\prime}=6$ in our implementation) for computation efficiency. Then the feature layers $\phi_{q}$ and $\phi_{k}$ are reshaped into 3D tensors $\phi^{\prime}_{q}$ and $\phi^{\prime}_{k}$ of size $BH\times WA\times C^{\prime}$ and $BH\times C^{\prime}\times WA$, respectively. In this way, we merge the angular and width dimensions ($s$ and $x$ or $t$ and $y$ in a light field) together to implement the non-local perception on the epipolar plane.

We apply batch-wise matrix multiplication between $\phi^{\prime}_{q}$ and $\phi^{\prime}_{k}$ and use a softmax function to produce an attention map $M$ as illustrated in Fig. 3(b). The attention map is composed of $BH$ matrices with shape $WA\times WA$. Each matrix can be considered as a 2D expansion map of a 4D tensor $M^{\prime}\in\mathbb{R}^{W\times A\times W\times A}$ (the batch and height dimensions are neglected). The point $M^{\prime}(x_{0},s_{0},x_{1},s_{1})$ indicates the response of light field position $L(x_{0},y,s_{0})$ to position $L(x_{1},y,s_{1})$ in the latent space. In other words, the attention map is able to capture correspondence among all the views in the input 3D light field.

We demonstrate the non-local perception of the proposed SAAM by visualizing a part of the attention map before the softmax function as shown in Fig. 4. In this example, there are two points $P_{A}$ and $P_{B}$ with remarkable visual features as shown in Fig. 4(a), and their corresponding points in other views are marked as $P_{A}^{\prime}$ ($P_{A}^{\prime\prime}$) and $P_{B}^{\prime}$ ($P_{B}^{\prime\prime}$). As the viewpoint changes along the angular dimension, the background point $P_{A}$ will be occluded by the foreground point $P_{B}$, which is demonstrated more obviously in Fig. 4(b). Fig. 4(c) shows three sub-maps extracted from the attention map $M^{\prime}$ with $s_{0}=1$ and $s_{1}=1,2,3$, respectively. It can be clearly seen that a point will have the highest response at the location of its correspondence in the attention map. For instance, the response $R(P_{B},P_{B}^{\prime})$ at the location $M^{\prime}(11,1,9,2)$ for the corresponding patch $(P_{B},P_{B}^{\prime})$ (the middle sub-figure of Fig. 4(c)), and the response $R(P_{B},P_{B}^{\prime\prime})$ at the location $M^{\prime}(11,1,7,3)$ for the corresponding patch $(P_{B},P_{B}^{\prime\prime})$ (the right sub-figure of Fig. 4(c)). For the occluded point $P_{A}$, the location of the maximum response changes from $M^{\prime}(5,1,5,1)$ to $M^{\prime}(5,1,7,3)$. In this case, the attention module is able to locate the occluded point $P_{A}^{\prime\prime}$ through its surrounding pixels. More demonstrations of spatial-angular attention map can be found in Sec. VII-A.

Feature $\phi_{v}$ and $\phi_{b}$ are obtained by another two $1\times 1\times 1$ convolutions in a similar manner as that for $\phi_{q}$ and $\phi_{k}$. The main difference is that the channel numbers of these two feature layers are $\alpha C^{\prime}$, where, $\alpha$ denotes the reconstruction factor (upsampling scale in the angular dimension) of the network. Another batch-wise matrix multiplication is applied between the attention map $M$ and $\phi^{\prime}_{v}$ (reshaped from $\phi_{v}$), resulting a 3D tensor $\phi^{\prime}_{a}\in\mathbb{R}^{BH\times WA\times\alpha C^{\prime}}$. We then reshape $\phi^{\prime}_{a}$ into a 5D tensor $\phi_{a}\in\mathbb{R}^{B\times W\times H\times A\times\alpha C^{\prime}}$.

Using the aforementioned SAAM, the reconstructed angular information can be well encoded in the channel dimension of the 5D tensor. By disentangling it with channel-to-angular pixel shuffling, we can reconstruct a high angular resolution light field (feature map) in a non-local manner. Specifically, we first multiply the feature layer $\phi_{a}$ by a trainable scale parameter (initialized as 0) and add back to the feature layer $\phi_{b}$. We then apply the channel-to-angular pixel shuffling and reconstruct a 5D tensor $\phi_{c}\in\mathbb{R}^{B\times W\times H\times(\alpha(A-1)+1)\times C^{\prime}}$. As illustrated in Fig. 5, the channel-to-angular pixel shuffling rearranges elements from channel dimension to angular dimension²²2Different from typical spatial super-resolution, upsampling a light field by $\alpha\times$ generates a light field of $\alpha(A-1)+1$ views. So we simply remove the last $\alpha-1$ views from the reconstructed light field.. The final output of the SAAM is generated by two convolutional layers with kernel sizes of $7\times 1\times 1$ and $1\times 1\times 7$, respectively.

By combining the proposed SAAM with the feature maps in the skip connections, the network is able to reconstruct light field with view consistency while also preserving the high frequency components. Detailed parameter setting of the SAAM is listed in Table II.

Typical learning-based light field reconstruction or view synthesis methods optimize the network parameters by formulating a pixel-wise loss between the inferred image and the desired view (or EPI [10]). Recently, researches [33, 62, 63, 14] show that formulating the loss function in the high-level feature space will motivate the restoration of high-frequency components. This high-level feature loss, also known as perceptual loss, can be computed from part of the feature layers in the autologous network [17] or other pre-trained networks [18], such as the commonly-used VGG network [19].

In this paper, we propose a spatial-angular perceptual loss that is specifically designed for the high-dimensional light field data. Existing approaches [14, 63, 29] for light field reconstruction apply perceptual loss between 2D sub-aperture images, neglecting the view consistency constraint in the angular dimension. Alternatively, we propose to use a 3D light field encoder to map the 3D light fields into high-dimensional feature tensors (width, height, angular and channel).

To achieve this, we design another 3D encoder-decoder network (auto-encoder) that learns to extract the high-level features for the proposed spatial-angular perceptual loss³³3The architecture of the 3D auto-encoder for the perceptual loss is different with that of the SAA-Net.. Note that the auto-encoder can be also generalized to a 4D form. But given that some light field datasets have only one angular dimension (e.g., light fields from gantry system in [15]) and the proposed SAA-Net also takes 3D light field as its input, we only adopts 3D convolution in the encoder and decoder.

The proposed auto-encoder for the spatial-angular perceptual loss uses the output of the SAA-Net or the desired 3D light field (ground truth) as input, as shown in Fig. 6. It has 12 convolutional layers ($\phi_{ae}^{(l)},l=1,2,\cdots,12$) with kernel size $3\times 3\times 3$ and 3 bilinear upsampling layers. The encoder part includes the first 6 convolutional layers, where 3 convolutional layers ($\phi_{ae}^{(2)},\phi_{ae}^{(4)},\phi_{ae}^{(6)}$) with stride 2 in each dimension are used to compress the light field from low-level pixel space into high-level feature space. The decoder part includes the last 6 convolutional layers and 3 upsampling layers to restore the light field from the latent representations. Detailed configuration of each layer (stride and number of channels) is shown in Fig. 6.

The auto-encoder learns how to extract high-level features through unsupervised learning, i.e., the network learns to predict the input 3D light field. The objective for training the auto-encoder is $\mathcal{L}_{AE}(L_{HR})=\|f_{AE}(L_{HR})-L_{HR}\|_{1}$, where $f_{AE}$ denotes the auto-encoder. We use the same training dataset (Sec. V-B), learning rate and optimizer (Sec. V-C) as those for the SAA-Net.

To compute the final spatial-angular perceptual loss, we feed the output of the SAA-Net $\hat{L}_{HR}$ as well as the ground truth light field $L_{HR}$ to the auto-encoder, as shown in Fig. 6. And the spatial-angular perceptual loss is defined as follows

where $\phi_{ae}^{(l)}(\cdot)$ $(l={2,4,6})$ denotes the feature maps obtained from the $l$th layer in the encoder, and $\lambda_{feat}=0.2,0.2,0.1$ is a set of hyperparameters for the proposed spatial-angular perceptual loss.

To prevent the potential possibility that different light field patches are mapped to the same feature vector [17], our loss function also contains a pixel-wise term $\mathcal{L}_{pix}$ using Mean Absolute Error (MAE) between $\hat{L}_{HR}$ and $L_{HR}$, i.e.,

Then the final loss function $\mathcal{L}_{SAA}$ for training the SAA-Net is defined as

The two terms are weighted by the set of hyperparameters $\lambda_{feat}$ in the perceptual loss.

We use light fields from the Stanford (New) Light Field Archive [64] as the training dataset, which contains 12 light fields⁴⁴4The light field Lego Gantry Self Portrait is excluded from the training dataset since the moving camera may influence the reconstruction performance. with $17\times 17$ views. Since the network input is 3D light fields, we can extract 17 $L(x,y,s)$ and 17 $L(y,x,t)$ in each 4D light field set. Similar with the data augmentation strategy proposed in [38], we augment the extracted 3D light fields using shearing operation [65]

where $S$ is the angular resolution of the 3D light field $L(x,y,s)$, and $L_{d}(x,y,s)$ is the resulting 3D light field with shear amount $d$. $L_{d}(y,x,t)$ can be obtained following a similar manner. In practice, we use two shear amounts $d=\pm 2$. The shearing-based data augmentation increases the number of training examples by 2 times. More importantly, the disparity effects in the augmented light field will be more obvious as shown in Fig. 7, enabling the network to address the large disparity problem.

To accelerate the training procedure, the extracted 3D light fields are cropped into sub-light fields with a spatial resolution of $64\times 24$ (width and height for $L(x,y,s)$ or height and width for $L(y,x,t)$) and a stride of 40 pixels. About $6.7\times 10^{5}$ examples can be extracted from the 3D light fields (original and sheared).

We train two models with reconstruction factors $\alpha=3,4$. The input/output angular resolution of the training samples for these two models are 6/16 and 5/17, respectively. Although the reconstruction factor of the network is fixed, we can achieve a flexible upsampling rate through network cascade. The training is performed on the Y channel (i.e., the luminance channel) of the YCbCr color space. We initialize the weights of both convolution and deconvolution layers by drawing randomly from a Gaussian distribution with a zero mean and a standard deviation of $1\times 10^{-3}$, and the biases by zero. The network is optimized by using ADAM solver [66] with learning rate of $1\times 10^{-4}$ ($\beta_{1}=0.9$, $\beta_{2}=0.999$) and mini-batch size of 28. The training model is implemented in the Tensorflow framework [67]. The network converges after $8\times 10^{5}$ steps of backpropagation, taking about 35 hours on a NVIDIA Quadro GV100.

In this experiment, the comparisons are performed on light fields from the MPI Light Field Archive [16] ($1\times 101$ views of resolution $960\times 720$, $1\times 97$ views for evaluation) and the CIVIT Dataset [15] ($1\times 193$ views of resolution $1280\times 720$) with upsampling scales $8\times$ and $16\times$. The performances with respect to both angular sparsity and non-Lambertian are taken into consideration. Since the vanilla version of the network by Yeung et al. [11]⁵⁵5In the modified implementation, every 8 (6) views are applied to reconstruct (synthesize) a 3D light field of 22 (21) views for the networks of reconstruction factor $\alpha=3$ ($\alpha=4$). and Meng et al. [63] (HDDRNet) were specifically designed for 4D light fields, we modify their convolutional layers to fit the 3D input while keeping its network architecture unchanged. The networks by Kalantari et al. [7], Yeung et al. [11], Mildenhall et al. [14] (LLFF) and Meng et al. [63] (HDDRNet) are re-trained using the same training dataset as our SAA-Net. Due to the particularity of the training datasets in [44], we do not retrain the DA²N. We perform network cascade to achieve different upsampling scales, i.e., two cascades for $8\times$ ($16\times$) upsampling using a network of reconstruction factor $\alpha=3$ ($\alpha=4$).

Fig. 8 shows the reconstruction results on three light fields, Bikes, FairyCollection and WorkShop, from the MPI Light Field Archive [16] with upsampling scale $16\times$ (disparity range up to 33.5px). The first and the third cases have complex occlusion structures, as shown in the top and the bottom row in Fig. 8. The baseline methods [7, 10, 11, 14] fail to reconstruct the complex structures. Among them, the depth and learning-based approach [7] and the MPI-based approach [14] fail to estimate proper occlusion relations between the foregrounds and the backgrounds. The second scene is a non-Lambertian case, i.e., a refractive glass before the toys. The approach by Kalantari et al. [7] cannot reconstruct the refractive object. And the EPIs reconstructed by the baseline methods [10, 11] appear severe aliasing effects due to the limited receptive field. The MPI-based approach LLFF [14] is able to reconstruct the non-Lambertian effects in this case, but produces ghosting artifacts as marked by the red arrow in Fig. 8.

Fig. 9 shows the reconstruction results on three light fields, Castle, Holiday and Flowers, from the CIVIT Dataset [15] with upsampling scale $16\times$ (disparity range about 14px). The first case and the third case have thin structures with complex occlusions. The depth and learning-based approach by Kalantari et al. [7] fails to estimate depth maps accurate enough to warp the input images, and the color CNN cannot correct the misaligned views, producing ghosting artifacts as shown in Fig. 9. For the second case, we demonstrate reconstructed EPIs in a highly non-Lambertian region. Caused by the depth ambiguity, the approach by Kalantari et al. [7] produces choppiness artifacts along the angular dimension. Due to the limited receptive field of the networks, the results by Wu et al. [10] and Yeung et al. [11] show aliasing effects in various degrees. In the demonstrated cases, LLFF [14] assigns wrong planes to the tiny structures, leading to ghosting and tearing artifacts.

Table III and Table IV list the quantitative measurements ($8\times$ and $16\times$ upsampling scales) on the light fields from the MPI Light Field Archive [16] and the CIVIT Dataset [15], respectively. Compared with the baseline approaches, the proposed SAA-Net shows superior performance on both light field datasets.

We evaluate the proposed approach using three Lytro light field datasets (113 light fields in total), the 30 Scenes dataset by Kalantari et al. [7], and the Reflective and Occlusions categories from the Stanford Lytro Light Field Archive [69]. In this experiment, we reconstruct a $7\times 7$ light field from $3\times 3$ views ($3\times$ upsampling) and a $8\times 8$ light field from $2\times 2$ views ($7\times$ upsampling). We compare our SAA-Net with 8 learning-based approaches, including 3 depth-based approaches (Kalantari et al. [7], LLFF [14] and Meng et al. [29]) and 5 approaches without explicit depth (Wu et al. [10], Wang et al. [42], Yeung et al. [11], HDDRNet [63] and DA²N [44]). Since the vanilla versions of the networks in [7, 11, 42, 63, 29] are trained on Lytro light fields, we use their open-source model (parameters) without re-training. The networks by Wu et al. [10] and Mildenhall et al. [14] (LLFF) are re-trained using the same dataset introduced in Sec. V-B. Note that the proposed network is not fine-tuned on any Lytro light field datasets, and the results are produced by the same set of network parameters for both $3\times$ and $7\times$ upsampling scales.

Table V lists the quantitative results on the evaluated Lytro light fields. The proposed SAA-Net shows competitive performance compared with the state-of-the-art light field reconstruction approach by Yeung et al. [11]. For the evaluated 113 light fields, the average PSNR / SSIM values of the proposed network are 42.55 / 0.984 for $3\times$ upsampling and 36.55 / 0.969 for $7\times$ upsampling. In comparison, the average PSNR / SSIM values of the baseline approaches with the highest performance are 41.82 / 0.984 (DA²N [44]) for $3\times$ upsampling and 35.81 / 0.938 (Meng et al. [29]) for $7\times$ upsampling. Since our network is not re-trained or fine-tuned on any Lytro light field dataset, these experimental results clearly demonstrate that our network can generalize well on light fields captured by different acquisition geometries.

We demonstrate two cases with relatively large disparities (maximum disparity up to 13px), IMG1743 from the 30 Scenes [7] and Occlusions 23 from the Occlusions category [69], as shown in Fig. 10. In both cases, the reconstruction results by Wu et al. [10] and Yeung et al. [11] show ghosting artifacts around the region with large disparity (background in the IMG1743 case, and foreground in the Occlusions 23 case) due to their limited receptive fields. The depth and learning-based approaches by Kalantari et al. [7] and Mildenhall et al. [14] (LLFF) produce promising results in the first case, but appear tearing artifacts near the occlusion boundaries as marked by the red arrows in the EPIs. In the second case, the approach by Kalantari et al. [7] fails to estimate proper depth information, introducing misalignment as shown by the EPI. LLFF [14] produces ghosting effects due to the incorrect plane assignments around the background region, as shown by the red arrows in the figure. In comparison, the proposed SAA-Net provides reconstructed light fields with higher view consistency (as shown in the demonstrated EPIs).

In this experiment, we empirically validate the modules and losses in our SAA-Net by performing the following ablation studies on different datasets. The results are listed in the last five rows in Table IV, III and V. First, we evaluate the network with increasing series of the MSR structure while removing the SAAM. We use “MSR structure-$X$” to represent the network with $X$ series of MSR structure. By increasing the series, the network will have a larger receptive field, e.g., the theoretical receptive field size of the MSR structure-4 reaches 247 pixels in the spatial dimension. However, since it is intractable to increase the actual size of the receptive field simply by using a deeper network [59], purely increasing the series of MSR structure has limitations in improving performance. This point is also verified by the ablation study. In comparison, the non-local attention is more effective than simply increasing the receptive field of the network.

In the second ablation study, we use a typical 3D U-net with the same convolutional layers as the backbone and remove the deconvolution layer in the skip connections, denoted as “w/o MSR structure” for short. The angular reconstruction is simply achieved by using deconvolution at the end of the network. The network parameters are kept comparable to the SAA-Net by adjusting the channel dimension of the network. For light fields from gantry systems [16, 15], the performance of the network decreases more than 2dB in terms of PSNR, as shown in Table IV and III. For light fields from Lytro Illum, the performance of the network decreases more than 1.3dB, as shown in Table V.

In the last ablation study, we train the proposed SAA-Net simply by using the pixel-wise term (MAE loss) without the proposed spatial-angular perceptual loss, denoted as “w/o SAP loss” for short. The performance (PSNR) decreases around 0.3dB as shown in Table IV, III and V.

In Fig. 11, we also visualize the comparison ($16\times$ upsampling) between the proposed SAA-Net, the network with different series of the MSR structure while removing the SAAM and the network without the MSR structure. The evaluated light fields are from the MPI Light Field Archive [16]. The results show that without using the proposed modules, especially the MSR structure, the SAA-Net appears severe aliasing effects around region with large disparity, e.g., the occluded leaves in the FairyCollection case and the hand writing on the reflective board in the Mannequin case.

We visualize four additional attention maps (before the softmax function) on scenes with small disparity, large disparity, occlusion and non-Lambertian effect, as shown in Fig. 12. The first case (Fig. 12(a)) shows a scene with a relatively small disparity (about 7 pixels). Due to the spatial downsampling operation in the MSR structure, the disparity of the light field features in the SAAM is about 1.75 pixels, as shown in the top left figure in Fig. 12(a). Three sub-maps $M^{\prime}(x_{0},s_{0},x_{1},s_{1}),s_{0}=2,s_{1}=1,2,3$ are shown in the bottom of Fig. 12(a), which visualize the correspondence captured by the attention mechanism. As we can see, the response with the highest value moves from $R(P_{A}^{\prime},P_{A})$ at position $M^{\prime}(11,2,12,1)$ to $R(P_{A}^{\prime},P_{A}^{\prime\prime})$ at position $M^{\prime}(11,2,9,3)$ along the angular dimension. In addition, the attention map also shows a high response value $R(P_{A}^{\prime},P_{B})$ at position $M^{\prime}(11,2,13,1)$ due to the fractional disparity. This indicates that the proposed SAAM has potential to capture a correspondence with sub-pixel accuracy.

In the second case (Fig. 12(b)), we demonstrate the spatial-angular attention on a scene with a large disparity (about 16 pixels). The spatial downsampling operation in the MSR structure reduces the disparity to 4 pixels. The response in the attention map moves from $R(P_{A}^{\prime},P_{A})$ at position $M^{\prime}(10,2,14,1)$ to $R(P_{A}^{\prime},P_{A}^{\prime\prime})$ at position $M^{\prime}(10,2,6,3)$ along the angular dimension. Although the SAAM only applies $1\times 1\times 1$ convolutions before the attention, it is capable to capture correspondence with large displacement.

Fig. 12(c) visualize the attention map of a scene with a complex occlusion structure (Mannequin in the MPI Light Field Archive [16]). The background white board is occluded by the foreground leaves (please refer to the sub-aperture image of the second row in Fig. 11). As shown by the demonstrated attention map in Fig. 12(c), the SAAM fails to capture a correct correspondence because of the occlusion. The highest response value is $R(P_{A}^{\prime},P_{B})$ at position $M^{\prime}(8,2,10,1)$, which is also a background point like $P_{A}$. However, the SAAM also generate a considerable response $R(P_{A}^{\prime},P_{A})$ at position $M^{\prime}(8,2,8,1)$, which we speculate is inferred from other non-occluded background points.

The fourth case in Fig. 12(d) demonstrates the spatial-angular attention on a scene with non-Lambertian effect. In this case, the positional relation between the corresponding points $P_{A}$, $P_{A}^{\prime}$ and $P_{A}^{\prime\prime}$ does not follow a clear depth cue, as clearly shown in the top right figure of Fig. 12(d). Analogously, the responses do not follow a regular disparity pattern along the angular dimension, as visualized by the at the bottom of Fig. 12(d). This result shows that the proposed SAAM is able to catch the correspondences even for regions with non-Lambertian effects.

Although we propose a multi-scale reconstruction structure to save GPU memory cost, the SAAM will still occupy a large GPU memory when dealing with an input light field with high spatial-angular resolution. For example, when reconstructing light fields from the MPI Light Field Archive [16] (spatial resolution $960\times 720$), we have to disassemble the 3D data into sub-light fields of resolution $960\times 24\times 25$ (width $\times$ height $\times$ angular). Our investigation shows that the disassembling will cause quality degradation on the reconstructed results.

We therefore apply the truncated Singular Value Decomposition (SVD) [70] to compact the 3D tensor $\phi^{\prime}_{q}\in\mathbb{R}^{BH\times WA\times C^{\prime}}$ and $\phi^{\prime}_{k}\in\mathbb{R}^{BH\times C^{\prime}\times WA}$ before computing the attention map

where $\tilde{\phi}$ denotes $\phi^{\prime}_{q}$ or $\phi^{{}^{\prime}T}_{k}$, $U$ and $V$ are two orthogonal matrices (ignoring the batch dimension), and $S$ is a diagonal matrix with singular values along its diagonal. By truncating the diagonal matrix $S$ with the largest $\tau$ singular values, we can get a good approximation $\tilde{\phi}\approx US_{\tau}V^{T}$ and also compress the 3D tensors. Since the rank of the matrices are $C^{\prime}=6$, the parameter of the truncated SVD $\tau=[1,2,\cdots,C^{\prime}]$.

Fig. 13 shows the performance on PSNR in function of the SVD decomposition using the largest $\tau=[1,2,\cdots,6]$ singular values. As we can see from the “SVD” curve, with no less than three singular values, the SVD decomposition will maintain the network performance without using fine-tuning. Moreover, since the decomposition enables us to feed the network with higher spatial resolution input, e.g., from $960\times 24\times 25$ to $960\times 64\times 25$, we can obtain a reconstruction result with even higher quality (0.04dB higher) when employing truncated SVD decomposition, as shown by the “SVD-Ext” (green dot) in the figure.

The non-local attention involves outer product of large scale matrices, especially for the high-dimensional light field data. For this reason, the proposed network takes almost 15% of its inference time on the SAAM. For the 3D light field from the MPI Light Field Archive [16], the network takes about 51 seconds to reconstruct a $1\times 97$ light field from $1\times 7$ views of spatial resolution $960\times 720$ ($16\times$ upsampling), i.e., 0.53s per view. For the 3D light field from the CIVIT Dataset [15], the network takes about 126 seconds to reconstruct a $1\times 193$ light field from $1\times 13$ views of spatial resolution $1280\times 720$ ($16\times$ upsampling), i.e., 0.65s per view. For a 4D light field from Lytro Illum, it takes about 17 seconds to reconstruct a $7\times 7$ light field from $3\times 3$ views of spatial resolution $536\times 376$ ($3\times$ upsampling), i.e., about 0.35s per view. And the reconstruction of a $8\times 8$ Lytro light field from $2\times 2$ views ($7\times$ upsampling) takes about 30 seconds, i.e., about 0.5s per view. The parameter number of our SAA-Net is about 338K. The above evaluations are performed on an Intel Xeon Gold 6130 CPU @ 2.10GHz with an NVIDIA TITAN Xp.

Although we apply a simple SVD decomposition to accerlate the network and compact the 3D tensor, the compression rate is limited by the rank of the matrices. Decomposing the attention map into the combination of small tensors [71] might solve this problem in a more essential way.

The another limitation of our proposed method is that repetitive patterns in the input light field can cause multiple plausible responses in the attention map, leading to misalignments in the reconstructed light fields. A possible solution is to introduce a smooth term in the attention map as in [54] to penalize multiple responses.