LF Tracy: A Unified Single-Pipeline Paradigm for Salient Object Detection in Light Field Cameras

The SOD task can be traced back to rule-based methodologies, which predominantly relied on visual attributes such as color, contrast, and spatial distribution to ascertain salient areas in images. In recent years, there has been a paradigm shift in the SOD community towards leveraging deep learning paradigms. Specifically, MENet [31] introduced iterative refinement and frequency decomposition mechanisms to improve detection accuracy. By utilizing transformer and multi-scale refinement architecture, Wang et al. [9] used high- and low-resolution images to achieve SOD. Furthermore, Zhang et al. [7] implemented SOD for panoramic images. Apart from those single-modality SOD networks, depth information is introduced to enhance performance, whereas multi-model fusion strategies [3, 8] are employed for RGB and thermal data.

For the SOD task of LF, Wang et al. [28] implemented a dual-pipeline neural network in the SOD community. Since then, the two-stream approach [21] for processing LF images has stood in a leading position in this field. Typically, this involves employing one backbone for processing AF images and another for FS images or the depth image extracted from LF sub-aperture images. Although the two-stream approach has seen considerable advancement in various tasks [36], it is typically applied to modalities that are isolated, such as depth and RGB images. For light field cameras, the depth, angular, and spatial information are embedded across different representations, i.e., AF images and FS images. Processing these images in an isolated manner buries the angular features of light field cameras, and thus remains a sub-optimal method [33].

Data augmentation (DA) has been thoroughly explored in various vision tasks such as image recognition, image classification, and semantic segmentation, proving effective in enhancing network performance and mitigating the issue of overfitting. The traditional data augmentation strategies can be roughly divided into five categories based on the adjusted purpose. 1) Flipping the image along its vertical and horizontal axis is a typical technique for increasing the diversity of data available for training. Furthermore, rotating an image at a certain angle is also a contributing factor. 2) Color jitter simulates images under different lighting and camera settings, enabling the trained model to better adapt to various scenarios. 3) Cutout [10] is introduced to drought or mismatch part of pixel-level information between neighboring pixels to increase the discrimination capability of the network. 4) Beyond deep learning, several works [11, 35] introduced machine learning-based strategies to boost the network capability. 5) Mixing-based methods [33, 15] leverage information from multiple images by generating blended input images.

Those methods demonstrate noticeable performance for the single image in the augmentation community. However, for light field cameras, the subtle angular information hidden within the interplay of multiple images cannot be captured through DA applied to individual images alone. Thus, establishing data connectivity across networks becomes crucial.

As shown in Fig. 4, the proposed network has two components: a four-stage encoder providing rich multi-dimensional information from different asymmetric data and the IA Module. The IA Module serves a dual purpose: 1) It overcomes the mismatching between the features established in-network connectivity through the same encoder block. 2) It can realign these features before sending them to the prediction head. The AFttention image $I_{AF}^{m}$ and FSttention stack $I_{FS}^{m}$ are described separately to provide a more intuitive description of the information flow and interaction process. The AFttention image and FSttention stack indicate the data source after MixLD. Furthermore, for simplicity, the following description is based on the stage one, which is the same for the other three stages. Especially, by applying the encoder block, the images are transferred into AF features $({F_{AF}}{\in}\mathbb{R}^{(64{\times}64{\times}64)})$ and FS features $({F_{FS}^{n}}{\in}\mathbb{R}^{(64{\times}64{\times}64)}|{n\in[1,12]})$. After applying the IA Module, the $13$ features are aggregated into one feature $(f_{1}{\in}\mathbb{R}^{(64{\times}64{\times}64)})$, which contains all spatial, angular, and depth information. After four stages, there is a set of feature maps $\{f_{l}|l\in{[1,4]}\}$ with channel dimension $\{64,128,320,512\}$. Only $f_{1}$ is described in detail here, as the processes for the other dimensions are identical. Furthermore, at the training stage, to cooperate with the structure loss [12] calculation, $f_{l}$ is also passed through one convolutional layer to compress channel information, as in Eq. (1).

where $Conv(64,1)(\cdot)$ indicates the convolutional layer with input channel $64$ and output channel one. $f_{M_{1}}$ denotes the feature after merging at the first stage. Furthermore, drawing upon the structure loss as outlined in [32], we have integrated the Tversky Loss [26] into our training process to improve supervision during training, specifically targeting a reduction in false positives and negatives.

To fuse the implicit angular, explicit spatial, and depth information from asymmetric data, we introduce a simple IA Module that follows a two-step interaction process. Firstly, given single feature $(F_{FS}^{n}|n\in[1,12])$, the FS-guided Querry and Key are generated through their respective convolutional layer. Through matrix multiplication, the attention map $(M{\in}\mathbb{R}^{(4096{\times}4096)})$ is obtained. The attention map integrates a broader context into the aggregation of local features and enhances the representative capability of the focus part. Furthermore, applying the third convolutional layer to $F_{AF}$, the AF image guided Value $(V_{AF}{\in}\mathbb{R}^{4096{\times}64})$ is generated, as in Eq. (2)-(5). Given that the SOD task is sensitive to hyper-parameters and module design, we adopt a dimension reduction method for Query and Key. For more details on dimension reduction, please refer to Sec. 5.2.

The tokens $(T{\in}\mathbb{R}^{4096{\times}64})$, which contains information from certain focal images, is obtained by multiplication of attention map and Value, as in Eq. (6).

After obtaining the tokens, the A-FS features $\hat{F}_{FS}^{n}$ are generated by applying the reshape operation. Note that the number of images has remained unchanged until now. This operation aims to enhance the spatial information at the corresponding depth by guiding the information from the FS image and, with the help of AF features, establish a connection between the global AF and FS information. Secondly, we introduce a set of leanable parameters to calculate the contribution of different FS features. To further enhance the spatial context information, a submission is undertaken, and the final result $f_{1}$ is obtained as in Eq. (7).

Given multi-scale features $\{f_{1},f_{2},f_{3},f_{4}\}$, interpolation and concatenation are conducted. Finally, by applying the convolutional layer following an interpolation, the mask $f$ is compressed and sent to the prediction head.

As depicted in Fig. 5, the primary objective of the specific data augmentation strategy for the LFSOD task is to amalgamate distinct representations inherent in light field camera, namely, AF image $(I_{AF})$, FS $(I_{FS}^{n}|n\in[1,12])$, and implicit angular information. This strategy is methodically partitioned into two discrete phases, each targeting specific aspects of the integration process. Initially, a non-intrusive approach is employed to integrate angular and depth information into the composite AF image while preserving the integrity of spatial data dimensions. Specifically, the data augmentation strategy can be described as following steps:

Firstly: (FS2AF) Following the FS setting [25], one FS slice $I_{FS}^{n}$ with dimension $\{3{\times}256{\times}256\}$ is randomly selected with a likelihood of $0.1$. This FS image is then subjected to a pixel-level fusion process, meticulously blending it into the AF image representation, as shown in Eq. (8).

where $\alpha$ denotes the degree of blending and $n$ indicates the quantities of focal images. $I_{AF}^{m}$ indicates the AF image after blending, i.e., AFttention image. In MixLD, $\alpha{=}1$ indicates no blending and $\alpha{=}0$ indicates that the AF Image is completely replaced. Only the AF image is altered during this process, while the FS images remain unchanged. Meanwhile, this procedure is not conducted for each interaction.

Secondly: (AF2FS) The AFttention image is integrated into all the FS images with a probability of $0.5$, as in Eq. (9).

where $\beta$ denotes also a super parameter for the degree of blending in stage two and $n$ denotes the quantities of focal images. $I_{F_{n}}^{m}$ indicates the FS after blending i.e., FSttention stack. This integration carried out with a fusion probability of $0.5$ instead of $0.1$, aims to make it more possible to enrich the FS with additional information. By blending the AF image into the FS images, each focal image retains its inherent depth information, gains implicit angular insights from the other focal image, and enhances its spatial geometric information from the AF image. Furthermore, the AFttention image $I_{AF}^{m}$ and FSttention stack $(I_{F_{n}}^{m}|n\in[1,12])$ are fed into the network. It is important to emphasize that both phases (FS2AF and AF2FS) of MixLD are conducted randomly. It is possible for data interaction to occur in only one phase, while the other remains non-interactive.

It is precisely through this form of blending that the neural network while learning the inherent AF and FS information, can break out of the conventional framework to learn implicit angular information. For detailed algorithms, please refer to the pseudocode presented in the Appendix.

Datasets: The experiments are conducted following the benchmark proposed by the PKU team [17]. The datasets involve traditional LFSOD datasets, which include LFSD [22], DUT-LF [41], HFUT [38] and a large-scale PKU dataset [17]. The images within the PKU dataset are sourced from terrestrial and aquatic environments. Two experiment strategies are conducted: I) training on DUT-LF $+$ HFUT, ${\sim}1000$ images, and evaluation on the whole LFSD dataset, the DUT-LF testing dataset, and HFUT testing dataset; II) training and testing on the PKU-LF dataset. PKU-LF dataset contains more than $10K$ images. For the ablation study, the experiments are based on experiment strategy one.

Setting Details: The image size for all the datasets is $256{\times}256$. Each scene is structured to contain exactly $12$ focal slices to meet specific coding requirements. This is achieved by strategically duplicating focal slices in the original order. Data augmentation is applied with Flipping, Cropping, Rotating, and MixLD for the training process. The blending parameter $\alpha,\beta$ are set into $0.5$ and $0.5$, respectively. The AdamW optimizer with a learning rate of $5e^{-5}$ and weight decay of $1e^{-4}$ is adapted for training. All the experiments are conducted on one A6000 GPU with a batch size of $6$. The training epochs are limited to $300$.

Evaluation Metrics: To analyze the results of different methods, we employ mean absolute error (MAE) [24] for a fair comparison. For F-measure ($F_{\beta}^{mean}$) [1], E-measure ($S_{\beta}^{man}$) [12], S-measure ($S_{\alpha}$), we compare them with the previously best methods.

To verify the efficiency of the approach, we compare the designed network with existing methods. Table 1 shows that the best performance of the proposed approach significantly outperforms existing methods across the LFSD series dataset [22, 41, 38] and PKU dataset [17] on MAE. Due to the variability in performance across different evaluation metrics and datasets, we follow the benchmark provided by the PKU team [17].

The proposed method significantly surpasses this integrated benchmark. The network’s performance is most effectively proved, particularly with the large-scale and richly varied PKU dataset. By establishing the pre-network connectivity and the in-network connectivity of LF data, the network reconnects the intrinsic relationships between different light field camera images, achieving a $23\%$ improvement in MAE compared with STSA₃. It should be noted that the training dataset of STSA₃ is an extension dataset (DUT-LF $+$ HFUT $+$ PKU-LF). We used the PKU-LF dataset, and the network performance still exceeded by 23%. In Table 2, we perform a comparison in terms of other evaluation criteria following PKU team [17]. While other networks may perform well in certain respects, LF Tracy still surpasses previous methods on a majority of metrics. This fully demonstrates the network’s superior comprehensive perception capabilities without being data-dependent.

It can be seen from Fig. 7 that the LF Tracy achieves outstanding accuracy across different scenarios by establishing intra-network and inter-network connectivity. Whether dealing with a single scene or complex scenarios, the network delivers excellent visualization results. Especially, for transparent backboards under varying lighting conditions, the network identifies the object through efficient information processing. Meanwhile, thin structures have always been a challenging issue in SOD tasks, yet the network has successfully identified both the necks of animals and the slender support poles of basketball hoops. Furthermore, the visual comparison results demonstrate the method’s superiority, as in Fig. 7. The proposed network accurately identifies the locations of objects, and notably, it precisely identifies challenging boundaries and lines. For the images in the middle row, the area with two pedestrians walking side by side is particularly challenging to discern. The varied colors and textures of their clothing present a significant challenge to the network. While other methods show numerous errors in this region, the proposed network achieves accurate identification.

In the experimental analysis, as shown in Table. 3, we ablated components of the approach to assess their contributions. The optimal performance achieved an MAE of $0.046$. Firstly, eliminating the data augmentation strategy MixLD resulted in a performance decrease, and adapting CutMib [33] has few contributions to the performance. This indicates the necessity of MixLD to connect the different data before sending them into the network. After that, we ablate the core component of the network, the IA module, and the multi-scale features are directly fused. The MAE dramatically increased. The observed significant disparity of $0.286$ highlights the effectiveness of the IA module. This module is integral for effectively realigning and managing the data imbalance across diverse sources. In particular, it is pivotal in reducing data mismatching between LF and AF images, facilitating more effective data integration, and improving accuracy with one stream encoder. Finally, without FS, the result is further reduced.

Parameters Analysis: The total Parameters of the designed LF Tray are $30M$. After removing the first stage in the IA module, the parameters decrease to $27M$. Furthermore, removing the entire IA module, the parameters fall into $24M$. With only 6$M$ parameters, the network is capable of intra-network data connections and efficient feature fusion. GFlops and FPS: When processing 12 FS images, i.e., handling a total of 13 light field images in a single training flow, the GFLOPs and FPS are 104.13 and 4.28, respectively. Without the IA module, these values are 84.8 GFLOPs and 4.73 FPS.

To demonstrate the contribution of the FS stack and the alignment and fusion capabilities of the IA module for asymmetric data, the ablation studies are conducted from three different aspects.

➀ Focal Stack Images: We compared the discrimination ability of the network with and without the IA module, using $2$, $3$, $5$, and $12$ FS images, respectively. As indicated in Table. 4, without the IA module, continuously stacking FS images does not enhance the network’s capability; rather, it negatively impacts the network. With the addition of the IA module, the focused range and implicit angular information in the FS are utilized, increasing the network’s discrimination ability.

➁ Fusion strategy in IA module: In Table 6, four different fusion strategies are compared. Firstly, we introduced an attention-based feature interaction process, accompanied by a set of learnable parameters, to achieve the fusion of information from different data sources. Then, we replaced this process with deformable cross attention [44]. Subsequently, we directly add the features point by point. Finally, we utilized cross-attention for feature interaction, directly adding the interacted feature maps. Although the point-by-point addition method has achieved significant results in semantic segmentation tasks, it does not work effectively for SOD tasks. Likewise, the method of deformable cross attention also did not surpass the method we proposed.

➂ Reduction rate: Last but not least, a set of experiments are conducted to deeply access the better hyper-parameters within IA module. Inspired by [19], the dimensions of the Query and Key are compressed in the IA module. Four different reduction rates are chosen. As shown in Table 6, over-reducing or under-reducing the channel can lead to performance degradation. The best option is to reduce the query and key dimensions to $1/8$ of the original size.

We conducted a series of experiments based on traditional datasets to assess the optimal feature extraction backbone. The PVTv2 [30] and the agent attention [19] are selected. To prevent pre-trained weights from causing an unfair comparison in the selection of backbones, we conducted experiments for $100$ epochs without pre-trained weights. Table 7 shows that the agent attention is ineffective for the dataset, and the performance on the LFSOD dataset does not improve with the increase in the number of parameters. Due to this reason, we have chosen PVTv2 as the backbone.

Interaction probability between texture and depth information: In determining the optimal combination for incorporating depth information into AF images (FS2AF) and augmenting each FS image with texture information (AF2FS). Several experiments are designed with occurrence probabilities set at 0.1, 0.5, and 0.9. From the Table. 9, it can be seen that: ➀ Assigning low occurrence probabilities (0.1) to both FS2AF and AF2FS minimally impacts the experimental outcomes, yet the performance metrics are analogous to those achieved with the CutMix augmentation technique. ➁ Excessive integration of depth information into AF images (probability set at 0.9 for FS2AF) leads to a significant loss of spatial information, affecting the network’s performance. ➂ While injecting spatial information into FS images improves the network’s ability to discriminate, excessive fusion can damage the valuable depth cues.

Blending rate analysis: To explore the optimal blending ratio of AF image and FS. We altered the parameter $\alpha$ in the first step, which involves blending one FS slice into AF images. Furthermore, in the second step, the parameter $\beta$ is adjusted to merge the blended AF image into FS. Due to the various combinations of $\alpha$ - $\beta$ pair, we only experimented with a few combinations based on $\alpha{\text{=}}\beta$. As shown in Table 9, the optimal outcome is achieved with a blending rate of $0.5$. Notably, deviations from this ratio, either by increasing or decreasing the blending rate, result in a discernible decline in performance.