Accurate RGB-D Salient Object Detection via Collaborative Learning

In this paper, we propose a novel CoNet for RGB-D SOD. The overall architecture is shown in Fig. 2. In this framework, three mutually beneficial collaborators, namely edge detection, coarse salient object detection and depth estimation, work together to aid accurate SOD through exploratory learning and timely interaction. From different perspectives of the SOD target, knowledges from edge, depth and saliency are fully exploited in a mutual-benefit manner to enhance the detector’s performance. A simplified workflow is given below.

First, a backbone network is used to extract features from original images. Five transition layers and a global guidance module (GGM) are followed to perform feature preprocessing and generate the integrated low-level feature $f_{l}$ and high-level feature $f_{h}$ (details are shown in Sec. 3.2). Then an edge collaborator is assigned to $f_{l}$ to extract edge information from the overabundant low-level feature. For the high-level feature $f_{h}$, saliency collaborator and depth collaborator work together to jointly enhance the high-level feature learning process of global semantics in a mutual-benefit manner. Finally, all learned knowledges from three collaborators ($Att_{edge}$, $Att_{sal}$ and $Att_{depth}$), as well as the integrated low-level and high-level feature ($F_{g}$), are uniformly handed to a knowledge collector (KC). Here, acting as a tutor, KC summarizes the learned edge, depth and saliency knowledges and utilizes them to predict accurate saliency results. We elaborate on the three collaborators and the KC in Sec. 3.3.

Specifically, we first formulate this problem by adding edge supervision on the top of integrated low-level feature $f_{l}$. The used edge ground truths (GT) (shown in Fig. 1) are derived from saliency GT using canny operator [4]. As shown in Fig. 2, $f_{l}$ is processed by a 1$\times$1 convolution operation and a softmax function to generate the edge map $M_{edge}$. Then, binary cross entropy loss (denoted as ${loss}_{e}$) is adopted to calculate the difference between $M_{edge}$ and edge GT. As the edge maps $M_{edge}$ in Fig. 2 and Fig. LABEL:fig:tri-att show, edge detection constraint is beneficial for predicting accurate boundaries of salient objects. Additionally, we also transfer the learned edge knowledge before the softmax function (denoted as $Att_{edge}$) to the knowledge collector (KC), where the edge information is further utilized to emphasize object boundaries. The reason why we use $Att_{edge}$ rather than $M_{edge}$ is to alleviate the negative influence brought by accuracy decrement of $M_{edge}$.

Stage one: The high-level feature $f_{h}$ is first processed by a 1 $\times$ 1 convolution operation and a softmax function to predict a coarse saliency map $S_{coarse}$. Here, binary cross entropy loss (denoted as $loss_{s}$) is used for training. Then, the learned saliency knowledge acts as a spatial attention map to refine the high-level feature in a similar way like [60]. But different from [60] which considers $S_{coarse}$ as attention map directly, we use the more informative feature map before softmax function (denoted as $Att_{sal}$) to emphasize or suppress each pixel of $f_{h}$ . Identify mapping is adopted to alleviate the errors in $Att_{sal}$ to be propagated to depth learning and accelerate network convergence. Formally, this procedure can be defined as:

$$Att_{sal}=W_{s}*f_{h}+b_{s},$$

(3)

$$\widetilde{f}_{h}=Att_{sal}\odot f_{h}+f_{h},$$

(4)

where $\odot$ means element-wise multiplication. $\widetilde{f}_{h}$ denotes the output saliency-enhanced feature.

Stage two: As pointed out in previous RGB-D researches [46, 65], the spatial information within depth image is helpful for better locating salient objects in a scene. In our network, we innovatively integrate depth learning into the high-level feature learning process, instead of directly taking depth image as input. This learning strategy enables our network to be free of using an extra depth network to make inference from an extra depth input, and thus being more lightweight and versatile. As in Fig. 2, a depth head with three convolution layers (defined as $\Psi(\cdot)$) is first used to make feature $\widetilde{f}_{h}$ adapt to depth estimation. Then, its output $\Psi(\widetilde{f}_{h})$ is followed by a 1 $\times$ 1 convolution operation to generate the estimated depth map $Att_{depth}$. Here, depth images act as GTs for supervision and we use smooth $L_{1}$ loss [22] to calculate the difference between $Att_{depth}$ and depth GT, where smooth $L_{1}$ loss is a robust $L_{1}$ loss proposed in [22] that is less sensitive to outliers than $L_{2}$ loss. Formally, the depth loss can be defined as:

$$\small Loss_{d}=\frac{1}{W\times H}\sum_{x=1}^{W}\sum_{y=1}^{H}\\ \begin{cases}0.5\times|\triangle(x,y)|^{2},&\text{if }|\triangle(x,y)|\leq 1,\\ |\triangle(x,y)|-0.5,&\text{if }\triangle(x,y)<-1\text{ or }\triangle(x,y)>1,\end{cases}$$

(5)

where $W$ and $H$ denote the width and height of the depth map. $\triangle(x,y)$ means the error between prediction $Att_{depth}$ and the depth GT in each pixel $(x,y)$. Since each channel of a feature map can be considered as a ‘feature detector’ [59], the depth knowledge $Att_{depth}$ is further employed to learn a channel-wise attention map $M_{c}$ for choosing useful semantics. Identify mapping operation is also adopted to enhance the fault-tolerant ability. This procedure can be defined as:

$$Att_{depth}=W_{d}*\Psi(\widetilde{f}_{h})+b_{d},$$

(6)

$$M_{c}=\sigma(GP(W_{c}*Att_{depth}+b_{c})),$$

(7)

$$f_{hc}=M_{c}\otimes\widetilde{f}_{h}+\widetilde{f}_{h},$$

(8)

where $w_{*}$ and $b_{*}$ are parameters to be learned. $GP(\cdot)$ means global pooling operation. $\sigma(\cdot)$ is the softmax function. $\otimes$ denotes channel-wise multiplication.

After these two stages, two collaborators can cooperatively generate optimal feature which contains affluent spatial cues and possesses strong ability to distinguish salient and non-salient regions.

As illustrated in Fig. 2, all knowledges learned from three collaborators (i.e. $Att_{edge}$, $Att_{sal}$, and $Att_{depth}$) and the concatenated multi-level feature $F_{g}=Concat(f_{l},f_{hc})$ are uniformly transferred to the KC. Those information are comprehensively processed in a triple-attention manner to give more emphasis to salient regions and object boundaries. In Fig. 2, we show a detailed diagram with visualized attention maps for better understanding. To be specific, $Att_{edge}$ and $Att_{sal}$ are first concatenated together to jointly learn a fused attention map $Att_{f}$, where the locations and boundaries of the salient objects are considered uniformly. Then, $F_{g}$ is in turn multiplied with the depth attention map $Att_{depth}$ and the fused attention map $Att_{f}$, which significantly enhances the contrast between salient and non-salient areas. Ablation analysis shows the ability of the KC to enhance the performance significantly.

There is a vital problem worth thinking about. The quality of $Att_{depth}$ and $Att_{f}$ might lead to irrecoverable inhibition of salient areas. Therefore, we add several residual connection operations [24] to the KC to retain the original features. Formally, this process can be defined as:

$$Att_{f}=\sigma(W_{f}*Concat(Att_{sal},Att_{edge})+b_{f}),$$

(9)

$$\widetilde{F}_{g}=Att_{depth}\odot F_{g}+F_{g},$$

(10)

$$F=Att_{f}\odot\widetilde{F}_{g}+\widetilde{F}_{g}.$$

(11)

In the end, $F$ is followed by a 1 $\times$ 1 convolution operation and an upsampling operation to generate the final saliency map $S_{final}$. Here, binary cross entropy loss (denoted as $loss_{f}$) is used to calculate the difference between $S_{final}$ and saliency GT. Thus, the total loss $L$ can be represented as:

$$\small L=\lambda_{e}Loss_{e}+\lambda_{s}Loss_{s}+\lambda_{d}Loss_{d}+\lambda_{f}Loss_{f},$$

(12)

where $Loss_{e}$, $Loss_{s}$, and $Loss_{f}$ are cross entropy loss and $Loss_{d}$ is a smooth $L_{1}$ loss. In this paper, we set $\lambda_{e}=\lambda_{s}=\lambda_{f}=1$ and $\lambda_{d}=3$.

To evaluate the performance of our network, we conduct experiments on seven widely used benchmark datasets.

For training, we split 800 samples from DUT-D, 1485 samples from NJUD, and 700 samples from NLPR as in [5, 6, 46]. The remaining images and other public datasets are all for testing to comprehensively evaluate the generation abilities of models. To reduce overfitting, we augment the training set by randomly flipping, cropping and rotating those images.

After introducing edge supervision (denoted as E), the boundaries of the saliency maps are sharper (b $vs$ c in Fig. LABEL:fig:ablation). The edge maps ($M_{edge}$) in Fig. 2 and Fig. LABEL:fig:tri-att also show the ability of our network in explicitly extracting object boundaries. By adding additional saliency supervision on $f_{h}$ (denoted as S), the performance can be further improved. However, by comparing (d) and (e), we can see that our mutual-benefit learning style between saliency collaborator and depth collaborator (denoted as $S_{SA}$ and $D_{CA}$) can further improve the detector’s ability to locate salient objects. This also verifies the strong correlation between saliency and depth. Finally, by using our proposed (KC), all learned edge, depth and saliency knowledges from three collaborators can be effectively summarized and utilized to give more emphasis to salient regions and object boundaries, improving the average MAE performance on two datasets by nearly 9.6% points. By comparing (e) and (f) in Fig. LABEL:fig:ablation, we can also see that salient regions in (f) are more consistent with the saliency GT and the object boundaries are explicitly highlighted benefiting from the comprehensive knowledge utilization. Those advances demonstrate that using our collaborative learning strategy is beneficial for accurate saliency prediction. We list some numerical results here for better understanding. The Root Mean Squared Error (RMSE) of depth prediction on NJUD and NLPR datasets are 0.3684 and 0.4696, respectively. The MAE scores of edge prediction are 0.053 and 0.044, respectively.

This is partly because the low-level features contain too much information and are relatively too coarse to predict saliency, and partly because the two tasks are to some extent incompatible, in which one is for highlighting the boundaries and another is for highlighting the whole salient objects. Hence, it is optimal to only add edge detection supervision on the low-level feature.
Saliency and Depth. In order to verify the effectiveness of the proposed mutual-benefit learning strategy on high-level feature $f_{h}$, we gradually add two collaborators and their mutual-benefit operations to the baseline model (c). As shown in Tab. LABEL:tab:mutual_ablation, adding saliency supervision (denoted as S) and adding depth supervision (denoted as D) are all beneficial for extracting more representative high-level semantic features. In addition, by gradually introducing our proposed mutual-benefit learning strategy between two collaborators (denoted as $S_{SA}$ and $D_{CA}$), spatial layouts and global semantics of high-level feature can be greatly enhanced, which consequently brings additional accuracy gains on both datasets. These results further verify the effectiveness of our collaborative learning framework.
Saliency, Edge and Depth. In our knowledge collector, all knowledge learned from three collaborators are summarized and utilized in a triple-attention manner. As the visualized attention maps in Fig. 2 and Fig. LABEL:fig:tri-att show, the edge knowledge ($Att_{edge}$) can help highlight object boundaries, and the depth and saliency knowledge ($Att_{depth}$ and $Att_{sal}$) can also be used to emphasize salient regions and suppress non-salient regions. We can see from Tab. LABEL:tab:mutual_ablation that both $Att_{edge}$ and $Att_{sal}$ are beneficial for enhancing the feature representation and improving the F-measure and MAE performance. In our framework, we adopt a better strategy that $Att_{edge}$ and $Att_{sal}$ are concatenated together to jointly emphasize salient objects and their boundaries. Finally, by comparing the results in the last two lines of Tab. LABEL:tab:mutual_ablation, we can see that by further utilizing the learned depth knowledge, the detector’s performance can be further improved. We visualize all internal results of the KC in Fig. LABEL:fig:tri-att for better understanding.

We compare results from our method with various state-of-the-art approaches on seven public datasets. For fair comparisons, the results from competing methods are generated by authorized codes or directly provided by authors.