Suppress and Balance: A Simple Gated Network for Salient Object Detection

Early saliency detection methods are based on low-level features and some heuristics prior knowledge, such as color contrast [1], background prior [62] and center prior [22]. Most of them using hand-crafted features, and more details about the traditional methods are discussed in [54].

With the breakthrough of deep learning in the field of computer vision, a large number of convolutional neural networks-based salient object detection methods have been proposed and their performance had been improved gradually. Especially, fully convolutional networks (FCN), which avoid the problems caused by the fully-connected layer, become the mainstream for dense prediction tasks. Wang et al. [50] use weight sharing methods to iteratively refine features and promote mutual fusion between features. Hou et al. [20] achieve efficient feature expression by continuously blending features from deep layers into shallow layers. However, the single-scale feature cannot roundly characterize various objects as well as image contexts. How to get multiscale features and integrate context information is an important problem in saliency detection.

Recently, the atrous spatial pyramid pooling module (ASPP) [6] is widely applied in many tasks and networks. The atrous convolution can enlarge the receptive field to obtain large-scale features and does not increase the computational cost. Therefore, it is often used in saliency detection networks. Zhang et al. [67] insert several ASPP modules into the encoder blocks of different levels, while Deng et al. [12] install it on the highest-level encoder block. Nevertheless, the repeated stride and pooling operations already make the top-layer features lose much fine information. With the increase of atrous rate, the correlation of sampling points further degrades, which leads to difficulties in capturing the changes of image details (e.g., lathy background regions between adjacent objects or spindly parts of objects). In this work, we propose a folded ASPP to alleviate these issues and achieve a local-in-local effect.

The gated mechanism plays an important role in controlling the flow of information and is widely used in the long short term memory (LSTM). In [2], the gate unit combines two consecutive feature maps of different resolutions from the encoder to generate rich contextual information. Zhang et al. [67] adopt gate function to control the message passing when combining feature maps at all levels of the encoder. Due to the ability to filter information, the gated mechanism can also be seen as a special kind of attention mechanism. Some saliency methods [7, 70, 57] employ attention networks. Zhang et al. [70] apply both spatial and channel attention to each layer of the decoder. Wang et al. [57] exploit the pyramid attention module to enhance saliency representations for each layer in the encoder and enlarge the receptive field. The above methods all unilaterally consider the information interaction between different levels either in the encoder or in the decoder. We integrate the features from the encoder and the decoder to formulate gate function, which plays the role of block-wise attention and model the overall distribution of all blocks in the network from the global perspective. While previous methods actually utilize the block-specific feature to compute dense attention weights for the corresponding block. Moreover, in order to take advantage of rich contextual information in the encoder, these methods directly feed the encoder features into the decoder and do not consider their mutual interference. Our proposed gate unit can naturally balance their contributions, thereby suppressing the response of the encoder to non-salient regions. Experimental results in Fig. 4 and Fig. 9 intuitively demonstrate the effect of multilevel gate units on the above two aspects, respectively.

The gate unit can control the message passing between scale-matching encoder and decoder blocks. By combining the feature maps of the previous decoder block, the gate value also characterizes the contribution that the current block of the encoder can provide. Fig. 3 shows the internal structure of the proposed gate unit. In particular, encoder feature ${E}^{i}$ and decoder feature ${D}^{i+1}$ are integrated to obtain feature ${F}^{i}$, and then it is fed into two branches, which includes a series of convolution, activation and pooling operations, to compute a pair of gate values ${G}^{i}$. The entire gated process can be formulated as,

where $Cat(\cdot)$ is the concatenation operation among channel axis, $Conv(\cdot)$ refers to the convolution layer, $S(\cdot)$ is the element-wise sigmoid function, and $P(\cdot)$ is the global average pooling. The output channel of $Conv(\cdot)$ is 2. The resulted gate vector ${G}^{i}$ has two different elements which correspond to two gate values in Fig. 3.

Given the gate values, they are applied to the FPN branch and the parallel branch for weighting the transition-layer features ${T}^{1}\sim{T}^{5}$, which are generated by exploiting $3\times 3$ convolution to reduce the dimension of ${E}^{1}\sim{E}^{4}$ and the Fold-ASPP to finely process ${E}^{5}$ (Please see Fig. 2 for details). Through multilevel gate units, we can suppress and balance the information flowing from different encoder blocks to the decoder.

In Fig. 4, we statistically demonstrate the curves of gate value with a convolutional level as the horizontal axis. It can be seen that the high-level encoder features contribute more contextual guidance to the decoder than the low-level encoder features in the FPN branch. This trend is just the opposite in the parallel branch. It is because the FPN branch is responsible to predict the main body of the salient object by progressively combining multilevel features, which needs more high-level semantic knowledge. While the parallel branch, as a residual structure, aims to fill in the details, which are mainly contained in the low-level features. In addition, some visual examples are shown in Fig. 9 demonstrate that multilevel gate units can significantly suppress the interference from each encoder block and enhance the contrast between salient and non-salient regions. Since the proposed gate unit is simple yet effective, a raw FPN network with multilevel gate units can be viewed as a new baseline for saliency detection task.

Most existing models either use progressive decoder [67, 53, 70, 57] or parallel decoder [12, 72], as shown in Fig. 5. The progressive structure begins with the top layer and gradually utilizes the output of the higher layer as prior knowledge to fuse the encoder features. This mechanism is not conducive to the recovery of details because the high-level features lack fine information. While the parallel structure easily results in inaccurate localization of objects since the low-level features without semantic information directly interfere with the capture of global structure cues. In this work, we mix the two structures to build a dual branch decoder to overcome the above restrictions. We briefly describe the FPN branch. Taking $D^{i}$ as an example, we firstly apply bilinear interpolation to upsample the higher-level feature $D^{i+1}$ to the same size as ${T}^{i}$. Next, to decrease the number of parameters, ${T}^{i}$ is reduced to $32$ channels and fed into gate unit $G^{i}$. Lastly, the gated feature is fused with the upsampled feature of $D^{i+1}$ by element-wise addition and convolutional layers. This process can be formulated as follows:

where $D^{1}$ is a single-channel feature map with the same size as the input image.

In the parallel branch, we firstly upsample ${T}^{1}\sim{T}^{5}$ to the same size of $D^{1}$. Next, the multilevel gate units are followed to weight the corresponding transition-layer features. Lastly, we combine $D^{1}$ and the gated features by cross-channel concatenation. The whole process is written as follows:

The final saliency map $S^{F}$ is generated by integrating the predictions of the two branches with a residual connection as shown in Fig. 5(c),

where $S(\cdot)$ is the element-wise sigmoid function.

In order to obtain robust segmentation results by integrating multiscale information, atrous spatial pyramid pooling (ASPP) is proposed in Deeplab [6]. And some works [67, 12] also show its effectiveness in saliency detection. The ASPP uses multiple parallel atrous convolutional layers with different dilation rates. The sparsity of atrous convolution kernel, especially when using a large dilation rate, results in that the association relationships among sampling points are too weak to extract stable features. In this paper, we apply a simple “Fold” operation to effectively relieve this issue. We visualize the folded convolution structure in Fig. 6, which not only further enlarges the receptive field but also extends each valid sampling position from an isolate point to a $2\times 2$ connected region.

Let $\mathbf{X}$ represent feature maps with the size of $N\times N\times C$ (C is the channel number). We slide a $2\times 2$ window on $\mathbf{X}$ in stride $2$ and then conduct atrous convolution with kernel size $K\times K$ in different dilation rates. Fig. 6 shows the computational process when $K=3$ and dilation rate is $2$. Firstly, we collect $2\times 2\times C$ feature points in each window from $\mathbf{X}$ and then it is stacked by channel direction, we call this operation ”Fold”, which is shown in Fig. 6\scriptsize1⃝. After the fold operation, we can get new feature maps with the size of $N/2\times N/2\times 4C$. A point on the new feature maps corresponds to a $2\times 2$ area on the original feature maps. Secondly, we adopt an atrous convolution with a kernel size of $3\times 3$ and dilation rate is $2$. Followed by the reverse process of “Fold” which is called “Unfold” operation, the final feature maps are obtained. By using the folded atrous convolution, in the process of information transfer across convolution layers, more contexts are merged and the certain local correlation is also preserved, which provides the fault-tolerance capability for subsequent operations.

As shown in Fig. 2, the Fold-ASPP is only equipped on the top of the encoder network, which consists of three folded convolutional layers with dilation rates $[2,4,6]$ to fit the size of feature maps. Just as group convolution [60] is a trade-off between depthwise convolution [10, 21] and vanilla convolution in the channel dimension, the proposed folded convolution is a trade-off between atrous convolution and vanilla convolution in the spatial dimension.

As shown in Fig. 2, we use the cross-entropy loss for both the intermediate prediction from the FPN branch and the final prediction from the dual branch. In the dual branch decoder, since the FPN branch gradually combines all-level gated encoding and decoding features, it has very powerful prediction ability. We expect that it can predict salient objects as accurately as possible under the supervision of ground truth. While the parallel branch only combines the gated encoding features, which is helpful to remedy the ignored details with the design of residual structure. Moreover, the supervision on $D^{1}$ can drive gate units to learn the weight of the contribution of each encoder block to the final prediction. We use the cross-entropy loss. The total loss L could be written as:

where $l_{s1}$ and $l_{sf}$ are respectively used to regularize the output of the FPN branch and the final prediction. The cross-entropy loss could be computed as:

where $P$ and $Y$ denote the predicted map and ground-truth, respectively.

Dataset. We evaluate the proposed model on five benchmark datasets. ECSSD [61] contains $1,000$ semantically meaningful and complex images with pixel-accurate ground truth annotations. HKU-IS [25] has $4,447$ challenging images with multiple disconnected salient objects, overlapping the image boundary. PASCAL-S [27] contains $850$ images selected from the PASCAL VOC 2009 segmentation dataset. DUT-OMRON [63] includes $5,168$ challenging images, each of which usually has complicated background and one or more foreground objects. DUTS [49] is the largest salient object detection dataset, which contains $10,553$ training and $5,019$ test images. These images contain very complex scenarios with high-diversity contents.
Evaluation Metrics. For quantitative evaluation, we adopt four widely-used metrics: precision-recall (PR) curve, F-measure score, mean absolute error (MAE) and S-measure score. Precision-Recall curve: The pairs of precision and recall are calculated by comparing the binary saliency maps with the ground truth to plot the PR curve, where the threshold for binarizing slides from $0$ to $255$. The closer the PR curve is to the upper right corner, the better the performance is. F-measure: It is an overall performance measurement that synthetically considers both precision and recall:

where $\beta^{2}$ is set to $0.3$ as suggested in [1] to emphasize the precision. In this paper, we report the maximum F-measure score across the binary maps of different thresholds. Mean Absolute Error: As the supplement of the PR curve and F-measure, it computes the average absolute difference between the saliency map and the ground truth pixel by pixel. S-measure: It is more sensitive to foreground structural information than the F-measure. It considers the region-aware structural similarity $S_{r}$ and the object-aware structural similarity $S_{o}$:

where $\alpha$ is set to $0.5$ [14].
Implementation Details. We follow most state-of-the-art saliency detection methods [45, 37, 55, 59, 53, 57, 66, 70, 67] to use the DUTS-TR as the training dataset which contains $10,553$ images. Our model is implemented based on the Pytorch repository and the hyper-parameters are set as follows: We train the GateNet on a PC with GTX 1080 Ti GPU for $40$ epochs with mini-batch size $4$. For the optimizer, we adopt the stochastic gradient descent (SGD). The momentum, weight decay, and learning rate are set as $0.9$, $0.0005$ and $0.001$, respectively. The “poly” policy [30] with the power of $0.9$ is used to adjust the learning rate. We adopt some data augmentation techniques to avoid overfitting and make the learned model more robust, which include random horizontally flipping, random rotation, random brightness, saturation and contrast changing. In order to preserve the integrity of the image semantic information, we only resize the image to $384\times 384$ instead of using a random crop.

We compare the proposed algorithm with seventeen state-of-the-art saliency detection methods, including the DCL [26], DSS [20], Amulet [69], SRM [51], DGRL [53], RAS [7], PAGRN [70], BMPM [67], R3Net [12], HRS [66], MLMS [58], PAGE [57], ICNet [55], CPD [59], BANet [45], BASNet [37] and Capsal [68]. For fair comparisons, all the saliency map of these methods are directly provided by their respective authors or computed by their released codes. To further show the effectiveness of our GateNet, we test its performance in both RGBD SOD and Video Object Segmentation tasks and include the results in appendix.

Quantitative Evaluation. Tab. 1 shows the experimental comparison results in terms of the F-measure, S-measure and MAE scores, from which we can see that the GateNet can consistently outperform other approaches across all five datasets and different metrics. In particular, the GateNet achieves significant performance improvement in terms of the F-measure compared to the second best method BANet [45] on the challenging DUTS-test ($0.870$ vs $0.852$ and $0.888$ vs $0.872$) and PASCAL-S ($0.882$ vs $0.866$ and $0.883$ vs $0.877$) datasets. This clearly demonstrates its superior performance in complex scenes. Moreover, some methods [26, 20, 51, 12] apply the post-processing techniques to refine their saliency maps. Our GateNet still performs better than them without any post-processing. We evaluate different algorithms using the standard PR curves in Fig. 7. It can be seen that our PR curves are significantly higher than those of other methods on five datasets.

Qualitative Evaluation. Fig. 1 and Fig. 8 illustrate some visual comparisons. In Fig. 1, other methods are severely disturbed by branches and weeds while ours can precisely identify the whole objects. And the GateNet can significantly suppress the background with similar shapes to salient objects (see the $1^{st}$ row in Fig. 8). Since the Fold-ASPP can obtain more stable structural features, it can help to accurately locate objects and separate adjacent objects well, but some competitors make adjacent objects stick together (see the $3^{th}$ and $4^{th}$ rows in Fig. 8). Besides, the proposed parallel branch can restore more details, therefore, the boundary information is retained well.

We detail the contribution of each component to the overall network.

Effectiveness of Backbones. Tab. 1 demonstrates that the performance of the gated network can be significantly improved by using better backbones such as ResNet-50, ResNet-101 or ResNeXt-101.

Effectiveness of Components. We quantitatively show the benefit of each component in Tab. 2. We take the results of the VGG-16 backbone with the FPN branch as the baseline. Firstly, the multilevel gate units are added to the baseline network. The performance is significantly improved with the gain of $2.94\%$, $2.17\%$ and $11.67\%$ in terms of the F-measure, S-measure and MAE, respectively. To show the effect of the gate units more intuitively, we visualize the features of different levels in Fig. 9. It can be observed that even if the dog has a very low contrast with the chair or the billboard (see the $1^{st}$ $\sim$ $4^{th}$ rows), through using multilevel gate units, the high contrast between the object region and the background is always maintained at each layer while the detail information is continually regained, thereby making salient objects be effectively distinguished. Besides, the gate units can avoid excessive suppression for the slender parts of objects (see the $5^{th}$ $\sim$ $8^{th}$ rows). The corners of the poster, the limbs and even tentacles of the mantis are retained well. Secondly, based on the gated baseline network, we design a series of experimental options to verify the effectiveness of the folded convolution and Fold-ASPP.

Tab. 3 illustrates the results in detail. We adopt the atrous convolution with dilation rates of $[2,4,6]$ and the same dilation rates are also applied to the folded convolution. It can be observed that the folded convolution consistently yields significant performance improvement at each dilation rate than the corresponding atrous convolution in terms of all three metrics. And the single-layer Fold(6) already performs better than the ASPP of aggregating three atrous convolution layers. The Fold-ASPP also naturally outperforms the ASPP with the gain of $1.17\%$ and $8.0\%$ in terms of the F-measure and MAE, respectively. Finally, we add the parallel branch to further restore the details of objects. In this process, the gate units, Fold-ASPP and parallel branch complement each other without repulsion.