Receptive Field Broadening and Boosting for Salient Object Detection

Computer vision tasks often need to extract detailed information and semantic information of an image. Detailed information does not depend on the depth of the network but requires larger input resolution, while semantic information is just the opposite. In order to balance these two processes, the dual-branch network is proposed to obtain these two features separately, such as BiSeNet (Yu et al. 2018). This method can better balance efficiency and performance. However, the backbones used by the two branches are often similar convolutional neural networks, which often brings a lot of redundancy. For example, StdcNet (Fan et al. 2021) propose that better results can be obtained without a separate detail branch. In fact, this two-stream fusion idea is not only suitable for the fusion between convolutional neural networks. Recently, more and more visual backbones have been proposed, such as ResNet (He et al. 2015), MobileNet (Howard et al. 2017), StdcNet (Fan et al. 2021), ShuffleNet (Zhang et al. 2018), Swin Transformer (Liu et al. 2021b), T2T-VIT (Yuan et al. 2021), etc. These models contain efficient lightweight CNN and high computational cost networks based on visual self-attention. Compared with fusing different convolutional neural networks, we propose to fuse CNN and transformer frameworks with different but complementary mechanisms.

The multi-scale perception ability of neural networks often depends on the range of the receptive field. The multi-branch module is an effective method to enhance the receptive field of convolutional neural networks. For example, Pyramid Pooling Moudle (Zhao et al. 2017) and Atrous Spatial Pyramid Pooling (Chen et al. 2017) enhance the local scale perception ability of the network by designing multi-branch modules with different receptive fields. PoolNet (Liu et al. 2019) has used PPM for SOD for the first time and achieved good results. BaNet (Su et al. 2019) proposes to improve ASPP and achieve the latest new capabilities at the time. The above methods can effectively improve the expressive ability of the network, but there are still shortcomings. The original intention of the multi-branch model is to allow different branches to capture specific features. However, most of the previous methods use a synchronous error back propagation strategy and seldom consider the specificity of the branch separately. In contrast, We use a similar multi-model boosting idea to achieve mutual enhancement of multiple branches. Each branch calculates the error separately and weights the error areas of the other branches.

In the past three years, the upper limit of SOD performance has been continuously broken by the latest methods. For example, F3Net (Wei, Wang, and Huang 2019) proposes Fusion, Feedback and Focus to detect salient objects in 2019 and has achieved the best performance at the time. The MAE reach 0.053 on the DUT-OMRON dataset. In 2020, LDF (Wei et al. 2020) decouples pixels based on the distance from the edge and iteratively optimizes the predicted maps, and its MAE on the same dataset reaches 0.051. In 2021, PA-KRN (Xu et al. 2021) proposes a strategy of positioning first, then segmentation, and its MAE reaches 0.050. VST (Liu et al. 2021a) uses the transformer framework for SOD for the first time, and its MAE reach 0.058. From the above results, it can be seen that the performance of SOD is getting better and better, but the improvement is very slow. We explore the differences between CNN and transformers in terms of features, computational complexity, and attention mechanisms. Based on this, we design a bilateral network and an Attention Feature Fusion Module to balance the two models and propose a Multi-Head Boosting method to compensate for the lack of association between regions. Due to the complementary combination, our method has achieved good results in both performance and efficiency. In particular, the MAE metric of our method on ECSSD has reached 0.040.

As can be seen from Fig. 1, CNN and transformers have both distinctiveness and complementarity. Therefore, we propose a bilateral network as shown in Fig. 3 to fully utilize the complementary advantages. The bilateral network contains a semantic branch and a detail branch. The semantic branch makes full use of the global view of the transformer model to extract global semantic features and enhance the receptive field of the overall network. The detail branch is responsible for extracting high frequency details. The ingenuity of the bilateral network is that it can take advantage of the two types of models at the lowest computational cost. The complex semantic branch is mainly responsible for extracting global attention information so it can be calculated under low-resolution input. The detail branch only requires a lightweight CNN as basic network. Therefore, the combination of transformer and CNN can efficiently balance semantics, details, and their calculation cost.

Specifically, we choose ResNet-18 (He et al. 2015) to build the detail branch. In order to enrich the detailed information extracted by this branch, we remove the first down-sampling operation to obtain more detailed features at high resolution. In this way, the resolution of the features output by each layer is $1/2$, $1/4$, $1/8$, and $1/16$ of the initial resolution. As for the semantic branch, we utilize Swin Transformer (Liu et al. 2021b) to extract global features. This branch first samples the image to a resolution of $56\times 56$, and then passes through the self-attention modules of the transformer in turn.

The working process of proposed bilateral network can be summarized as follows: Input the $352\times 352$ resolution image into the lightweight CNN to obtain multi-level detailed features $R_{1}$,$R_{2}$,$R_{3}$ and $R_{4}$. Besides, the input image is down-sampled to a resolution of $56\times 56$. Then we input it to the semantic branch to obtain features $T_{1}$, $T_{2}$, $T_{3}$, and $T_{4}$ and reshape the features. Finally, Use the attention feature fusion module $AF_{i}$ to fuse $T_{i}$ and $R_{i}$, where $i\in\{1,2,3,4\}$.

Unlike the features of CNN, the features of transformer model are composed of vectors stretched by pixel blocks. Given the transformer features, we first restore the previous positional relationship of the vector to get feature $X$ of Fig. 4. The transformer model calculates the correlation between all patch blocks in the space through the vector inner product, while CNN establishes the connection between all channels in the local space. The former constructs better spatial correlation, while the latter has stronger channel correlation. In order to better integrate the two types of features, we design an Attention Feature Fusion Module (AF). AF uses the respective characteristics of the two types of features to enhance the expression of each other. The specific process can be expressed as:

	$\displaystyle F_{mid}$	$\displaystyle=\mathcal{C}_{br}(\mathcal{U}(\mathcal{R}_{ca}(Y)\otimes X)ⓒ(\mathcal{R}_{sa}(X)\otimes Y)),$		(1)
	$\displaystyle F_{out}$	$\displaystyle=\mathcal{C}_{br}(\mathcal{U}(\mathcal{C}(X))\oplus F_{mid}\oplus\mathcal{C}(Y)),$		(2)

where $X$ and $Y$ correspond to the features in Fig. 4. $\mathcal{R}_{sa}$ and $\mathcal{R}_{ca}$ represent the operations corresponding to the crossed arrows in the figure. $\mathcal{U}$ represents the upsampling operation. $\mathcal{C}$ represents convolution, and the subscript $br$ represents a Batch Normalization and a ReLU activation. $\oplus$, $\otimes$ represent pixel-wise addition and multiplication operations, respectively. $ⓒ$ denotes feature concatenation operation. $F_{mid}$ is the intermediate result, $F_{out}$ is the final output.

Specifically, the feature $X$ obtained by transformer contains rich semantic information, while the low level details are lost. CNN feature $Y$ is obtained by convolution, the spatial receptive field is smaller, but the correlation between channels is stronger. Therefore, we use the features of $X$ to select the spatial Features of $Y$, and use the channel information of $Y$ to enhance $X$. Finally, the final result is obtained by fusing all the information through the residual structure.

The design idea of Multi-Head Boosting (MHB) is to improve the overall effect of the multi-branch module by decoupling the training process and adding branch complementary losses. It can be roughly divided into two parts: multi-path decoupling and boosting loss. Next, we will introduce these two processes in detail.

Multi-path decoupling aims to decouple multi-branch modules trained in parallel into multiple models. The significance is that the training process of different branches can be isolated, thereby reducing the gradient correlation between each branch and enhancing the scale specificity of different branches. ASPP integrates multi-channel features in the module, single-channel training cannot be performed directly, therefore we cannot use multi-channel features in the testing process. Our solution is to delay feature merging to the final result layer. In this way, the purpose of single-channel training and multi-channel testing can be realized.

Boosting Loss aims to enhance the complementarity between branches. As shown in Fig. 3, the forward propagation process will activate all branches and obtain $N$ predicted maps $\{P_{i}|i=1,2,3,...,N\}$. The random number $X$ is used to determine which branch performs error back propagation, and the prediction errors of other branches are regarded as weight. In this way, each branch will pay attention to the mispredicted parts of other branches. The calculation process of weight can be expressed as:

$\displaystyle W_{x}=\sum_{i=1}^{N}\mathcal{L}_{bce}(P_{i},g)(i\neq X)+1,$

(3)

where $g$ represents the ground-truth map. $\mathcal{L}_{bce}$ denotes the pixel-level binary cross-entropy error, which can be expressed as:

$\displaystyle\mathcal{L}_{bce}(p,g)=-((g\otimes log(p))\oplus(\overline{g}\otimes log(\overline{p}))),$

(4)

where $p$ denotes predicted map. $log$ represents a pixel-by-pixel logarithmic operation. $\overline{g}$ and $\overline{p}$ denote the inversion operation pixel by pixel. The boosting loss can be calculated according to the calculated weight. It can be expressed as:

$\displaystyle\mathcal{L}_{b}(p,g,w)=\mathcal{L}_{wbce}(p,g,w)+\mathcal{L}_{wiou}(p,g,w),$

(5)

where $\mathcal{L}_{wbce}$ and $\mathcal{L}_{wiou}$ represent weighted binary cross entropy loss and weighted intersection of union loss. They can be expressed as:

$\displaystyle\mathcal{L}_{wbce}(p,g,w)=\frac{\mathcal{S}(\mathcal{L}_{bce}(p,g)\otimes w)}{\mathcal{S}(w)},$

(6)

$\displaystyle\mathcal{L}_{wiou}(p,g,w)=1-\frac{\mathcal{S}(p\otimes g\otimes w)}{\mathcal{S}((p\oplus g)\otimes w)-\mathcal{S}(p\otimes g\otimes w)},$

(7)

where $\mathcal{S}$ represents the operation of summing all pixels. Other symbols are consistent with the previous description.

In general, MHB decomposes the training process of multi-scale modules so that multiple branches receive training with random samples and relieves the mutual influence of the error back propagation process. BL further enhances the complementarity between each branch. The experiments prove that MHB and BL can effectively improve SOD performance.

We utilize the sum of Binary Cross Entropy (BCE) and Intersection of Union (IoU) as the loss function, which is widely used in LDF (Wei et al. 2020), etc. In addition to the final prediction map, we will also supervise the output features of each Convolution Block (CB) in Fig. 4. The total loss function can be expressed as:

$\displaystyle\mathcal{L}_{tot}=\sum_{i=1}^{M}\mathcal{L}_{bi}(F_{i},g)+\mathcal{L}_{b}(P_{4},g,W_{x}),$

(8)

where $F_{i}$ represents the prediction map of each CB module, and $P_{4}$ represents the final result. $M$ denotes the number of CB modules. $\mathcal{L}_{bi}$ can be expressed as:

$\displaystyle\mathcal{L}_{bi}(p,g)=\mathcal{L}_{wbce}(p,g,W_{1})+\mathcal{L}_{wiou}(p,g,W_{1}),$

(9)

where $W_{1}$ represents a matrix with all $1$ values, which means that the auxiliary loss is not weighted.

The evaluation indicators are mainly as follows: Mean Absolute Error (MAE), max and mean F-measure ($F_{\beta}^{*},F_{\beta}^{m}$) (Yang et al. 2013), Maximum enhanced-alignment measure ($E_{\xi}$) (Fan et al. 2018), structure measure ($S_{m}$) (Fan et al. 2017) and precision-recall (PR).

The experiment involves the following datasets: DUT-OMRON (Yang et al. 2013) (5168), ECSSD (Yan et al. 2013) (1000), HKU-IS (Li and Yu 2015) (4447), PASCAL-S (Li et al. 2014) (850), DUTS-TE (Wang et al. 2017) (5019), DUTS-TR (Wang et al. 2017) (10553). We choose DUTS-TR as the training set and other datasets as the test set.

We use NVDIA GTX 2080Ti to train our network. The input resolution of the detail branch is $352\times 352$, and the input resolution of the first transformer block is $56\times 56$. The maximum learning rate of the backbone is 0.004, and the other parts are expanded by ten times. And the learning rate will first increase and then decrease when training our model. The optimization method uses Stochastic Gradient Descent. Batch size is set to 26, and the epoch is set to 32. The data augmentation methods involves multi-scale training, random flipping and cropping. The prediction results do not need any post-processing.

The experimental process involves the state-of-the-art methods of the last three years. Four of them in 2019 include BANet (Su et al. 2019), BASNet (Qin et al. 2019), PoolNet (Liu et al. 2019) and CPD (Wu, Su, and Huang 2019). Seven methods in 2020, including LDF (Wei et al. 2020), MINet (Pang et al. 2020), GCPANet (Chen et al. 2020), GateNet (Zhao et al. 2020), DFI (Liu, Hou, and Cheng 2020), ITSDNet (Zhou et al. 2020) and U2Net (Qin et al. 2020). And there methods published in 2021: PAKRN (Xu et al. 2021), VST (Liu et al. 2021a) and PFSNet (Ma, Xia, and Li 2021). The evaluation process uses a unified evaluation code to evaluate the published saliency map.

The verification results prove that the proposed method achieves a breakthrough in performance compared with the latest method. First of all, it can be seen from the PR curve in Fig. 5 that the PR curve of the previous methods has not improved significantly, but our method is significantly better than other methods. In addition, the performance comparison results in Tab. 1 can also verify the effectiveness of the method in this paper. Our method has mostly surpassed the previous method in five commonly used indicators. And the performance improvement is very significant. Taking the ECSSD dataset as an example, the MAE indicator reached $0.033$ in 2020, and it has been reduced to $0.031$ in 2021, a reduction of only $6\%$. But our method reached $0.022$, an decrease of $29\%$ compared to last year. The experimental data on DUTS-TE shows that the MAE metric of LDF (Wei et al. 2020) in 2020 has reached $0.034$, while the result in 2021 is basically unchanged. But our method reduces the MAE indicator to $0.025$, which is a $24\%$ reduction. From the results of F-measure, the proposed method greatly improves the F-measure value calculated at different thresholds.

In order to verify the innovations of this paper one by one, we conduct ablation experiments on the proposed Bilateral Network, Attention Fusion Module AF, Multi-Branch Decoupling, and Boosting Loss. The experimental details are shown in Tab. 2.