Towards Complex Backgrounds: A Unified Difference-Aware Decoder for Binary Segmentation

As presented in Fig. 6, the inputs of the DGM are the guide map $M\in{R}^{1\times H\times W}$ and the background-aware features $F\in{R}^{C\times\frac{H}{4}\times\frac{W}{4}}$ from the two branches.

Firstly, the background-aware feature $F$ is upsampled to the same sample size as $M$, while the copy operation is performed on $M$ to extend the channel number of the guide map to the same as that of $F$. The formulation is as follows:

Secondly, the cross-attention operation is performed between $F_{0}$ and $M_{0}$. $M_{0}$ is reshaped and transposed to be $G\in{R}^{N\times C}$ and $F_{0}$ is reshaped to obtain two feature maps $Q\in{R}^{C\times N}$ and $K\in{R}^{C\times N}$. After this, the guidance is performed between $Q$ and $G$. In detail, matrix multiplication is performed between $Q$ and $G$ and the result is reshaped to obtain the map $R\in{R}^{C\times C}$. $R$ is then processed by softmax normalization to obtain the probability value, which represents the relationship of the difference between the foreground and the background. The process can be summarized as:

Meanwhile, matrix multiplication is used between $K$ and $R$ and the result is reshaped to the attention features $A\in{R}^{C\times H\times W}$. Finally, a residual structure is introduced to obtain the final enhanced $E\in{R}^{C\times H\times W}$:

where $\beta$ is a learnable parameter.

As shown in Fig. 6, the DEM is used to separably learn the foreground and background features, while adaptively fusing the two to enhance the difference between the foreground and the background. As mentioned before, the guide map can be directly used to generate the coarse map $P\in{R}^{1\times H\times W}$ by using the sigmoid function. In the coarse map $P$, a pixel value closer to one means that the pixel belongs to the foreground, to a large extent, and vice versa. Therefore, $P$ and $1-P$ are used to represent the probability of being foreground and background, respectively. These two probability maps are then used to extract the features from $E$:

where $D_{f}\in{R}^{C\times H\times W}$ represents the foreground features, and $D_{b}\in{R}^{C\times H\times W}$represents the background features. To enlarge the difference between the foreground and the background, subtraction between $D_{f}$ and $D_{b}$ is utilized. Furthermore, two learnable parameters $\theta$ and $\epsilon$ are introduced to adaptively fuse the two types of features, and obtain the difference-aware features $D\in{R}^{C\times H\times W}$:

In fact, the subtraction used in Eq.5 can be replaced with addition, due to the introduction of the two learnable parameters. Finally, the difference-aware features $D$ are processed by two $3\times 3$ convolution blocks and converted to the refined map by the sigmoid function.

We kept the same experimental setup as used for other SOD methods[10]. The training dataset had 10,553 images, and we conducted experiments on six datasets. The DUT-OMRON dataset[68] has 5,168 images with complex objects. The PASCAL-S dataset[69] consists of 850 challenging images. The HKU-IS dataset [43] contains 4,447 images with multiple foreground objects. The Extended Complex Scene Saliency Dataset (ECSSD)[70] and Salient Objects Dataset (SOD)[71] contain 1,000 images and 300 images, respectively. The DUTS dataset[72] is made up of the DUTS-TR (10,553 images for training) and DUTS-TE (5,019 images for testing) sets. In the experiments, the DUTS-TR set was used for the training, and the DUTS-TE set was used for the testing.

For the SOD task, we selected 16 methods to evaluate the proposed difference-aware decoder: methods based on the VGG-16 backbone, $i.e.$, CPD (VGG-16)[21], EGNet (VGG-16)[1], MINet (VGG-16)[48], GateNet (VGG-16)[50], PoolNet + (VGG-16)[73]; methods based on the ResNet-50 backbone, $i.e.$, CPD (ResNet-50)[21], PoolNet (ResNet-50)[74], BANet (ResNet-50)[75], EGNet (ResNet-50)[1], MINet (ResNet-50)[48], GateNet (ResNet-50)[50], PoolNet + (ResNet-50)[73], DFI (ResNet-50) [76]; methods based on the Res2Net-50 backbone, $i.e.$, CSF (Res2Net-50) [77], PoolNet (Res2Net-50) [73]; and a method based on the PVT-v2-b2 backbone, $i.e.$, PVT-SOD (PVT-v2-b2)[78]. We directly used the results or the code provided by the related authors for the comparison. The proposed difference-aware decoder was tested on the VGG-16, ResNet-50, Res2Net-50, and PVT-v2-b2 backbones, for a fair comparison. It is worth noting that PVT-v2-b2 only has a four-layer encoder structure. Therein, for the proposed DAD, we select the first and fourth layer features for Stage A; whereas the first, second and third layers for Stage B.

In order to compare the visual effects of the different SOD methods, several salient images are selected for visualization in Fig. 8. Compared with the other state-of-the-art SOD methods, the difference-aware decoder achieves the best segmentation effects. For the first image, the biggest difficulty is the segmentation of the bus roof from the sky, which are highly mixed. The compared methods of GateNet, MINet, and CSF fail to successfully segment the bus roof from the sky. The PoolNet and PoolNet+ methods can somehow deal with this problem, but there are still some flaws. The difference-aware decoder, with ResNet-50, Res2Net-50, and PVT-v2-b2 as the backbone, can perfectly distinguish the clear outline of the bus from the background. For the second and the third images, it is apparent that there are ambiguous regions in the segmentation results of the other methods, but the proposed method with different backbones can achieve maps that are much more similar to the ground truth.

Table I lists the overall quantitative evaluation results of the different methods obtained on the six datasets. The best score for each backbone is highlighted in bold, and the best score for each metric is marked in red. It is clear that the proposed difference-aware decoder with PVT-v2-b2 backbone achieves the best results, for all the datasets and evaluation metrics. What is more, with the different backbones of VGG-16, ResNet-50, and Res2Net-50, the difference-aware decoder always achieves the best performance, indicating the transferability of the proposed decoder architecture for different backbones.

Compared to the other methods, the difference-aware decoder improves the evaluation metrics by a large margin. The main reason for this advantage is that the proposed difference-aware decoder can more effectively enlarge the difference between the foreground and background, by fusing the guide map and the background-aware features.

Four datasets were used to evaluate the proposed model: CHAMELEMON [79], CAMO [80], COD10K [2], NC4K[36]. These four datasets differ significantly. NC4K is the largest dataset, which contains 4,121 images downloaded from the Internet. COD10K includes 78 camouflaged categories, with 3,040 training images and 2,026 test images in total. CAMO provides 1,250 images in total, including 1,000 training images and 250 test images of eight categories. The smallest dataset is the CHAMELEON dataset, which does not provide a training set, and contains only 76 images for testing. According to [11, 33, 2], we used the training images from the CAMO and COD10K datasets as the training set (4,040 images) and the test images from the CHAMELEMON [79], CAMO [80], COD10K [2], and NC4K[36] as the test set.

In order to evaluate the proposed difference-aware decoder, we selected 12 state-of-the-art methods for comparison: methods based on the ResNet-50 backbone, $i.e.$, SINet (ResNet-50) [2], LSR (ResNet-50) [36], MGL-R (ResNet-50)[53], MGL-S (ResNet-50)[53], UGTR (ResNet-50)[81], PFNet (ResNet-50) [52], EINet (ResNet-50)[82]; methods based on the Res2Net-50 backbone, $i.e.$, SINet V2 (Res2Net-50)[11] and EINet (Res2Net-50)[82]; and methods based on the PVT-v2-b2 backbone, $i.e.$, SINet V2 (PVT-v2-b2)[11], EINet (PVT-v2-b2)[82], and DTIT (PVT-v2-b2) [51]. The proposed difference-aware decoder was tested on the ResNet-50, Res2Net-50, and PVT-v2-b2 backbones, for a fair comparison.

In order to compare the visual effects of the different methods, we selected several challenging camouflaged images for the visualization, which are shown in Fig. 9. In the first image, the left person is perfectly hidden in the trees. The LSR, PFNet, and SINet V2 methods and the proposed difference-aware decoder with ResNet-50 backbone fail to segment the body of the left person. However, the difference-aware decoder with Res2Net-50 and PVT-b2-v2 backbones can clearly detect both persons. For the second image, the two soldiers are hidden in the rocks, and the other compared methods cannot segment them accurately, but the difference-aware decoder with different backbones can find them successfully. For the third image, the creatures are mixed with their surroundings and are difficult to distinguish. The segmentation results achieved by the compared methods are incomplete, referring to the ground truth, whereas the difference-aware decoder with Res2Net-50 and PVT-b2-v2 backbones can achieve much better segmentation maps. From our understanding, the backbone is important for the COD task. However, the proposed decoder paradigm can make the best use of different backbones to achieve a superior accuracy.

Table II lists the overall quantitative evaluation results of the different methods obtained on the four datasets. On the ResNet-50 backbone, the proposed difference-aware decoder achieves the best score in the $E_{\phi}$ metric for all four datasets. With regard to the $S_{\alpha}$ and $F^{w}_{\beta}$ metrics, the difference-aware decoder achieves the best scores on the CAMO dataset and the second-best scores on the NC4K dataset. With regard to the $M$ metric, the difference-aware decoder achieves the best scores on the CAMO, CHAMELEON, and NC4K datasets. For the COD task, Res2Net-50 and PVT-v2-b2 are the better backbones[2, 51]. On these two backbones, the proposed difference-aware decoder outperforms SINet V2 and DTIT by a large margin, for all the metrics and datasets. The main reason for this advantage is that the proposed difference-aware decoder can more effectively enlarge the difference between the camouflaged objects and background, by fusing the guide map and the background-aware features.

According to the experimental settings used for PraNet [3], we tested the trained model on three public datasets: ClinicDB[83], ColonDB[84], and ETIS[85]. The recently proposed Polyp-PVT[12] utilizes PVT-v2-b2[20] as the backbone, and shows a great performance in polyp segmentation.

In order to evaluate the proposed difference-aware decoder, we selected eight state-of-the-art methods for comparison: UNet[86], UNet++[87], MSEG[38], ACSNet[39], PraNet[3], EU-Net[40], SANet[41], and Polyp-PVT[12]. Following Polyp-PVT, the difference-aware decoder also used PVT-v2-b2 as the backbone for the experiments.

In Fig. 10, two examples are selected for visualization. From these two examples, it can be seen that the polyps are highly mixed with the background. The compared methods of PraNet, EUNet, and SANet fail to achieve efficient segmentation, and Polyp-PVT can achieve only a rough map. However, the difference-aware decoder can segment the complete outline of the polyps successfully, compared to the ground truth.

Table III lists the overall quantitative evaluation results of the different methods obtained on the three datasets. For the $Dice$, $IoU$, $F^{w}_{\beta}$, and $S_{\alpha}$ metrics, the difference-aware decoder achieves the best performance on the ClinicDB, ColonDB, and ETIS datasets. For the $M$ metric, the difference-aware decoder obtains the best accuracy on the ClinicDB and ColonDB datasets. These quantitative evaluation results reflect the effectiveness of the proposed decoder architecture in the polyp segmentation task.

The MSD[4] dataset was chosen for the mirror detection experiments. The MSD dataset consists of 4,018 pairs of images, with 3,063 pairs of images for training and 955 pairs of images for testing.

In order to evaluate the proposed difference-aware decoder, we selected two general segmentation methods: PSPNet[26], and DeepLab v3+[24]; and three state-of-the-art methods designed for mirror detection: MirrorNet[4], PMD-Net[88], and LSA[89]. For a fair comparison, we used ResNeXt-101 [90] pre-trained on ImageNet-1K [91] as the backbone for all the methods. Conditional random fields (CRF)[92] was also utilized as a common post-processing method for all the methods during the inference.

Fig. 11 illustrates several examples for the visual comparison between the proposed method and the other state-of-the-art methods. In these examples, it can be found that the backgrounds of the mirrors are complex and it is difficult for the other methods to segment relatively clear bodies for the mirrors. In contrast, the difference-aware decoder can achieve a segmentation map that is closer to the ground truth, showing the superiority of the proposed method.

Following the evaluation strategy used for other methods[89], we used the F1-score ($F1$), Intersection over Union ($IoU$), accuracy ($Acc$), and $M$ metric to measure the performance of the proposed difference-aware decoder and the other compared methods. From Table IV, it can be found that the proposed method achieves the best score in all the metrics.

In Stage A, we chose layers 1 and 5 from the backbone for the guide map generation; and in Stage B, we fused layers 2 4 for the background-aware features extraction. In the following, we attempt to explain why we divided the multi-layer features in this way. From experience [22, 2], the highest-level features are always used for the coarse map generation. For a complete analysis, we combined layer 5 with different layer(s) to finish Stage A, and the other layer(s) for Stage B. Table V presents the results of the different combinations of multi-layer features. It can be clearly observed that the strategy of combining layers 1 and 5 for Stage A can achieve the best accuracy on the CAMO, ECSSD, and PASCAL-S datasets, while almost achieving the best results on the COD10K dataset. Therefore, in the experiments, we fixed layers 1 and 5 for Stage A. The explanation for this is that the high resolution of layer 1 can make up more spatial details for layer 5, which is an extremely effective way to obtain a fine guide map. It needs to be clarified that the multi-layer features were divided without overlapping. This is mainly done to enhance the differences between the guide map from Stage A and the background-aware features from Stage B.

In Stage B, the proposed MFF module was used to integrate the features at different levels. In fact, the upsample/downsample function is utilized to resize layers 2 and 4 to the size of layer 3. In Table 5, Top-Down means that layer 2 and layer 3 were downsampled to the same size as layer 4, while Bottom-Up means that layer 3 and layer 4 are upsampled to the size of layer 4. Table VI indicates that the proposed strategy can achieve the best detection accuracy. This is mainly because, if the resolution is low, the proportion of the features that relates to the difference is also reduced. In addition, if resolution is blindly pursued, it would result in the loss of semantic information and an increase in the computational complexity. The joint pyramid upsampling (JPU[93]) method also takes the three feature maps as the inputs and utilizes a context module to generate a high-resolution feature map, which is similar to the MFF module. The MFF module was therefore compared with the JPU method [93]. From Table VI, it can be seen that the proposed MFF module obtains better results than the JPU method[93].

The FEM was used in Stages A and B to enhance the receptive field of the feature map, so as to obtain sufficient contextual information to establish the dependency between the foreground and the background. As is well known, modules such as ASPP [22], Dense ASPP[60], and RFB [59] can also have the same effect. Therein, we replaced the proposed FEM with ASPP/FRB for the ablation study. From Table VII, it can be seen that the proposed FEM can achieve the best accuracy in the COD/SOD tasks. Furthermore, we removed the dilation rate in the FEM (w/o DR) and also found that the performance dropped. It can be concluded that the consecutive atrous convolutions with different dilation rates are helpful, while the proposed FEM can outperform the other well-known modules designed for the enhancement of the receptive field.

In Stage C, the DGM is utilized to enhance the foreground features from the two branches. The DGM can be regarded as a plug-and-play module. Table VIII lists the results of the difference-aware decoder with/without the DGM. It can be seen that the features are enhanced by the guide map in a way similar to cross-attention, and the prior knowledge of the foreground and the background on the guide map can be used to complete the screening of the features, which is the key to improving the model’s performance.

Also in Stage C, the DEM is proposed to further enhance the difference between the foreground and the background. We chose the simple foreground ($F$), the background ($B$), and the subtraction of $F-B$ as the output features. From Table IX, it is clear that $F-B$ can achieve the best results, which validates the original intent of the proposed fusion design. It can be concluded that the difference-aware fusion of the two can more efficiently distinguish the background and precisely extract the foreground objects.

The DAE used in Stage C can be regarded as a plug-and-play module, and can be utilized multiple times. As can be observed in Table X, repeating the DAE three times can lead to a significant performance degradation. From our understanding, the map generated from the output of the DAE is regarded as the guide map for the next extractor. However, the map from the output of the DAE is very similar to that of the feature map from the MFF module. Therein, the DAE fails to improve the difference features. From Table X, it is apparent that repeating the DAE twice obtains the best results.