An Empirical Study of Attention Networks for Semantic Segmentation

Semantic segmentation is a task which assigns a category to all pixels in a picture [24]. Many methods have been developed to tackle problems ranging from automatic driving [20], virtual reality [18], human-machine interaction [17], ,scene understanding [26], medical image analysis [14],etc. Recently, the common strategy to solve semantic segmentation problems is the deep structure network, which is much more accurate than traditional methods.

A common architecture of a semantic segmentation network is widely regarded as an encoder network, followed by an end-to-end structure decoder network. The encoder is a pretrained backbone, such as Resnet [6]. The function of the decoder is to map the feature semantic calculated by the encoder to the pixel space to make pixel-level classification.

Non-Local Net bridges self-attention in machine translation and the Non-Local operations in computer vision. The self-attention networks introduced in the following part can be regarded as typical examples of this network. The proposed Non-Local method calculates the interaction between every two points to obtain global dependencies without the limitation of convolution kernel size. Non-Local method is similar to utilize a convolution kernel with size of the feature map, which can maintain more contextual information. The structure of Non-Local block is shown in Fig. 1.

In the figure, $"\bigoplus"$ means element-wise sum, and $"\bigotimes"$ means matrix multiplication. Each operation includes softmax function. The blue boxes mean $1\times 1$ convolutions.

Based on Non-Local Net, the following works can be divided into two types, as shown in Fig. 2. One is enriching the obtained contextual information, such as DNLNet, DANet, RecoNet, and FLANet, and the other is reducing the computation cost, such as EMANet, ISSANet, CCNet, and AttaNet.

Non-Local Net achieves scores of 78.5$\%$ mIoU on Cityscapes and 43.2$\%$ mIoU on ADE20K. Compared with the baseline, which predicts results by the backbone, Non-Local Net gets 4$\%$ mIoU increment on Cityscapes validation set [8]. It demonstrates the proposed Non-Local operations directly capture global dependencies and significantly improve the segmentation performance. However, it also leads to high computation complexity. The FLOPs of Non-Local Net is much higher than its variations that are designed to reduce the computation cost.

DANet achieves scores of 80.6$\%$ mIoU on Cityscapes and 43.8$\%$ mIoU on ADE20K. Compared with Non-Local Net, the increment of computation complexity comes from the channel attention module. Compared with the attention map obtained by Non-Local Net, which size is $HW\times HW$, the attention map generated by the channel attention module is $C\times C$, which is not a big number. Therefore, DANet does not increase high computation on Non-Local Net. Tab.1 shows the FLOPs is slightly over Non-Local Net, but the accuracy is much higher. It demonstrates the proposed Dual Attention block successfully enriches the contextual information without cost too much computation.

DNLNet achieves scores of 79.4$\%$ mIoU on Cityscapes and 43.7$\%$ mIoU on ADE20K. Compared with Non-Local Net, DNLNet splits the computation of Non-Local block into two terms, a whitened pairwise term and a unary term. The unary term tends to model salient boundaries and the whitened pairwise tends to obtains within-region relationships. Decoupling these two terms can enhance their learning. The the unary term is achieved by $1\times 1$ convolution, and whitened pairwise term is consist of the spatial attention module with the whitened operation. Therefore, the increment of computation complexity only comes from the unary term, which can be overlooked, and the computation complexity is similar to Non-Local Net.

FLANet achieves scores of 82.1$\%$ mIoU on Cityscapes and 46.68$\%$ mIoU on ADE20K, which are the best results among these attention networks. Unlike DANet, which considers channel and spatial dimensions separately, FLANet proposes Fully Attentional Block to encode channel and spatial attention in a single similarity map. And the improved accuracy shows it is important to integrate both dimensions. Besides, compared with DANet, FLANet utilizes global average pooling to obtain a global view in channel Non-Local mechanism. Therefore it takes less computation.

Denoised NL achieves scores of 81.1$\%$ mIoU on Cityscapes and 44.3$\%$ mIoU on ADE20K. Besides, it has the least computation cost among these attention networks. Denoted NL utilizes convolution layers to obtain a smaller input feature map, which dramatically reduces the computation cost. What’s more, different from DANet and FLANet, which enrich the contextual information by leading in the channel dimension, Denoised NL enriches the neighbor information to reduce noises in the attention map. The experiment results demonstrate that the contextual information can be more precise without inter-class and intra-class noises.

EMANet achieves scores of 78.6$\%$ mIoU on Cityscapes and 43.3$\%$ mIoU on ADE20K. Compared with Non-Local Net, EMANet simplifies the input to a set of compact bases by EM algorithm and can converge in only three times iterations. Therefore, EMANet greatly reduces the complexity, the computation is much more efficient than Non-Local Net. What’s more, the performance is slightly over Non-Local Net on Cityscapes dataset.

ISANet achieves scores of 81.9$\%$ mIoU on Cityscapes and 43.5$\%$ mIoU on ADE20K. Since ISANet factorizes the computation of the dense affinity matrix to the product of two sparse affinity matrices, the FLOPs of ISANet is much less than Non-Local Net. Besides, the ISANet can adjust the group number of the sparse affinity matrices to achieve the best computation efficiency. In the experiment, we set the group number to 8 to minimize the computation cost.

CCNet achieves scores of 78.8$\%$ mIoU on Cityscapes and 44.0$\%$ mIoU on ADE20K. Compared to Non-Local Net, which links each pixel with all pixels, CCNet links each element to the row and column where the element is located, which is called criss-cross attention. It can obtain the contextual information of all the pixels on its criss-cross path. CCNet can adjust the recurrent times of RCCA module. In the experiment, we set the recurrent number to two, which achieves a good trade-off between speed and accuracy. Although indirectly obtained global dependencies cannot generate better results than direct calculation, CCNet utilizes the prior knowledge that the similarity in criss-cross path is more important. The RCCA module utilizes the similarity in criss-cross path twice to enhance the attention map. Therefore, the accuracy is better than Non-Local Net.

Tab.1 shows the comparison of FLOPs and memory among different attention modules. The table shows that Denoised NL, FLANet, EMANet, and CCNet achieve the highest computation efficiency. They take much fewer GFLOPS than other networks. Besides, Fig.3 shows that FLANet achieves a good trade-off between accuracy and speed.

This section also analyses the per-class mIoU of attention networks for the Cityscape dataset to find their characteristics. The quantitive results of experiments can be found in Tab.2. From the table, all networks can get high scores for classes that are big objects and account for a large percentage of the dataset, such as roads and buildings. However, the results are not well for other classes that are hard to distinguish and recognize, such as walls, fences, and poles. Most classes such as Non-Local Net, EMANet, CCNet, and DNLNet cannot predict trains correctly. However, attention networks that enrich the contextual information, such as DANet and FLANet, can achieve high accuracy on trains with non-trivial increments. Besides, Denoised NL and FLANet achieve the best accuracy on per-class results, and the overall accuracy is much higher than other networks. For road,wall, build,wall,sidewalk,pole,sign,traffic light, terrain,person,sky,train,car,motorbike, and bike, FLANet achieves the highest accuracy. Besides, for vegetation, rider, truck, and bus, Denoised NL achieves the highest accuracy. This section conducts experiments and visualizes the prediction on the class bus and train, and the qualitative results can check Fig.4 and Fig.5.

In Fig.4, the yellow box indicates Denoised NL achieves the highest performance on the class bus. Denoised NL captures clear edge information, and the prediction inside the bus is consistent. Compared with Denoised NL, EMANet and CCNet can capture clear edge information, but the prediction inside the bus shows an inconsistency. This observation shows why the mIoU is much lower than other networks.

In Fig.5, the results show FLANet can make a much better prediction on the class train. Other networks cannot capture correct details in the train. For example, the inconsistency in the train shows the train was predicted as a wall or bus. And the tree on the left of the train is misclassified. Besides, the wire on the top cannot be predicted. These observations confirm why FLANet can achieve non-trivial increments on the class train.

This section also conducts experiments on ADE20K datasets. Following previous works, it does not analyse the per-class mIoU. Since the ADE20K dataset contains 150 classes, it is pretty hard to analyse the characteristic of each class for all networks. Tab.3 shows the overall performance by mIoU. The results show that FLANet and Denoised NL can also achieve the highest accuracy.

Besides, Tab.4 shows the comparison of these attention networks with SOTA methods on Cityscapes test set. Denoised NL and FLANet also outperform most SOTA methods, except HSSA. HSSA is the latest work which get the best prediction by MaskFormer [1] which is transformer structure. Therefore, the computation cost of HSSA is much higher than FLANet. When HSSA utilizes ResNet-101 as backbone, the mIoU is 83.02%, which is outperformed by FLANet. In conclusion, for most segmentation tasks, FLANet is a good choice. It achieves the highest accuracy with high computation efficiency.