Polyp-SES: Automatic Polyp Segmentation with Self-Enriched Semantic Model

Traditional methods [30, 43, 19, 29] primarily rely on low-level features such as geometric characteristics, which often result in missed or inaccurate detections due to similarities with neighboring tissues. Recent advancements in deep learning have revolutionized the polyp segmentation by autonomously learning complex features. Among these innovations, U-Net [26] obtaines significant improvements across various medical imaging tasks By the simplicity and effectiveness design. ACSNet [47] refines the conventional skip connections within the U-Net [26] and selects adaptive features based on a channel attention mechanism. Pranet [7] utilizes reverse attention mechanisms to refine boundary details in the global feature map through iterative stages enhancing segmentation predictions. MSNet [49] introduces multi-scale subtraction architecture in order to capture intricate details, eliminate redundancy and complementary information between the multi-scale features. SSformer [34] adopts a systematic feature fusion approach, gradually integrating both local and global contextual information, resulting in precise object delineation and boundary detection, while also capturing fine-grained details and comprehensive scene context. Polyp-PVT [5] presents a similarity aggregation module to extract local pixel and global semantic cues from the polyp area.

Transformer [31], initially successful in NLP, has garnered prominence for their potential in computer vision tasks. Leveraging the transformer mechanism, ViTs [6] effectively captures global dependencies and long-range spatial relationships, enabling comprehensive predictions based on the entire image context. By employing shifted windows instead of fixed-size patches, Swin [16] captures spatial relationships between adjacent patches, leading to enhanced feature representation and learning capabilities. Pyramid Vision Transformer (PVT) [35] incorporates a pyramid feature extraction mechanism to capture multi-scale information from input images. Through the combination of features from various scales, PVT [35] demonstrates proficiency in capturing both local details and global context, facilitating accurate dense prediction. UniFormer [15] integrates the strengths of convolutional neural networks (CNNs) and vision transformers (ViTs) into a concise transformer format. This innovative design empowers UniFormer [15] to efficiently capture both local redundancies and complex global dependencies, facilitating effective representation learning. MetaFormer [46] has recently demonstrated the commendable performance in computer vision tasks. This study meticulously examines various token mixers, spanning from basic operators like identity mapping or global random mixing to established techniques such as separable convolution and vanilla self-attention.

In computer vision tasks, the encoder plays a crucial role in capturing spatial information and contextual cues from input images. Transformer-based encoding methods [5, 48, 14] offer the ability to capture long-range dependence information across different areas within the input image. Metaformer [46] has recently introduced new insights into designing transformer architecture and has shown significant performance improvements in various computer vision tasks. Motivated by these findings, our study adopts a vision metaformer encoder known as Caformer [46] as a reliable and competitive backbone for feature extraction. Given an input image $I\in\mathbb{R}^{H\times W\times 3}$, the encoder extracts four levels of features denoted as $\{F_{i}|(i\in(1,2,3,4)\}$. Among these feature maps, $F_{1}$ provides detailed appearance information, while $F_{2}$, $F_{3}$, and $F_{4}$ offer high-level features.

Where $H$, $W$, $C$ represent the height, width spatial, channel dimensions, respectively. In practice, we set $H$ and $W$ to 352, $C_{i}\in(64,128,320,512)$.

The encoder features represent crucial and distinctive information essential for detecting polyp objects. Local features capture intricate details and boundaries, whereas global features provide contextual insights and spatial relationships among various structures. To effectively capture both global and local spatial information, we propose Local-to-Global Spatial Fusion (LGSF), as illustrated in Fig. 2.

The local stage conducts four parallel dilated convolutions [45] with a dilation rate of $\{1,2,4,8\}$ to extract local features at various spatial scales. Each dilated convolution is followed by batch normalization (BN) and a rectified linear unit (ReLU). Resultant features from four dilated convolutions are aggregated to obtain the local feature representation. The resulting feature representation is then processed by a spatial attention mechanism (SA) [38] to suppress the irrelevant regions. The detail of the process is provided below:

The global stage incorporates non-local operation [2] to explore the long-range relationships between each pixel in the spatial space. This stage applies a convolution layer to obtain a feature representation. The resulting feature representation is transposed and followed by a Softmax function, and a Hadamard operation on the input feature $F$ to create a pixel relationship context. This context is then passed through an MLP layer to enhance the relationship representation. In the end, the resulting representation is followed by a sigmoid ($\sigma$) function and another Hadamard operation on the input feature $F$ to attain the global feature representation. The global stage can be delineated as follows:

The final feature representations are obtained by combining local and global information, followed by a convolution layer. Multi-Scale Feature Aggregation (MSFA) module is used to synthesize multi-scale feature representations. This module fuses refined high-level features through a process as depicted in Fig. 2.The first two features, $F_{3}$ and $F_{4}$ undergo bilinear upsampling to match the spatial dimensions of all three features before they are concatenated. To enhance the capture of non-linear features, we further employ a series of convolutional layers, BN and ReLU. Finally, a sigmoid ($\sigma$) function is conducted to produce the output $M_{initial}$, which serves as the initial global feature map.

The shallow layer features are closer to the input more than deep layer features, they preserve more of the original image’s details and structure. Therefore, we leverage the low-level features $F_{1}$ to query implicit semantics from the initial global feature map, thereby providing supplementary semantics to the deep features. The semantic-enriched deep features are then decoded to yield two semantic-enriched segmentation, $M_{1}$ and $M_{2}$. The detailed structure of the Self-Enriched Semantic (SES) module is displayed in Fig. 2. The process can be formulated as following:

Firstly, we consider the distribution of pixel values in patch-level images to represent the initial global feature map $M_{initial}$ to two distinct types of semantic areas including $S_{1}$ and $S_{2}$. Considering patch-level images where exist polyp objects, $S_{1}$ includes patches with pixel values varying from $0$ and $1$, often indicating noise or ambiguous boundaries that have not been sufficiently explored, whereas $S_{2}$ consists of patches with pixel values closer to $1$, representing solid structures of polyp objects. This categorization helps us differentiate between areas of interest and those that may introduce variability or noise. We then employ the $F_{1}$ as query and the $S_{1}$ acts as key-value pairs, applying Cross-layer spatial Attention (CA)[31] to ascertain the relevance of $F_{1}$ and $S_{1}$. The resultant feature is then sent to high-level features through Attention Gate (AG) units progressively. In the end, we fuse semantic-enriched high-level features using Multi-Scale Fusion Aggregation (MSFA) to achieve a semantic-enriched global feature map, $M_{1}$. By Implementing the same operation to $F_{1}$ and $S_{2}$, we also obtain $M_{2}$. Finally, we concatenate $M_{1}$ and $M_{2}$ before passing them through a convolution layer followed by a sigmoid ($\sigma$) function to predict the final global feature map, $M$.

Following recent cutting-edge solutions for the polyp segmentation task, we employ five widely-used benchmark datasets to assess the efficacy of our proposed model. These datasets include Kvasir [13], ClinicDB [1], ColonDB [30], ETIS [28], and EndoScene [32]. Table 1 provides a comprehensive overview of each dataset, including their specific usage details and objectives.

We employ various standard metrics to assess and compare the performance of polyp segmentation algorithms. The Dice score quantifies the spatial agreement between the predicted segmentation mask and the ground truth mask, whereas the IoU score computes the ratio of their overlapping area to the combined area. Both scores range between $0$ and $1$, with higher values indicating better segmentation performance. The MAE computes the average absolute difference between individual pixels in the predicted and ground truth masks. These evaluation metrics offer a comprehensive assessment of the segmentation performance, considering both spatial alignment and pixel-level precision.

We utilize the power of RTX $3090$ GPU to accelerate both the training and inference stages of our model. Throughout the training process, we monitor various metrics including loss function, mDice, mIoU, and MAE scores to assess the performance and guide the training process. The total duration of training amounts to approximately $2$ hours to achieve optimal performance. Detailed training parameters are provided in Table 2.

This section conducts a comprehensive evaluation focusing on two critical aspects: Learning ability, which verifies the segmentation performance on the seen dataset, and generalization ability, which evaluates the capacity of the model to generalize effectively to unseen data. A total of sixteen state-of-the-art models from the domain of the polyp segmentation, including U-Net [26], UNet++ [50], PraNet [7], SFA [8], MSEG [12], ACSNet [47], DCRNet [44], EU-Net [25] and SANet [36], alongside newer models such as Polyp-PVT [5], ADSNet [22], CaraNet [18], TransUnet [3], Transfuse [48], UCTransNet [33], SSFormer [34], are collected for comparative analysis. The performance of these models is meticulously evaluated on five benchmark datasets using mDice, mIoU, and Mean Absolute Error (MAE) scores. In order to ensure fairness and reproducibility in our comparative analysis, we meticulously maintained consistency across training, validation, and testing datasets for all assessed models. Following the methodology outlined in PraNet [7], we adopt an identical dataset configuration as illustrated in Table 1, comprising $900$ and $548$ images sourced from the Kvasir and ClinicDB datasets as the training set, with the remaining $64$ and $100$ images allocated as the respective test set to evaluate the learning ability. Additionally, we utilize the ColonDB, ETIS, and EndoScene datasets, which were not included in the training phase, to assess generalization ability.

Learning ability. In the learning ability experiment, the domain of the test and train set is similar. Table 3 presents the results of different cutting-edge models on the Kvasir and ClinicDB datasets. Our method demonstrates outstanding performance compared to recently published models on both datasets, as evidenced by the mDice, mIoU, and MAE scores. Specifically, our method obtains a mDice score of $0.924$, a mIoU score of $0.875$ on the Kvasir dataset, outperforming the second-best model ADSNet [22]. On the ClinicDB dataset, our model achieves a mDice score and mIoU of $0.945$ and $0.902$, respectively, showcasing an improvement compared to TransFuse [48]. These results underscore the robustness and effectiveness of the proposed method in terms of learning ability.

Generalization ability. We conduct a through evaluation of the polyp segmentation baselines to assess their generalization performance on unseen datasets, as shown in Table 4. It can be observed that our method demonstrates competitive performance across all three datasets compared to other techniques. Specifically, our model is higher than the second-best ADSNet [22] on the ColonDB dataset in term of mDice score and mIoU score. On ETIS dataset, although Transfuse [48] exhibits notable performance with a mIoU score of $0.826$, its corresponding mDice score is lower at $0.737$. In contrast, our results achieve a mDice score of $0.805$, outperforming all other models, alongside a mIoU score of $0.756$. These findings highlight the stable performance of our proposed approach, which excels in both mDice and mIoU scores where other methods may have limitations. Additionally, our model demonstrates remarkable improvement on the EndoScene dataset, with mDice score, mIoU score, and MAE score of $0.911$, $0.847$, and $0.005$, respectively. These results underscore the superior generalization capability of our proposed method.

Qualitative results. We present qualitative results comparing our model with other polyp segmentation baselines across five datasets, depicted in Fig. 3 and Fig. 4. The segmentation results of the compared methods are sourced from the publicly available Polyp-PVT [5]. We can observe that our model produces clear and precise segmentation outcomes across a variety of polyp structures. Furthermore, it effectively identifies and segments polyp objects under different variations in image quality, minimizing artifacts and extraneous regions while maintaining exceptional segmentation accuracy. These findings underscore the efficiency and accuracy of our proposed segmentation algorithm, even in challenging spatial scenarios where previous methods have struggled.

In the ablation study section, we conduct experiments to validate the necessity and effectiveness of each proposed module in the overall architecture individually. Our standard polyp segmentation architecture includes an Encoder, Decoder and Self-Enriched Semantic (SES). The ablation studies are conducted on all five polyp datasets, evaluating based on mDice and mIoU scores.