Polyp-PVT: Polyp Segmentation with Pyramid Vision Transformers

Traditional Methods. Computer-aided detection is an effective alternative to manual detection, and a detailed survey has been conducted on detecting ulcers, polyps, and tumors in wireless capsule endoscopy imaging [15]. Early solutions for polyp segmentation were mainly based on low-level features, such as texture [2], geometric features [2], or simple linear iterative clustering superpixels [3]. However, these methods have a high risk of missed or false detection due to the high similarity between polyps and surrounding tissues.

Deep Learning-Based Methods. Deep learning techniques [16, 17, 18, 19, 20, 21, 22, 23, 24, 25] have greatly promoted the development of polyp segmentation tasks. Akbari et al. [26] proposed a polyp segmentation model using a fully convolutional neural network, whose segmentation results are significantly better than traditional solutions. Brandao et al. [27] used the shape from the shading strategy to restore depth, merging the result into an RGB model to provide richer feature representations. More recently, encoder-decoder-based models, such as U-Net [4], UNet++ [28], and ResUNet++ [29], have gradually come to dominate the field with excellent performance. Sun et al. [30] introduced a dilated convolution to extract and aggregate high-level semantic features with resolution retention to improve the encoder network. Psi-Net [31] introduced a multi-task segmentation model that combines contour and distance map estimation to assist segmentation mask prediction. Hemin et al. [32] first attempted to use a deeper feature extractor to perform polyp segmentation based on Mask R-CNN [33].

Different from the methods based on U-Net [4, 28, 34], PraNet [5] uses reverse attention modules to mine boundary information with a global feature map, which is generated by a parallel partial decoder from high-level features. Polyp-Net [35] proposed a dual-tree wavelet pooling CNN with a local gradient-weighted embedding level set, effectively avoiding erroneous information in high signal areas, thereby significantly reducing the false positive rate. Rahim et al. [36] proposed to use different convolution kernels for the same hidden layer for deeper feature extraction with MISH and rectified linear unit activation functions for deep feature propagation and smooth non-monotonicity. In addition, they adopted joint generalized intersections, which overcome scale invariance, rotation, and shape differences. Jha et al. [37] designed a real-time polyp segmentation method called ColonSNet. For the first time, Ahmed et al. [38] applied the generative adversarial network to the field of polyp segmentation. Another interesting idea proposed by Thambawita et al. [39] is introducing pyramid-based augmentation into the polyp segmentation task. Further, Tomar et al. [40] designed a dual decoder attention network based on ResUNet++ for polyp segmentation. More recently, MSEG [41] improved the PraNet and proposed a simple encoder-decoder structure. Specifically, they used Hardnet [42] to replace the original backbone network Res2Net50 backbone network and removed the attention mechanism to achieve faster and more accurate polyp segmentation. As an early attempt, Transfuse [43] was the first to employ a two-branch architecture combining CNNs and transformers in a parallel style. DCRNet [44] uses external and internal context relations modules to separately estimate the similarity between each location and all other locations in the same and different images. MSNet [45] introduced a multi-scale subtraction network to eliminate redundancy and complementary information between the multi-scale features. Providing a comprehensive review of polyp segmentation is beyond the scope of this paper. In Tab. I, however, we briefly survey representative works related to ours.

Transformers use multi-head self-attention (MHSA) layers to model long-term dependencies. Unlike the convolutional layer, the MHSA layer has dynamic weights and a global receptive field, making it more flexible and effective. The transformer [65] was first proposed by Vaswani et al. for the machine translation task and has since extensively influenced the natural language processing field. To apply transformers to computer vision tasks, Dosovitskiy et al. [66] proposed a vision transformer (ViT), which was the first pure transformer for image classification. ViT divides an image into multiple patches, which are sequentially sent to a transformer encoder after being encoded, and then an MLP is used to perform image classification. HVT [67] is based on a hierarchical progressive pooling method to compress the sequence length of a token and reduce the redundancy and number of calculations in ViT. The pooling-based vision transformer [68] draws on the principle of CNNs whereby, as the depth increases, the number of feature map channels increases, and the spatial dimension decreases. Yuan et al. [69] pointed out that the simple token structure in ViT cannot capture important local features, such as edges and lines, which reduces the training efficiency and leads to redundant attention mechanisms. T2T ViT was thus proposed to use layer-by-layer tokens-to-token transformation to gradually merge neighboring tokens and model local features while reducing the token’s length. TNT [70] employs a transformer suitable for fine-grained image tasks, which divides the original image patch and conducts self-attention mechanism calculations in smaller units. Meanwhile, external and internal transformers are used to extract global and local features.

To adapt to dense prediction tasks such as semantic segmentation, several methods [71, 72, 73, 74, 75, 76, 77] have also introduced the pyramid structure of CNNs to the design of transformer backbones. For instance, PVT-based models [71, 72] use a hierarchical transformer with four stages, showing that a pure transformer backbone can be as versatile as its CNN counterparts, and performs better in detection and segmentation tasks. In this work, we design a new transformer-based polyp segmentation framework, which can accurately locate the boundaries of polyps even in extreme scenarios.

As shown in Fig. 2, our Polyp-PVT consists of 4 key modules: namely, a pyramid vision transformer (PVT) encoder, cascaded fusion module (CFM), camouflage identification module (CIM), and similarity aggregation module (SAM). Specifically, the PVT extracts multi-scale long-range dependencies features from the input image. The CFM is employed to collect semantic cues and locate polyps by aggregating high-level features progressively. The CIM is designed to remove noise and enhance low-level representation information of polyps, including texture, color, and edges. The SAM is adopted to fuse the low- and high-level features provided by the CIM and CFM, effectively transmitting the information from the pixel-level polyp to the entire polyp.

Given an input image $I\in\mathbb{R}^{H\times W\times 3}$, we use the transformer-based backbone [71] to extract four pyramid features $X_{i}\in\mathbb{R}^{\frac{H}{2^{i+1}}\times\frac{W}{2^{i+1}}\times C_{i}}$, where $C_{i}\in\{64,128,320,512\}$ and $i\in\{1,2,3,4\}$. Then, we adjust the channel of three high-level features $X_{2}$, $X_{3}$ and $X_{4}$ to 32 through three convolutional units and feed them (i.e., $X_{2}^{{}^{\prime}}$, $X_{3}^{{}^{\prime}}$, and $X_{4}^{{}^{\prime}}$) to CFM to fuse, leading a feature map $T_{1}\in\mathbb{R}^{\frac{H}{8}\times\frac{W}{8}\times 32}$. Meanwhile, low-level features $X_{1}$ are converted to $T_{2}\in\mathbb{R}^{\frac{H}{4}\times\frac{W}{4}\times 64}$ by the CIM. After that, the $T_{1}$ and $T_{2}$ are aligned and fused by SAM, yielding the final feature map $F\in\mathbb{R}^{\frac{H}{8}\times\frac{W}{8}\times 32}$. Finally, $F$ is fed into a $1\times 1$ convolutional layer to predict the polyp segmentation result $P_{2}$. We use the sum of $P_{1}$ and $P_{2}$ as the final prediction. During training, we optimize the model with a main loss $\mathcal{L}_{\rm main}$ and an auxiliary loss $\mathcal{L}_{\rm aux}$. The main loss is calculated between the final segmentation result $P_{2}$ and the ground truth (GT), which is used to optimize the final polyp segmentation result. Similarly, the auxiliary loss is used to supervise the intermediate result $P_{1}$ generated by the CFM.

Due to uncontrolled factors in their acquisition, polyp images tend to contain significant noise, such as motion blur, rotation, and reflection. Some recent works [78, 79] have found that the vision transformer [66, 71, 72] demonstrates stronger performance and better robustness to input disturbances than CNNs [17, 16]. Inspired by this, we use a vision transformer as our backbone network to extract more robust and powerful features for polyp segmentation. Different from [73, 66] that uses a fixed “columnar” structure or shifted windowing manner, the PVT [71] is a pyramid architecture whose representation is calculated with spatial-reduction attention operations; thus it enables to reduce the resource consumption. Note that the proposed model is backbone-independent; other famous transformer backbones are feasible in our framework. Specifically, we adopt the PVTv2 [72], which is the improved version of PVT with a more powerful feature extraction ability. To adapt PVTv2 to the polyp segmentation task, we remove the last classification layer and design a polyp segmentation head on top of four multi-scale feature maps (i.e., $X_{1}$, $X_{2}$, $X_{3}$, and $X_{4}$) generated by different stages. Among these feature maps, $X_{1}$ gives detailed appearance information of polyps, and $X_{2}$, $X_{3}$, and $X_{4}$ provide high-level features.

To balance the accuracy and computational resources, we follow recent popular practices [80, 5] to implement the cascaded fuse module (CFM). Specifically, we define $\mathcal{F}(\cdot)$ as a convolutional unit composed of a $3\times 3$ convolutional layer with padding set to 1, batch normalization [81] and ReLU [82]. As shown in Fig. 2 (b), the CFM mainly consists of two cascaded parts, as follows:

(1) In part one, we up-sample the highest-level feature map $X_{4}^{{}^{\prime}}$ to the same size as $X_{3}^{{}^{\prime}}$ and then pass the result through two convolutional units $\mathcal{F}_{1}(\cdot)$ and $\mathcal{F}_{2}(\cdot)$, yieldings: $X_{4}^{1}$ and $X_{4}^{2}$. Then, we multiply $X_{4}^{1}$ and $X_{3}^{{}^{\prime}}$ and concatenate the result with $X_{4}^{2}$. Finally, we use a convolution unit $\mathcal{F}_{3}(\cdot)$ to smooth the concatenated feature, yielding fused feature map $X_{34}\in\mathbb{R}^{\frac{H}{16}\times\frac{W}{16}\times 32}$. The process can be summarized as Eqn. 1.

where “$\odot$” denotes the Hadamard product, and $\text{Concat}(\cdot)$ is the concatenation operation along the channel dimension.

(2) As shown Eqn. 2, the second part follows a similar process to part one. Firstly, we up-sample $X_{4}^{{}^{\prime}}$, $X_{3}^{{}^{\prime}}$, $X_{34}$ to the same size as $X_{2}^{{}^{\prime}}$, and smooth them using convolutional units $\mathcal{F}_{4}(\cdot)$, $\mathcal{F}_{5}(\cdot)$, and $\mathcal{F}_{6}(\cdot)$, respectively. Then, we multiply the smoothed $X_{4}^{{}^{\prime}}$ and $X_{3}^{{}^{\prime}}$ with $X_{2}^{{}^{\prime}}$, and concatenate the resulting map with up-sampled and smoothed $X_{34}$. Finally, we feed the concatenated feature map into two convolutional units (i.e., $\mathcal{F}_{7}(\cdot)$ and $\mathcal{F}_{8}(\cdot)$) to reduce the dimension, and obtain $T_{1}\in\mathbb{R}^{\frac{H}{8}\times\frac{W}{8}\times 32}$, which is also the output of the CFM.

Low-level features often contain rich detail information, such as texture, color, and edges. However, polyps tend to be very similar in appearance to the background. Therefore, we need a powerful extractor to identify the polyp details.

As shown in Fig. 2 (c), we introduce a camouflage identification module (CIM) to capture the details of polyps from different dimensions of the low-level feature map $X_{1}$. Specifically, the CIM consists of a channel attention operation [83] $\text{Att}_{c}(\cdot)$ and a spatial attention operation [84] $\text{Att}_{s}(\cdot)$, which can be formulated as:

The channel attention operation $\text{Att}_{c}(\cdot)$ can be written as follow:

where $x$ is the input tensor and $\sigma(\cdot)$ is the Softmax function. $P_{\rm max}(\cdot)$ and $P_{\rm avg}(\cdot)$ denote adaptive maximum pooling and adaptive average pooling functions, respectively. ${\mathcal{H}_{i}(\cdot)},i\in\{1,2\}$ shares parameters and consists of a convolutional layer with $1\times 1$ kernel size to reduce the channel dimension 16 times, followed by a ReLU layer and another $1\times 1$ convolutional layer to recover the original channel dimension. The spatial attention operation $\text{Att}_{s}(\cdot)$ can be formulated as:

where $R_{\rm max}(\cdot)$ and $R_{\rm avg}(\cdot)$ represent the maximum and average values obtained along the channel dimension, respectively. $\mathcal{G}(\cdot)$ is a $7\times 7$ convolutional layer with padding set to 3.

To explore high-order relations between the lower-level local features from CIM and higher-level cues from CFM. We introduce the non-local [85, 86] operation under graph convolution domain [87] to implement our similarity aggregation module (SAM). As a result, SAM can inject detailed appearance features into high-level semantic features using global attention.

Given the feature map ${T_{1}}$, which contains high-level semantic information, and ${T_{2}}$ with rich appearance details, we fuse them through self-attention. First, two linear mapping functions $W_{\theta}(\cdot)$ and $W_{\phi}(\cdot)$ are applied on ${T_{1}}$ to reduce the dimension and obtain feature maps $Q\in\mathbb{R}^{\frac{H}{8}\times\frac{W}{8}\times 16}$ and $K\in\mathbb{R}^{\frac{H}{8}\times\frac{W}{8}\times 16}$. Here, we take a convolution operation with a kernel size of $1\times 1$ as the linear mapping process. This process can be expressed as follows:

For ${T_{2}}$, we use a convolutional unit $W_{g}(\cdot)$ to reduce the channel dimension to 32 and interpolate it to the same size as ${T_{1}}$. Then, we apply a Softmax function on the channel dimension and choose the second channel as the attention map, leading to ${T^{{}^{\prime}}_{2}}\in\mathbb{R}^{\frac{H}{8}\times\frac{W}{8}\times 1}$. These operations are represented as $\text{F}(\cdot)$ in Fig. 3. Next, we calculate the Hadamard product between $K$ and ${T^{{}^{\prime}}_{2}}$. This operation assigns different weights to different pixels, increasing the weight of edge pixels. After that, we use an adaptive pooling operation to reduce the displacement of features and apply a center crop on it to obtain the feature map $V\in\mathbb{R}^{4\times 4\times 16}$. In summary, the process can be formulated as follows:

where $\text{AP}(\cdot)$ denotes the pooling and crop operations.

Then, we establish the correlation between each pixel in $V$ and $K$ through an inner product, which is written as follows:

where ”$\otimes$” denotes the inner product operation. $V^{\mathsf{T}}$ is the transpose of $V$ and $f$ is the correlation attention map.

After obtaining the correlation attention map $f$, we multiply it with the feature map $Q$, and the result features are fed to the graph convolutional layer [86] $\text{GCN}(\cdot)$, leading to $G\in\mathbb{R}^{4\times 4\times 16}$. Same to [86], we calculate the inner product between $f$ and $G$ as Eqn. 9, reconstructing the graph domain features into the original structural features:

The reconstructed feature map $Y^{\prime}$ is adjusted to the same channel sizes with $Y$ by a convolutional layer $W_{z}(\cdot)$ with $1\times 1$ kernel size, and then combined with the feature ${T_{1}}$ to obtain the final output $Z\in\mathbb{R}^{\frac{H}{8}\times\frac{W}{8}\times 32}$ of the SAM. Eqn. 10 summarizes the details of this process:

Our loss function can be formulated as Eqn. 11:

where $\mathcal{L}_{\rm main}$ and $\mathcal{L}_{\rm aux}$ are the main loss and auxiliary loss, respectively. The main loss $\mathcal{L}_{\rm main}$ is calculated between the final segmentation result $P_{2}$ and ground truth $G$, which can be written as:

The auxiliary loss $\mathcal{L}_{\rm aux}$ is calculated between the intermediate result $P_{1}$ from the CFM and ground truth $G$, which can be formulated as:

$\mathcal{L}^{w}_{\text{IoU}}(\cdot)$ and $\mathcal{L}^{w}_{\text{BCE}}(\cdot)$ are the weighted intersection over union (IoU) loss [88] and weighted binary cross entropy (BCE) loss [88], which restrict the prediction map in terms of the global structure (object-level) and local details (pixel-level) perspectives. Unlike the standard BCE loss function, which treats all pixels equally, $\mathcal{L}^{w}_{\text{BCE}}(\cdot)$ considers the importance of each pixel and assigns higher weights to hard pixels. Furthermore, compared to the standard IoU loss, $\mathcal{L}^{w}_{\text{IoU}}(\cdot)$ pays more attention to the hard pixels.

We implement our Polyp-PVT with the PyTorch framework and use a Tesla P100 to accelerate the calculations. Considering the differences in the sizes of each polyp image, we adopt a multi-scale strategy [5, 41] in the training stage. The hyperparameter details are as follows. To update the network parameters, we use the AdamW [89] optimizer, which is widely used in transformer networks [71, 73, 72]. The learning rate is set to 1e-4 and the weight decay is adjusted to 1e-4 too. Further, we resize the input images to $352\times 352$ with a mini-batch size of 16 for 100 epochs. More details about the training loss cures, parameter setting, and network parameters are shown in Fig. 4, Tab. II, and Tab. III, respectively. The total training time is nearly 3 hours to achieve the best (e.g., 30 epochs) performance. For testing, we only resize the images to $352\times 352$ without any post-processing optimization strategies.

We employ six widely-used evaluation metrics, including Dice [90], IoU, mean absolute error (MAE), weighted F-measure (${F}_{\beta}^{w}$) [91], S-measure (${S}_{\alpha}$) [92], and E-measure (${E}_{\xi}$) [93, 94] to evaluate the model performances. Among these metrics, Dice and IoU are similarity measures at the regional level, which mainly focus on the internal consistency of segmented objects. Here, we report the mean value of Dice and IoU, denoted as mDic and mIoU, respectively. MAE is a pixel-by-pixel comparison indicator that represents the average value of the absolute error between the predicted value and the true value. Weighted F-measure (${F}_{\beta}^{w}$) comprehensively considers the recall and precision and eliminates the effect of considering each pixel equally in conventional indicators. S-measure (${S}_{\alpha}$) focuses on the structural similarity of target prospects at the region and object level. E-measure (${E}_{\xi}$) is used to evaluate the segmentation results at the pixel and image level. We report the mean and max value of E-measure, denoted as $m{E}_{\xi}$ and $max{E}_{\xi}$, respectively. The evaluation toolbox is derived from https://github.com/DengPingFan/PraNet.

Datasets. Following the experimental setups in PraNet [5], we adopt five challenging public datasets, including Kvasir-SEG [13], ClinicDB [8], ColonDB [10], Endoscene [14] and ETIS [9] to verify the effectiveness of our framework.

Models. We collect several open source models from the field of polyp segmentation, for a total of nine comparative models, including U-Net [4], UNet++ [28], PraNet [5], SFA [95], MSEG [41], ACSNet [49], DCRNet [44], EU-Net [52] and SANet [7]. For a fair comparison, we use their open-source codes to evaluate the same training and testing sets. Note that the SFA results are generated using the released test model.

Settings. We use the ClinicDB and Kvasir-SEG datasets to evaluate the learning ability of the proposed model. ClinicDB contains 612 images, which are extracted from 31 colonoscopy videos. Kvasir-SEG is collected from the polyp class in the Kvasir dataset and includes 1,000 polyp images. Following PraNet, we adopt the same 900 and 548 images from ClinicDB and Kvasir-SEG datasets as the training set, and the remaining 64 and 100 images are employed as the respective test sets.

Results. As can be seen in Tab. IV, our model is superior to the current methods, demonstrating that it has a better learning ability. On the Kvasir-SEG dataset, the mDic score of our model is 1.3% higher than that of the second-best model, SANet, and 1.9% higher than that of PraNet. On the ClinicDB dataset, the mDic score of our model is 2.1% higher than that of SANet and 3.8% higher than that of PraNet.

Settings. To verify the generalization performance of the model, we test it on three unseen (i.e., Polycentric) datasets, namely ETIS, ColonDB, and EndoScene. There are 196 images in ETIS, 380 images in ColonDB, and 60 images in EndoScene. It is worth noting that the images in these datasets belong to different medical centers. In other words, the model has not seen their training data, which is different from the verification methods of ClinicDB and Kvasir-SEG.

Results. The results are shown in Tab. VI and Tab. V. As can be seen, our Polyp-PVT achieves a good generalization performance compared with the existing models. And our model generalizes easily to multicentric (or unseen) data with different domains/distributions. On ColonDB, it is ahead of the second-best SANet and classical PraNet by 5.5% and 9.6%, respectively. On ETIS, we exceed the SANet and PraNet by 3.7% and 15.9%, respectively. In addition, on EndoScene, our model is better than SANet and PraNet by 1.2% and 2.9%, respectively. Moreover, to prove the generalization ability of Polyp-PVT, we present the max Dice results in Fig. 5, where our model shows a steady improvement on both ColonDB and ETIS. In addition, we show the standard deviation (SD) of the mean dice (mDic) between our model and others in Tab. VII. As seen, there is not much difference in SD between our model and the comparison model, and they are both stable and balanced.

Fig. 6 and Fig. 7 show the visualization results of our model and the compared models. We find that our results have two advantages.

• •

  Our model is able to adapt to data under different conditions. That is, it maintains a stable recognition and segmentation ability under different acquisition environments (different lighting, contrast, reflection, motion blur, small objects, and rotation).

• •

  The model segmentation results have internal consistency and predicted edges are closer to the ground-truth labels. We also provide FROC curves on ColonDB in Fig. 8, and our result is at the top, indicating that our effect achieves the best.

We describe in detail the effectiveness of each component on the overall model. The training, testing, and hyperparameter settings are the same as mentioned in Sec. III-G. The results are shown in Tab. VIII.

Components. We use PVTv2 [72] as our baseline (Bas.) and evaluate module effectiveness by removing or replacing components from the complete Polyp-PVT and comparing the variants with the standard version. The standard version is denoted as “Polyp-PVT (PVT+CFM+CIM+SAM)”, where “CFM”, “CIM” and “SAM” indicate the usage of the CFM, CIM, and SAM, respectively.

Effectiveness of CFM. To analyze the effectiveness of the CFM, a version of “Polyp-PVT (w/o CFM)” is trained. Tab. VIII shows that the model without the CFM drops sharply on all five datasets compared to the standard Polyp-PVT. In particular, the mDic is reduced from 0.937 to 0.915 on ClinicDB.

Effectiveness of CIM. To demonstrate the ability of the CIM, we also remove it from Polyp-PVT, denoting this as “Polyp-PVT (w/o CIM)”. As shown in Tab. VIII, this variant performs worse than the overall Polyp-PVT. Specifically, removing the CIM causes the mDic to decrease by 1.8% on Endoscene. Meanwhile, it is obvious that the lack of the CIM introduces significant noise (please refer to Fig. 10). In order to further explore the internal of CIM, the feature visualizations of the two main configurations inside the CIM are shown in Fig 9. It can be seen that the low-level features have a large amount of detailed information. Still, the differences between polyps and other normal tissues cannot be mined directly from this information. Thanks to the channel attention and spatial attention mechanism, information such as details and edges of polyps can be discerned from a large amount of redundant information.

Effectiveness of SAM. Similarly, we test the effectiveness of the SAM module by removing it from the overall Polyp-PVT and replacing it with an element-wise addition operation, which is denoted as “Polyp-PVT (w/o SAM)”. The performance of the complete Polyp-PVT shows an improvement of 2.9% and 3.1% in terms of mDic and mIoU, respectively, on ColonDB. Fig. 10 shows the benefits of SAM more intuitively. It is found that the lack of the SAM leads to more detailed errors or even missed inspections. As reported in Tab IX, we add more results on the GCN in the SAM module. The experimental results further illustrate that GCN plays a key role. The effect of the lack of GCN is significantly reduced, and the effect is improved after replacing it with convolution. Still, GCN can significantly exceed the capabilities of the convolution module. The experimental results also verified the importance of GCN’s large receptive field and rotation insensitivity to polyp segmentation. The rotational robustness of GCN is stronger than convolutions. As shown in Tab X, under the condition of large rotation (15 degrees), GCN has better adaptability to image rotation than convolutions. To further explore the role of SAM, we visualized P1 and P2, and the results of P1 and P2 are shown in Fig 11. Compared with P1, P2 has higher reliability in error recognition and identification of uncertain regions. This is mainly due to the large number of low-level details collected by CIM and mining local pixels and global semantic cues from the polyp area of SAM.

To validate the superiority of the proposed model, we conduct experiments on the video polyp segmentation datasets. For a fair comparison, we re-train our model with the same training datasets and use the same testing set as PNS-Net [64, 97]. We compare our model on three standard benchmarks (i.e., CVC-300-TV [96], CVC-612-T [8], and CVC-612-V [8]) against six cutting-edge approaches, including U-Net [4], UNet++ [28], ResUNet++ [29], ACSNet [49], PraNet [5], PNS-Net [64], in Tab. XI and Tab. XII. Note that PNS-Net provides all the prediction maps of the compared methods. As seen, our method is very competitive and far ahead of the best existing model, PNS-Net, by 3.1% and 6.7% on CVC-612-V and CVC-300-TV, respectively, in terms of mDice.

Although the proposed Polyp-PVT model surpasses existing algorithms, it still performs poorly in certain cases. We present some failure cases in Fig. 12. As can be seen, one major limitation is the inability to detect accurate polyp boundaries with overlapping light and shadow ($1^{st}$ row). Our model can identify the location information of polyps (green mask in $1^{st}$ row), but it regards the light and shadow part of the edge as the polyp (red mask in $1^{st}$ row). More deadly, our model incorrectly predicts the reflective point as a polyp (red mask in $2^{nd}$ and $3^{rd}$ rows). We notice that the reflective points are very salient in the image. Therefore, we speculate that the prediction may be based on only these points. More importantly, we believe that a simple way is to convert the input image into a gray image, which can eliminate the reflection and overlap of light and shadow to assist the model in judgment.