TEC-Net: Vision Transformer Embrace Convolutional Neural Networks for Medical Image Segmentation

Different from traditional medical image segmentation algorithms, the algorithms based on deep learning can learn the high-dimensional feature information of medical images through a multi-layer network structure. Among various deep-learning networks related to medical image segmentation, CNN perform extremely well. CNN can effectively learn from large-scale medical datasets to distinguish features and extract the prior knowledge, making them an important part of smart medical image analysis systems.

In 2015, Ronnebreger et al. were inspired by the FCN Long et al. (2015) network and designed the first end-to-end network U-Net Ronneberger et al. (2015) for medical image segmentation in the ISBI cell tracking challenge. U-Net adopts a symmetric encoder and decoder structure, which can make full use of the local details of medical images and reduce the dependence on training datasets. Therefore, on the case of small datasets, U-Net can still achieve good medical image segmentation results. Based on U-Net, Alom et al. designed R2U-Net Alom et al. (2018) by combining U-Net, ResNet Song et al. (2020), and recurrent neural network (RCNN) Girshick et al. (2014), which has achieved good performance on multiple medical image segmentation datasets such as blood vessels and retinas. To further improve the performance of U-Net, Gu et al. introduced dynamic convolution Chen et al. (2020) into U-Net and proposed CA-Net Gu et al. (2020). Experiments on medical datasets show that CA-Net can not only improve the segmentation accuracy of medical images but also reduce the training time of the network. Inspired by the idea of residual connection and deformable convolution Dai et al. (2017), Yang et al. added a residual deformable convolution to U-Net, and proposed DCU-Net Yang et al. (2022). DCU-Net shows a more advanced segmentation effect than U-Net on DRIVE medical dataset. Lei et al. designed SGU-Net Lei et al. (2023) based on U-Net, and proposed an ultralight convolution module and additional adversarial shape-constraint that can significantly improve the segmentation accuracy of abdominal medical images through self-supervised training. Although CNN have made great progress in network structure, the main reason for their success is due to the invariance in dealing with different scales and the inductive bias in local modeling. Although this fixed receptive field improves the computational efficiency of CNN, it also limits the ability to capture the relationship between distant pixels in medical images and lacks the ability to model medical images in a long-range.

In 2017, Vaswani et al. Vaswani et al. (2017) proposed the first Transformer network. Because of its unique structure, Transformer obtains the ability of processing indefinite-length input, establishs long-range dependency relationship, and captures global features of input data. Transformer’s success is mainly attributed to the self-attention (SA) mechanism because it can capture long-range dependency.

With the excellent performance of the Transformer in NLP fields, ViT Dosovitskiy et al. (2020) firstly applies Transformer to the field of image processing, capturing the global context information of input images through multiple cascaded Transformer layers, making Transformer a great success in image classification tasks. Then, Chen et al. proposed TransUNet Chen et al. (2021), which opened a new era of Transformer in the field of medical image segmentation. As TransUNet directly uses the Transformer network designed for NLP in the field of image segmentation, the size of the input image block is fixed and the calculation is massive. To solve this problem, Valanarasu et al. proposed MedT Valanarasu et al. (2021) for medical image segmentation. It adds the gating mechanism to the network, and the gating parameters can be automatically adjusted to obtain the position embedding weight suitable for datasets of different sizes. Since images are more diverse than text and have high resolution, Cao et al. proposed a pure Transformer network Swin-Unet Cao et al. (2023) for medical image segmentation by combining the shifted window multi-head self-attention (SW-MSA) in Swin Transformer Liu et al. (2021). Swin-Unet achieved the most advanced segmentation performance at that time on Synapse and ACDC multi-organ segmentation datasets. In order to better use Transformer to process dermoscopic image data, Wang et al. designed BAT Wang et al. (2021a) network based on the edge detection idea. The proposed boundary-wise attention gate (BAG) can fully utilize the prior knowledge of image boundaries to capture more details of medical images. BAT achieves amazing segmentation Dice value on the skin lesion datasets and surpasses many of the latest medical image segmentation networks.

Compared with the previous vanilla convolution Ronneberger et al. (2015), dynamic convolution Chen et al. (2020) Li et al. (2021) and deformable convolution Dai et al. (2017), our DDConv can not only adaptively change the weight coefficient and deformation offset of the convolution according to the medical image task, but also better adapt to the shape of organs and small lesions with large-scale changes in medical images, and additionally, it can improve the local feature expression ability of the segmentation network. Compared with the self-attention mechanism in existing Transformer architectures Cao et al. (2023) Wang et al. (2021a), our (S)W-ACAM requires fewer parameters and less computation while it is capable of capturing the global cross-dimensional long-range dependency in the medical images, and improving the global feature expression ability of the segmentation network. Our TEC-Net does not require a large amount of labeled data for pre-training, but it can maximize the retention of local details and global semantic information in medical images. It achieves the best segmentation performance on dermoscopic images, liver datasets, and cardiac multi-organ datasets.

The fusion of local and global features are clearly helpful for improving medical image segmentation. CNN capture local features of medical images through convolution operation and hierarchical feature representation. In contrast, the Transformer network realizes the extraction of global features in medical images through the cascaded self-attention mechanism and the matrix operation with context interaction. In order to make full use of local details and global semantic features of medical images, we design a parallel interactive network architecture TEC-Net. The overall architecture of the network is shown in Fig. 1(a).

TEC-Net fully considers the complementary properties of CNN and Transformer. During the forward propagation process, TEC-Net continuously feeds the local details extracted by the CNN branch to the decoder of the Transformer branch. Similarly, TEC-Net also feeds the global long-range relationships captured by the Transformer branch to the decoder of the CNN branch. Obviously, the proposed TEC-Net provides better local and global feature representation than pure CNN or Transformer networks, and it shows great potential in the field of medical image segmentation.

Specifically, TEC-Net consists of four components: a patch embedding model, a dynamically adaptive CNN branch, a cross-dimensional fusion Transformer branch, and a feature fusion module. Among them, the dynamically adaptive CNN branch and the cross-dimensional fusion Transformer branch follow the design of U-Net and Swin-Unet, respectively. The dynamically adaptive CNN branch consists of seven main stages. By using the weight coefficient and deformation offset adaptive DDConv in each stage, the segmentation network can better understand the local semantic features of medical images, better perceives the subtle changes of human organs or lesions, and improves the ability to extract multi-scale change targets in medical images. Similarly, the cross-dimensional fusion Transformer branch also consists of seven main stages. By using (S)W-ACAM attention in each stage, as shown in Fig. 1(b), the segmentation network can better understand the global dependency of medical images to capture the position information between different organs, and improves the separability of segmented objects and the background in medical images.

Although our TEC-Net can effectively improve the feature representation of medical images, it requires a large number of training data and network parameters due to the dual-branch structure. As the conventional Transformer network contains a lot of multi-layer perceptron (MLP) layers, which not only aggravates the training burden of the network but also causes the number of model parameters rise sharply, resulting in the slow training for the model. Inspired by the idea of the Ghost network Han et al. (2020), we redesign the MLP layer in the original Transformer and proposed a lightweight perceptron module (LPM). The LPM can help our TEC-Net not only achieve better medical image segmentation results than MLP but also greatly reduces the number of parameters and computational costs, even the Transformer can achieve good results without a lot of labeled data training. It is worth mentioning that the dual-branch structure involves mutually symmetric encoders and decoders so that the parallel interaction network structure can maximize the preservation of local features and global features in medical images.

Vanilla convolution has spatial invariance and channel specificity, so it has a limited ability to change different visual modalities when dealing with different spatial locations. At the same time, due to the limitations of the receptive field, it is difficult for vanilla convolution to extract features of small targets or targets with blurred edges. Therefore, vanilla convolution inevitably has poor adaptability and weak generalization ability for feature representation of complex medical images. Although the existing deformable convolution Dai et al. (2017) and dynamic convolution Chen et al. (2020) Li et al. (2021) outperforms vanilla convolution to a certain extent, they still face a problem of balancing the performance and size of networks when dealing with medical image segmentation.

In order to overcome the shortcomings of popular convolution operations, this paper proposes a new convolution strategy namely DDConv, as shown in Fig. 2. It can be seen that DDConv can adaptively learn the kernel deformation offset and weight coefficients according to the specific task and data distribution, so as to realize the change of both the shapes and the values of convolution kernels. It can effectively deal with the problems of large data distribution differences and large target deformation in medical image segmentation. Also, DDConv is plug-and-play and can be embedded in any network structure.

The shape change of the convolutional kernel in DDConv is based on the network learning of the deformation offsets. The segmentation network first samples the input feature map $X$ using a square convolutional kernel $S$, and then performs a weighted sum with a weight matrix $M$. The square convolution kernel $S$ determines the range of the receptive field, e.g., a $3\times 3$ convolution kernel can be expressed as:

$$S=\{(0,0),(0,1),(0,2),...,(2,1),(2,2)\},$$

(1)

then the output feature map $Y$ at the coordinate $\varphi_{n}$ can be expressed as:

$$Y\left(\varphi_{n}\right)={\sum\limits_{\varphi_{m\in S}}{S\left(\varphi_{m}\right)}}\cdot X\left(\varphi_{n}+\varphi_{m}\right),$$

(2)

when the deformation offset $\bigtriangleup\varphi_{m}=\left\{m=1,2,3,\ldots,N\right\}$ is introduced in the weight matrix $M$, $N$ is the total length of $S$. Thus the Equation (2) can be expressed as:

$$Y\left(\varphi_{n}\right)={\sum\limits_{\varphi_{m\in S}}{S\left(\varphi_{m}\right)}}\cdot X\left(\varphi_{n}+\varphi_{m}+\bigtriangleup\varphi_{m}\right).$$

(3)

Through network learning, an offset matrix with the same size as the input feature map can be finally obtained, and the matrix dimension is twice that of the input feature map.

To show the convolution kernel of DDConv is dynamic, we first present the output feature map of vanilla convolution:

$$y=\sigma(W\cdot x),$$

(4)

where $\sigma$ is the activation function, $W$ is the convolutional kernel weight matrix and $y$ is the output feature map. In contrast, the output of the feature map of DDConv is:

$$\hat{y}=\sigma\left(\left(\alpha_{1}\cdot W_{1}+\ldots+\alpha_{n}\cdot W_{n}\right)\cdot x\right),$$

(5)

where $n$ is the number of weight coefficients, $\alpha_{n}$ is the weight coefficients with learnable parameters and $\hat{y}$ is the output feature map generated by DDConv. DDConv achieves dynamic adjustment of the convolution kernel weights by linearly combining different weight matrices according to the corresponding weight coefficients before performing the convolution operation.

According to the above analysis, we can see that DDConv realizes the dynamic adjustment of the shape and weights of the convolution kernel. Compared with directly increasing the number and size of convolution kernels, the DDConv is simpler and more efficient. The proposed DDConv not only solves the problem of poor adaptive feature extraction ability of fixed-size convolution kernels but also overcomes the defect that different inputs share the same convolution kernel parameters. Consequently, our DDConv can be used to improve the segmentation accuracy of small targets and large targets with blurred edges in medical images.

The self-attention mechanism is the core computing unit in Transformer networks, which realizes the capture of long-range dependency of feature maps by utilizing matrix operations. However, the self-attention mechanism only considers the dependency in the spatial dimension but not the cross-dimensional dependency between spatial and channels Hong et al. (2021). Therefore, when dealing with medical image segmentation with low contrast and high-density noise, the self-attention mechanism is easy to confuse segmentation targets with their background, resulting in poor segmentation results.

To solve the problems mentioned above, we propose a new cross-dimensional self-attention module called (S)W-ACAM. As shown in Fig. 3, (S)W-ACAM consists of four parallel branches, the top two branches are the conventional dual attention module and the bottom two branches are cross-dimensional attention modules. Compared to popular self-attention modules such as spatial self-attention, channel self-attention, and dual self-attention, our proposed (S)W-ACAM can not only fully extract the long-range dependency of both spatial and channels, but also capture the cross-dimensional long-range dependency between spatial and channels. These four branches complement each other, provide richer long-range dependency relationships, enhance the separability between the foreground and background, and thus improve the segmentation results for medical images.

The standard Transformer architecture Dosovitskiy et al. (2020) uses the global self-attention method to calculate the relationship between one token and all other tokens. This calculation method is complex since the computational costs will increase exponentially with the increase of image size. In order to improve the calculation efficiency, we use the shifted window calculation method similar to that in Swin Transformer Liu et al. (2021), which only calculates the self-attention in the local window. However, in the face of our (S)W-ACAM four branches module, using the shifted window method to calculate self-attention does not reduce the overall computational complexity of the module. Therefore, we also designed the compact convolutional projection. First, we reduce the local size of the medical image through the shifted window operation, then we compress the channel dimension of feature maps through the compact convolutional projection, and finally calculate the self-attention. It is worth mentioning that this method can not only better capture the global information of medical images but also significantly reduce the computational costs of the module.

Suppose an image contains $h\times w$ windows, each window size is $M\times M$, then the complexity of the (S)W-ACAM, the global MSA in the original Transformer, and the (S)W-MSA in the Swin Transformer are compared as follows:

$$\Omega\left({MSA}\right)=4hwC^{2}+2(hw)^{2}C,$$

(6)

$$\Omega\left({(S)W\mbox{-}MSA}\right)=4hwC^{2}+2M^{2}hwC,$$

(7)

$$\Omega\left({(S)W\mbox{-}ACAM}\right)=\frac{hwC^{2}}{4}+M^{2}hwC,$$

(8)

if the former term of each formula is a quadratic function of the number of patches $h\cdot w$, the latter term is linear when $M$ is fixed (the default is 7). Then the computational cost of (S)W-ACAM is smaller than MSA and (S)W-MSA.

Among the four parallel branches of (S)W-ACAM, two branches are used to capture channel correlation and spatial correlation, respectively, and the remaining two branches are used to capture the correlation between channel dimension $C$ and space dimension $H$ and vice versa (between channel dimension $C$ and space dimension $W$). After adopting the shifted window partitioning method, as shown in Fig. 1(b), the calculation process of continuous Transformer blocks is as follows:

$${\hat{T}}^{l}=W\mbox{-}ACAM\left({LN\left(T^{l-1}\right)}\right)+T^{l-1},$$

(9)

$$T^{l}=LPM\left({LN\left({\hat{T}}^{l}\right)}\right)+{\hat{T}}^{l},$$

(10)

$${\hat{T}}^{l+1}=SW\mbox{-}ACAM\left({LN\left(T^{l}\right)}\right)+T^{l},$$

(11)

$$T^{l+1}=LPM\left({LN\left({\hat{T}}^{l+1}\right)}\right)+{\hat{T}}^{l+1},$$

(12)

where ${\hat{T}}^{l}$ and $T^{l}$ represent the output features of (S)W-ACAM and LPM, respectively. W-ACAM represents window adaptive complementary attention, SW-ACAM represents shifted window adaptive complementary attention, and LPM represents lightweight perceptron module. For the specific attention calculation process of each branch, we follow the same principle in Swin Transformer as follows:

$$Attention\left({Q,K,V}\right)=SoftMax\left({\frac{QK^{T}}{\sqrt{C/8}}+B}\right)V,$$

(13)

where relative position bias $B\in{{\mathbb{R}}^{{{M}^{2}}\times{{M}^{2}}}}$, $Q,K,V\in\mathbb{R}^{M^{2}\times\frac{C}{8}}$ are query, key, and value matrices respectively. $\frac{C}{8}$ represents the dimension of query/key, and $M^{2}$ represents the number of patches.

After obtaining ${Out}_{1}$, ${Out}_{2}$, ${Out}_{3}$ and ${Out}_{4}$, the final feature fusion output is:

$$Out=\lambda_{1}\cdot{Out}_{1}+\lambda_{2}\cdot{Out}_{2}+\lambda_{3}\cdot{Out}_{3}+\lambda_{4}\cdot{Out}_{4},$$

(14)

where $\lambda_{1}$, $\lambda_{2}$, $\lambda_{3}$ and $\lambda_{4}$ are learnable parameters that enable adaptive control of the importance of each attention branch.

Different from other self-attention mechanisms, our proposed (S)W-ACAM can fully capture the correlation between spatial and channels, and reasonably use the context information of medical images to achieve long-range dependency modeling. Since our (S)W-ACAM effectively overcomes the defect that the conventional self-attention only focuses on the spatial self-attention of images and ignores the channel and cross-dimensional self-attention, it achieves better feature representation for medical images with high-density noise, low contrast and complex background.

In our task, three loss functions are used for model training, namely, the overall loss $L_{TEC}$ of TEC-Net network, the loss $L_{CNN}$ of CNN branch and the loss $L_{Trans}$ of Transformer branch.

$$L_{TEC}=L_{MSE}\left({y_{i}^{TEC},y_{i}^{label}}\right)+L_{Dice}\left({y_{i}^{TEC},y_{i}^{label}}\right),$$

(15)

$$L_{CNN}=L_{MSE}\left({y_{i}^{CNN},y_{i}^{label}}\right)+L_{Dice}\left({y_{i}^{CNN},y_{i}^{label}}\right),$$

(16)

$$L_{Trans}=L_{MSE}\left({y_{i}^{Trans},y_{i}^{label}}\right)+L_{Dice}\left({y_{i}^{Trans},y_{i}^{label}}\right),$$

(17)

where $L_{MSE}(\bullet)$ represents mean squared error loss, and $L_{Dice}(\bullet)$ represents Dice loss. $y_{i}^{TEC}$, $y_{i}^{CNN}$, $y_{i}^{Trans}$ and $y_{i}^{label}$ represent the final predicted image using our TEC-Net network, the predicted image from the CNN branch output, the predicted image from the Transformer branch output, and the label, respectively. The total loss function of TEC-Net network can be expressed as:

$$L_{Total}=\lambda L_{TEC}+\left({\left({1-\lambda}\right)/2}\right)L_{CNN}+\left({\left({1-\lambda}\right)/2}\right)L_{Trans},$$

(18)

where $\lambda=\delta e^{-5{({1-k})}^{2}}$, $\lambda$ is a Gaussian ramp-up curve, $k$ represents the number of epochs.

We have built a TEC-Net-T as a base network with a model size of 11.58 M and a computing capacity of 4.53 GFLOPs. In addition, we built the TEC-Net-B network to make a fair comparison with the latest networks such as CvT Wu et al. (2021) and PVT Wang et al. (2021b). The window size is set to 7, and the input image size is $224\times 224$. Other network parameters are set as follows:

• •

  TEC-Net-T: $layer~{}number=\left\{2,~{}2,~{}6,~{}2,~{}6,~{}2,~{}2\right\}$, $H=\left\{3,~{}6,~{}12,~{}24,~{}12,~{}6,~{}3\right\}$, $D=96$

• •

  TEC-Net-B: $layer~{}number=\left\{2,~{}2,~{}18,~{}2,~{}18,~{}2,~{}2\right\}$, $H=\left\{4,~{}8,~{}16,~{}32,~{}16,~{}8,~{}4\right\}$, $D=96$

where $D$ represents the number of image channels when entering the first layer of the dynamically adaptive CNN branch and the cross-dimensional fusion Transformer branch, $layer~{}number$ represents the number of Transformer blocks used in each stage, and $H$ represents the number of multiple heads in the Transformer branch.

We conducted experiments on the skin lesion segmentation dataset (ISIC2018) from the International Symposium on Biomedical Imaging (ISBI) Codella et al. (2019), the Liver Tumor Segmentation Challenge dataset (LiTS) from the Medical Image Computing and Computer Assisted Intervention Society (MICCAI) Bilic et al. (2023) and the Automated Cardiac Diagnosis Challenge dataset (ACDC) from the University Hospital of Dijon (France) Bernard et al. (2018). These three datasets have different data types and data distributions. Among them, the ISIC2018 dataset is electron microscope images, which contain 2,594 dermoscopic images for training, but the ground truth images of the testing set have not been released, so we performed a five-fold cross-validation on the training set for a fair comparison. The LiTS dataset is a CT image of the human abdomen, containing 131 3D CT liver scans, where 100 scans of which are used for training, and the remaining 31 scans are used for testing. The ACDC dataset is an MRI image of the human heart, which contains cardiac short-axis Cine MRI data from 100 patients. The ACDC dataset includes healthy patients, patients with previous myocardial infarction, dilated cardiomyopathy, hypertrophic cardiomyopathy, and abnormal right ventricle, with 20 scans for each group. These data are obtained over a 6 years period using two MRI scanners of two magnetic strengths (1.5T and 3.0T). In addition, all images are empirically resized to 224×224 for efficiency.

All the networks are implemented on NVIDIA GeForce RTX 3090 24GB and PyTorch 1.7. We utilized Adam with an initial learning rate of 0.001 to optimize the networks. The learning rate decreases in half when the loss on the validation set has not dropped by 10 epochs. We used mean squared error loss (MSE) and Dice loss as loss functions in our experiment.

In the experiment on the ISIC2018 dataset, we conducted an overall evaluation for SOTA networks and the proposed TEC-Net using five indicators: Dice (DI), Jaccard (JA), Sensitivity (SE), Accuracy (AC), and Specificity (SP) Chang et al. (2009). In the experiment on the LiTS-Liver dataset, we conducted an overall evaluation for SOTA networks and the proposed TEC-Net using five indicators: DI, VOE, RVD, ASD, and RMSD. In the experiment on the ACDC dataset, we conducted an overall evaluation for SOTA networks and the proposed TEC-Net using DI and 95HD Taha and Hanbury (2015).

The values of DI, JA, AC, SE, and SP range from 0 to 100, and better segmentation results mean that the values of evaluation indicators such as DI, JA, AC, SE, and SP will be higher, while the values of evaluation indicators such as VOE, RVD, ASD, RMSD, and 95HD will be lower Li et al. (2020).The 95HD is defined as the 95th quantile of Hausdorff distances (HD) instead of the maximum Lei et al. (2022).

In this paper, we selected popular SOTA networks for medical image segmentation networks U-Net Ronneberger et al. (2015), R2UNet Alom et al. (2018), Attention Unet Oktay et al. (2018), CENet Gu et al. (2019), 3D Unet Çiçek et al. (2016), V-Net Milletari et al. (2016), Swin-Unet Cao et al. (2023), TransUNet Chen et al. (2021), CvT Wu et al. (2021), PVT Wang et al. (2021b), CrossForm Wang et al. (2021c) and the proposed TEC-Net to conduct a comprehensive comparison of the three different modalities datasets, the ISIC2018, the LiTS-Liver and the ACDC.

TABLE I shows the quantitative analysis results of the proposed TEC-Net and the competitive CNN and Transformer networks on the ISIC2018 dataset. From the experimental results, we can conclude that our TEC-Net needs the minimum number of parameters and the lowest computational costs, and can obtain the best segmentation effect on the dermoscopic images without adding pre-training. Moreover, our TEC-Net-T network requires only 11.58 M of parameters and 4.53 GFLOPs of computational costs, but still achieves the second-best segmentation effect. Our TEC-Net-B network, BAT, CvT, and CrossForm have similar parameters or computational costs, but on the ISIC2018 dataset, the division Dice value of our TEC-Net-B is 1.02%, 3.00%, and 3.79% higher than that of the BAT, CvT, and CrossForm network respectively. In terms of other evaluation indicators, our TEC-Net-B is also significantly better than other competitive networks.

TABLE II shows the quantitative analysis results of the proposed TEC-Net and the competitive networks on the LiTS-Liver dataset. It can be seen from the experimental results that our TEC-Net shows great advantages in medical image segmentation, which further verifies the integrity of TEC-Net in extracting local and global features for medical images. It is worth noting that the TEC-Net-B and TEC-Net-T networks achieve good results in medical image segmentation in the first and second place, with the least number of model parameters and computational costs. The division Dice value of our TEC-Net-B network without pre-training is 1.20%, 1.03%, and 1.01% higher than that of the Swin-Unet, TransUNet, and CvT network with pre-training. In terms of other evaluation indicators, our TEC-Net-B is also significantly better than other competitive networks.

TABLE III shows the quantitative analysis results of the proposed TEC-Net and the competitive networks on the ACDC dataset. From the experimental results, it can be seen that the proposed TEC-Net still exhibits significant advantages on MRI type multi-organ segmentation datasets. Both TEC-Net-T and TEC-Net-B provide state-of-the-art segmentation effects for the left ventricle (LV), right ventricle (RV), and left ventricular myocardium (MYO). Among them, LV provides the best segmentation effect, while MYO provides a poor segmentation effect. Compared with the latest CvT, PVT, and CrossForm, the average segmentation performance of TEC-Net-B improves by 0.98%, 1.45%, and 1.38%, respectively, while the average 95HD decreases by 0.72%, 0.78%, and 0.74%, respectively. It also proves that TEC-Net has strong generalization performance on different datasets and can be flexibly applied to medical image segmentation tasks in different modalities and collection environments.

From the visualization in Fig. 4, it can be clearly seen that the TEC-Net network can effectively extract local detail features and global semantic features in medical images to the maximum extent possible. Therefore, it provides accurate segmentation results for segmentation targets with irregular edges and large deformation scales. Fig. 4(a) shows the final output result from the CNN branch, and Fig. 4(b) shows the final output result from the Transformer branch. We can find that the CNN branch captures more accurate local detail information of segmented targets. However, due to the fact that CNN branches mainly capture local features of images, they are more susceptible to noise interference. However, the Transformer branch captures more accurate position information of segmented targets. Due to the fact that the Transformer branch mainly captures the global features of the image, it is easy to overlook the detailed information of the segmented target. So, after integrating the CNN branch with the Transformer branch, the TEC-Net network fully inherits the structural and generalization advantages of CNN and Transformer, providing better local and global feature representations for medical images, demonstrating great potential in the field of medical image segmentation.

In order to fully prove the effectiveness of different modules in our TEC-Net, we conducted a series of ablation experiments on the ISIC2018 dataset. As shown in TABLE IV, we can see that the DDConv and (S)W-ACAM proposed in this paper show good performance, and the combination of these two modules, TEC-Net shows the best medical image segmentation effect.

In order to better understand the TEC-Net network, we visualized the feature maps of each stage of the network, as shown in Fig. 5. We used the linear interpolation method to restore the deep feature map with the low spatial resolution to the same size as the input and output images. In the visualization, red represents areas that the network is more concerned about, while blue represents areas that the network is less concerned about.

Through visualization images, we can clearly see the entire process of feature extraction, feature fusion, and other stages that the image undergoes after being input into the network. In the two branches of the encoder, the model can analyze and identify the semantic information of images. In the two branches of the decoder, after skip connections and feature fusion operations, the model began to pay more attention to the semantic features of target regions. During this process, the information exchange and fusion between CNN and Transformer branches play an important role in the accurate segmentation of target regions.

As a whole structure, as the number of network layers continues to deepen, the TEC-Net network gradually refines the localization and contour segmentation of semantic objects in medical images, proving the effectiveness of our proposed TEC-Net network for global information modeling and accurate segmentation of medical images.