CTformer: Convolution-free Token2Token Dilated Vision Transformer for Low-dose CT Denoising

The LDCT problem has gained lots of attention in the community due to its potential of reducing X-ray radiation. However, compared to NDCT images, LDCT images suffer from severe noise and artifacts [2] when they are applied to clinical applications. To overcome this problem, two types of algorithms have been investigated: traditional algorithms and convolutional neural networks (CNNs) [3, 4]. i) Traditional algorithms such as iterative methods suppress the artifacts and noise by using a physical model based on a certain prior. Unfortunately, these algorithms are hard to be adopted in commercial CT scanners because of the hardware limitations and high computational cost [5]. ii) With the advent of deep learning, CNNs have been a prevailing approach for LDCT image denoising. Despite the superior learning ability aided by big data[6], CNNs are reported to be limited in capturing long-range contextual information in images [7, 8, 9, 10], which will adversely affect the retrieval of richer structural information in denoised images.

Recently, the transformer model [8] has shown excellent performance in computer vision [11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21]. Dosovitskiy et al. proposed the first vision transformer (ViT) by simply mapping an image into 16$\times$16 patches (this operation is commonly referred to as tokenization) in analogy to words in a sentence in natural language processing [14]. Yuan et al. further proposed a Token2Token method to empower the transformer model with a diverse information encoding [10]. Next, Liu et al. designed a swin transformer to include patch fusion and cyclic shift to enlarge the perception of contextual information in tokens [9]. Moreover, Choromanski et al. proposed a Performer transformer to reduce the computational complexity of the self-attention by approximating the inherent softmax operator [21]. Currently, the transformer model is poised to replace CNNs as the mainstream deep learning model. On the one hand, compared to CNNs, the transformer model is good at capturing global information and long-range feature interactions, resulting in the utilization of richer information. As shown in Fig. 1, the transformer has diversified and effective features, while the CNN model has many inactive features. On the other hand, the transformer model enjoys higher visual interpretability by the virtue of its inherent self-attention block [22, 23, 24]. However, a typical CNN model contains no generic explanation modules [25].

Despite the success and great promise, the transformer has been little investigated in LDCT denoising. In our opinion, the transformer model is suitable for LDCT denoising problem. Other than the effectiveness, a transformer is more desirable for physicians because it is self-explanatory [26], e.g., allowing a physician to make sense of the model’s logic. To the best of our knowledge, Zhang et al. pioneered to apply the transformer in LDCT denoising [27]. Although this model achieves the state-of-the-art performance, it has imperfections in three aspects: i) The model uses the vanilla transformer which can not fully explore the potential of the transformer, as relevant studies are rapidly advancing. ii) Intensive convolutions are included in the model, making their model essentially a hybrid model. Thus, the merits of using a transformer are insufficiently justified. iii) Their work neglects the interpretability that is essential for clinical applications [28].

We aim to fully explore the potential of transformers in LDCT denoising. Specifically, we propose a Convolution-free Token2Token Dilated Vision Transformer (CTformer) for low-dose CT denoising. The CTformer has the following characteristics: i) Although the convolution is instrumental to capture local features when it is combined with transformers on small datasets, it is not a necessity for the performance because the token rearrangement can also help complement the local information. Therefore, we completely exclude convolution operations in the proposed CTformer. To the best of our knowledge, the CTformer is the first pure transformer for LDCT denoising. ii) The dilation and a cyclic shift are used in the Token2Token to enlarge the receptive field, thereby gaining broader contextual information from the feature maps and reducing the computational cost. iii) We utilize an overlapped inference mechanism to address the boundary artifact that is common in the encoder-decoder denoising models. iv) We develop interpretability for the CTformer with the visual attention maps and an explanatory graph that shed light on how the CTformer discriminates key structures from noise as well as hierarchical attention flow across layers. Experiments results suggest that the CTformer delivers superior denoising performance over other state-of-the-arts with fewer trainable parameters and multiply-accumulate operations (MACs).

In summary, our contributions are threefold: i) This work is the forerunner to apply the vision transformer to LDCT denoising problem. What’s more, the proposed CTformer is the first pure transformer. ii) We introduce dilation and cyclic shift to enhance the tokenization process in the model, utilize a new inference mechanism to fix the boundary artifacts, and develop the interpretation methods to unveil the model’s denoising patterns. iii) Our experimental results demonstrate the superior denoising performance and model efficiency of the CTformer for LDCT denoising.

The previous studies for the LDCT denoising problem can be categorized into two classes.

Traditional algorithms. Typically, these methods incorporate a physical prior into an iterative reconstruction framework to suppress noise. For example, compressed sensing (CS) has been widely used for the LDCT problem by adopting a sparse representation [29], i.e., the total variation minimization assumes that the clean image is piecewise constant whose gradients are sparse [30, 31, 32, 33]. Xu et al. used a dictionary to construct the sparse representation [34] for LDCT denoising. In addition to the sparsity prior, Ma et al. designed a non-local mean prior to utilize the image voxels across the whole image rather than the local region [35]. However, increasingly more studies [36, 37, 38, 39] implied that the traditional algorithms are surpassed by deep learning models driven by big data.

Convolution models. CNNs have been used for the LDCT image reconstruction. Wu et al. used a K-sparse autoencoder to learn the image features in an unsupervised fashion and minimize the distance between a normal-dose image and an iterative reconstruction result in the feature space of the autoencoder [36]. Liu et al. proposed a 3D residual convolutional network to estimate an iterative reconstruction (IR) image from an LDCT analytic reconstruction image [40]. Their method can save time because it avoids the time-consuming iterative reconstruction. He et al. proposed the 3pADMM method to address the problems of hyper-parameter optimization and prior knowledge selection in LDCT reconstruction [41].

Besides, a majority of deep LDCT denoising models focused on image post-processing. The paper of Chen et al. was a pioneer work which employed the convolution, deconvolution, and shortcut connections to prototype a residual encoder-decoder convolution neural network (RED-CNN) [37]. Yang et al. used the generative adversarial network with Wasserstein distance (WGAN) aided by a perceptual loss to improve the quality of denoised images [38]. Due to the excellent performance of WGAN in generating faithful real-world CT images and the role of the perceptual loss in structural fidelity, this model alleviated the over-smoothness in the denoised images. Li et al. employed a GAN armed with the structural similarity loss, the perceptual loss, the adversarial loss, and the sharpness loss to preserve structural details and sharp boundaries [42]. Fan et al. constructed a quadratic neuron-based autoencoder for LDCT image denoising with more robustness and efficiency as opposed to conventional CNN-based methods [39]. It is the first autoencoder based on a new type of neurons. Huang et al. proposed a two-stage residual CNN [43], where the first stage uses stationary wavelet transform for texture denoising, and the second one enhances the image structure via combining the average of NDCT images and the denoised image from the first stage.

However, CNN-based models typically lack the ability to capture global contextual information due to the limited receptive fields, thus less efficient to model the structural similarity across the whole image [27, 1, 44].

In the supervised setting, with a deep learning model, the LDCT denoising task is to learn a mapping from a paired noisy LDCT image $\boldsymbol{x}$ to a clean NDCT image $\boldsymbol{y}$. Mathematically, a neural network can be trained by optimizing a mean square error (MSE) loss function as follows:

where $f(\boldsymbol{W};\boldsymbol{x})$ is a neural network, and $\boldsymbol{W}$ is a collection of parameters for simplicity.

In this part, our model is trained and evaluated on a publicly available dataset. First, we demonstrate the superior denoising performance and the model efficiency of the CTformer over its counterparts. Then, we confirm the effectiveness of the overlapped inference mechanism. Finally, we elaborate on the model interpretability of the CTformer with the aforementioned interpretation methods.

Dataset. A publicly released dataset from 2016 NIH-AAPM-Mayo Clinic LDCT Grand Challenge³³3https://www.aapm.org/GrandChallenge/LowDoseCT/ [49] is used for model training and testing. The dataset includes $2,378$ $3.0$mm slice thickness of low-dose (quarter) and normal-dose (full) CT images from ten anonymous patients. We select the patient L$506$ data for evaluation, while the rest nine patients for model training. Data augmentation is also applied. We generate more training images by randomly rotating ($90$, $180$, or $270$ degrees) and flipping (up/down, left/right) the original image.

Experiment settings. We list the detailed experimental settings in the following:

• •

  The experiments are running on Ubuntu 18.04.5 LTS, with Intel(R) Core (TM) i9-9920X CPU @ 3.50GHz using PyTorch 1.5.0 [50] and CUDA 10.2.0. The model is trained with four NVIDIA GTX 2080Ti 11G GPUs.

• •

  The intermediate transformer block takes the authentic design, while the transformer blocks in the CTformer modules take Performer to facilitate model training. The embedding dimension for all transformer blocks is $64$.

• •

  In tokenization/detokenization, the kernel for the unfolding/folding is set to $7$ with a stride of $2$ to reduce computational cost. For the four CTformer modules, the cyclic shift strides in the T2TD/IT2TD blocks are $\{2,2,-2,-2\}$. The kernel sizes of the unfolding/folding operations are 3 with dilations of $\{2,1,1,2\}$, respectively. The strides are set to 1 to avoid the information loss. Thus, according to Eq. (9), the corresponding token numbers $n_{1}$, $n_{2}$, and $n_{3}$ for the CTformer module A, CTformer module B, and the intermediate transformer layer are computed as follows:

  $$\begin{cases}n_{1}=\Big{(}\left\lfloor\frac{64-1\times(7-1)-1}{2}+1\right\rfloor\Big{)}^{2}=841\\ n_{2}=\Big{(}\left\lfloor\frac{\sqrt{841}-2\times(3-1)-1}{1}+1\right\rfloor\Big{)}^{2}=625\\ n_{3}=\Big{(}\left\lfloor\frac{\sqrt{625}-1\times(3-1)-1}{1}+1\right\rfloor\Big{)}^{2}=529,\end{cases}$$

  (11)

  here $\mathrm{spatial}(d)=\sqrt{841}=29$ and $\mathrm{spatial}(d)=\sqrt{625}=25$ can be calculated from the reshaping process in Eq. (5). The transformer token numbers in the decoder are symmetrically arranged as $\{625,841\}$.

• •

  We randomly extract $4$ patches from all available slices for training through $4000$ epochs with a batch size of $16$. In a training batch, fewer patches with more images lead to less fluctuations and bias than more patches with fewer images because many patches from a single image usually cannot represent the overall data distribution.

• •

  Adam is adopted to minimize the MSE loss with an initial learning rate of ${1.0\times 10^{-5}}$, which gradually decreases to ${1.0\times 10^{-6}}$ with a scheduled decay rate.

• •

  A margin size of $16$ is used for overlapped inference.

Denoising performance. The performance of the CTformer is compared to other state-of-the-arts, e.g., RED-CNN [37], WGAN-VGG [38], MAP-NN [51], and AD-NET [52]. The selected models are all popular low-dose CT or natural image denoising models that were published in flagship journals. We retrain all the models based on their officially-disclosed codes.

Fig. 6 shows the results of different networks on L$506$ with Lesion No. $575$, and Fig. 7 demonstrates the ROIs from the rectangular area marked in Fig. 6. It can be seen that all methods can alleviate noise and artifacts to some extent, but the CTformer generates the clearest and the most perceptually-pleasing denoised images. Specifically, per the ROIs from Fig. 7, we find that WGAN-VGG and MAP-NN seem to introduce additional shadows and tissues. While the RED-CNN and AD-NET produce a smoother and clearer image relative to WGAN-VGG and MAP-NN, there still exists blotchy noise around the lesion. In contrast, the CTformer satisfactorily supresses the noise and artifacts, maintains high-level spatial smoothness, and keeps the structural details in the restored image. Therefore, we conclude that the CTformer is the best denoiser compared to its competitors.

Additionally, two metrics: structural similarity (SSIM) and root mean square error (RMSE) are adopted to quantitatively assess the quality of the denoised images. For fairness, we evaluate the model complexity with the number of trainable parameters (#param.) and MACs. Table I shows the average SSIM and RMSE on all slices of L506. Among the state-of-the-art methods, only AD-NET achieve an SSIM score over $0.91$, and only MAP-NN and AD-NET have an RMSE score below $10$. In contrast, our CTformer has the highest SSIM of $0.9121$ and smallest RMSE of $9.0233$. Concerning model complexity, MAP-NN has the highest MACs of $13.79$G because it uses a lot of repeated modules, while WGAN-VGG has the greatest number of trainable parameters of $34.07$M because it uses VGG as a feature extractor. In contrast, the CTformer has the smallest number of parameters and the lowest MACs. Compared to its competitors, our model has the best performance with the lowest computational cost.

Model efficiency. Model efficiency is an important issue in deep learning. To further verify the model efficiency of the CTformer, we compare the CTformer with RED-CNN, MAP-NN and AD-NET by checking the SSIM and RMSE scores from different model sizes. For the CTformer, we change the model size by revising the embedding size of the intermediate transformer block. The embedding sizes are set to $\{64,256,512,1024\}$, respectively. While for other models, we vary their sizes by using different number of filters in each layer. The filter numbers in RED-CNN, MAP-NN, and AD-Net are $\{64,96,128,256\}$, $\{64,128,256,400\}$, and $\{64,96,128,256\}$, respectively.

Fig. 8 shows the SSIM and RMSE scores of different models with respect to the number of parameters and MACs. The highlights of Fig. 8 are that the SSIM curves of the CTformer lie on the top left of other curves, while its RMSE curves lies on the bottom left. When the number of parameters and MACs are close, the CTformer always delivers the best scores compared to the RED-CNN, MAP-NN and AD-NET. We summarize that the CTformer has the superior model efficiency to its competitors.

Eliminating boundary artifacts. The overlapped inference is performed to eliminate boundary artifacts as shown in Fig. 9(a). From the ROIs in Fig. 9(b), we can see that the boundary artifacts are obvious when $\eta$ is 0 or 1 but soon become hardly perceivable when $\eta$ further increases. It is worth noting that as $\eta$ varies, the boundary artifacts can appear in different regions because the size of the patches integrated in the final image is different. To further confirm the effectiveness of the overlapped inference, quantitative analysis on the patient L$506$ is also conducted. As seen from Table II, the SSIM and RMSE scores improve fast when $\eta$ goes from 0 to 20 with a better performance on $16$. The corresponding ratio of the extra computation over the authentic computation $\sigma$ is calculated from Eq.(10): $\big{(}\frac{\lceil 512/(64-2\times 16)\rceil}{\lceil 512/64\rceil}\big{)}^{2}=4$. To sum up, the overlapped inference can sufficiently address the dense boundary artifacts.

Visual interpretation. To reveal the latent learning behavior in the CTformer, we visualize the attention maps $\mathrm{Att}=\mathrm{softmax}(\mathbf{QK}^{\top}/\sqrt{d_{k}})$ in each layer. Specifically, we derive attention maps by averaging all grids of $\mathrm{Att}$ and resize it to the size of the original image. Then, the attention map is superimposed on the image with a transparence rate 0.4.

As shown in Fig. 10, the attention map in the first layer highlights the key object parts. Specifically, there are more attentions on the edges rather than the composition of key structures like bones. Moreover, there are scattered dotted attentions on the protruding texture in the original image. For the attention map in the second layer, it basically resembles the pattern in the first attention, but sparser and less focused on the structures. Next, the pattern in the third layer becomes semantically implicit. Finally, the attention in the fourth layer tends to ignore the edges of objects and emphasize the content where noise is concentrated.

Since attentions in different layers focus on different structures, we construct an explanatory graph to illustrate the flow of attention across various layers. In our experiments, the object nodes are represented by the pixel coordinates of the image. We select the top 60 activations in the attention maps as nodes using TopK/LM selection and identify the highest activation under each node’s influence. By applying the proposed method, the whole TopK and LM graph are obtained in Figs. 11 and 12, respectively.

From the TopK graph in Fig. 11, it can be seen that the attention flows across different testing slices have very similar patterns. First, from the first to the second layer, the attentions on the edges still favor other edges in the next layer as indicated by the white circle in Fig. 11. Second, all high activations from the second layer move to the top area of the third layer in a latent manner. Last, all the top attentions in the third layer spread across the noisy area in the fourth layer. While the TopK graph identifies the flow of the top activations, the LM graph illustrates that of the local protuberant objects. As shown in Fig. 12, the attention graphs of different slices using LM are also analogous. Compared to TopK graph, one principal distinction in the LM graph is that groups of local maximum activations tend to implicitly concentrate on the same point in the next layer. The white circles in Fig. 12 illustrate some concurrent points. Therefore, by inspecting the two attention graphs, the dynamic flow can be clearly followed. We can figure out how the object parts are co-activated and thus go through similar level of noise reduction.

In summary, the latent learning behavior of the CTformer can be visually interpreted statically and dynamically. This makes the proposed model more transparent and reliable for diagnostic decisions.

In this part, comparative experiments are conducted to study the impact of the T2TD block, the cyclic shift operation and the number of the intermediate transformer blocks.

Impact of T2TD block. T2TD blocks are used in the CTformer to enhance the feature integration in the tokenization stage. Compared to fixed-region tokenization, the tokens in T2TD blocks are extracted from various regions of the original images. To verify the effectiveness of this part, a Sole-ViT model without the T2TD module is designed. We only adopt a sole convolution in the tokenization stage with a filter size of $8$ and a stride of $8$. Then five layers of transformer with an embedding size of $256$ are applied for feature extraction and denoising. $256$ rather than $64$ embedding size is used because the model size and MACs are close to our model as shown in III. Finally, a detokenization with deconvolution is employed to transform the tokens back to desired image domain. By investigating the conjunction area inside the blue circle in Fig. 13(g), we can see that Sole-ViT brings in extra blotchy tissues. Meanwhile, Fig. 14 shows that the CTformer converges faster than Sole-ViT and has better scores with a margin of $0.0235$ on SSIM and $3.3362$ on RMSE.

Impact of cyclic shift. In this work, the cyclic shift is performed in the T2TD blocks to enhance the perceptual fields of our model. Fig. 13 shows that CTformer with cyclic shift enjoys more spatial smoothness compared to the CTformer without cyclic shift. The latter introduces some additional noise components. Quantitative results from Table III also confirm the effectiveness of cyclic shift in improving the SSIM and RMSE of the model by $0.0026$ and $0.1337$, respectively.

Impact of block number. In terms of the number of intermediate transformer blocks, we evaluate the CTformer with $1$, $2$, $4$, and $8$ blocks to identify the influence. When the block number grows, the network goes deeper. The computational cost increases slowly, but the actual training time climb up dramatically. However, Table III indicates that the CTformer with only one block yields the best performance over the ones with more blocks.

In this paper, we have proposed a novel convolution-free transformer empowered by dilated tokenization and cyclic shift for LDCT denoising, which is referred to as the CTformer. To the best of our knowledge, the proposed CTformer is the first pure transformer model for LDCT denoising. Also, we have developed the interpretation methods for the proposed CTformer to decode its hidden behavior. Moreover, we have proposed the overlapped inference to address the boundary artifacts that are common in an encoder-decoder model. Experimental results have demonstrated that the CTformer outperforms its competitors in terms of the denoising performance and model efficiency. In the future, more efforts can be made to translate the CTformer into other medical denoising problems.