DuDoTrans: Dual-Domain Transformer Provides More
Attention for Sinogram Restoration in Sparse-View CT Reconstruction

Human body tissues, such as bones and organs, have different X-ray attenuation coefficients $\mu$. When considering a 2D CT image, the distribution of the attenuation coefficients $\mathbf{X}=\mu(a,b)$, where $(a,b)$ indicate positions, represents the underlying anatomical structure. The principle of CT imaging is based on the fundamental Fourier Slice Theorem, which guarantees that the 2D image function $\mathbf{X}$ can be reconstructed from the obtained dense projections (called sinograms). When imaging, projections of the anatomical structure $\mathbf{X}$ are indeed inferred by the emitted and received X-ray intensities according to the Lambert-Beer Law. Further, when under a polychromatic X-ray source with an energy distribution $\eta(E)$, the CT imaging process is given as:

$$\mathbf{Y}=-\log\int\eta(E)\exp\{-\mathcal{P\mathbf{X}}\},$$

(1)

where $\mathcal{P}$ represents the sinogram generation process, i.e., Radon transformation (commonly defined with a fan-beam imaging geometry). With the above forward process, CT imaging aims to reconstruct $\mathbf{X}$ from the obtained projections $\mathbf{Y}=\mathcal{P}\mathbf{X}$ (abbreviation for simplicity) with the estimated/learned function ${\mathcal{P}}^{\dagger}$. In practical SVCT, the projection data $\mathbf{Y}$ is incomplete, where the total $\alpha_{max}$ projection views are sampled uniformly in a circle around the patient. This reduced sinogram information heavily limits the performance of previous methods and results in artifacts. In order to alleviate the phenomena, many works have been recently proposed, which can be categorized into the following two groups: Iterative-based reconstruction methods [27, 23, 38, 2, 16] and Deep-learning-based reconstruction methods[13, 34, 14, 45, 46]. Different from these prevalent works, our method DuDoTrans is based on deep-learning, but is the only dual-domain method based on Transformer.

Based on the powerful attention mechanism [29, 4, 10, 36, 26, 41] and patch-based operations, Transformer is applied to many vision tasks [9, 28, 12, 22]. Especially, Swin Transformer [22] incorporates such an advantage with the local feature extraction ability of CNNs. With such an intuitive manner, Swin-Transformer-based models [19] have relieved the limitation of memory in previous Vision Transformer-based models. Based on these successes and for better modeling the global features of medical images, Transformer has been applied to medical image segmentation [6, 44, 3, 20], registration [39], classification [37, 31], and achieved surprising improvements. Nevertheless, few works explore Transformer structures in SVCT reconstruction. Although TransCT [40] attempts to suppress the noise artifacts in low-dose CT with Transformer, they neglect the consideration of global sinogram characteristics in their design, which is taken into account in DuDoTrans.

Sinogram restoration is extremely challenging since the intrinsic information not only contains spatial structures of human bodies, but follows the global sampling process. Specifically, each line $\{\mathbf{{Y}_{i}}\}_{i=1}^{{H}_{s}}$ of a sinogram $\mathbf{Y}$ are sequentially sampled with overlapping information of surrounding sinograms. In other words, 1-D components of sinograms heavily correlate with each other. The global characteristic makes it difficult to be captured with traditional CNNs, which are powerful in local feature extraction. For this reason, we equip this module with the Swin-Transformer structure, which enables it with long-range dependency modeling ability. As shown in Fig. 2, SRT consists of $m$ successive residual blocks, and each block contains $n$ normal Swin-Transformer Module (STM) and a spatial convolutional layer, which have the capacity of both global and local feature extraction. Given the degraded sinograms, we first use a convolutional layer to extract the spatial structure $\mathbf{F}_{conv}$. Considering it as $\mathbf{F}_{{STM}_{0}}$, then $n$ STM components of each residual block output $\{\mathbf{F}_{{STM}_{i}}\}_{i=1}^{m}$ with the following formulation:

$$\mathbf{F}_{{STM}_{i}}=\mathnormal{M}_{conv}(\prod_{j=1}^{n}\mathnormal{M}_{swin}^{j}(\mathbf{F}_{{STM}_{i-1}}))+\mathbf{F}_{{STM}_{i-1}},$$

(2)

where $\mathnormal{M}_{conv}$ denotes a convolutional layer, $\{\mathnormal{M}_{swin}^{j}\}_{j=1}^{n}$ denotes $n$ Swin-Transformer layers, and $\prod$ represents the successive operation of Swin-Transformer Layers. Finally, the enhanced sinograms are estimated with:

$$\tilde{\mathbf{Y}}=\mathbf{Y}+\mathnormal{M}_{conv}(\mathnormal{M}_{conv}(\mathbf{F}_{{STM}_{m}})+\mathbf{F}_{{STM}_{0}}).$$

(3)

As a restoration block, ${\mathcal{L}}_{SRT}$ is used to supervise the output of the SRT:

$$\mathcal{L}_{SRT}=\|\mathbf{\tilde{Y}}-\mathbf{Y}_{gt}\|_{2},$$

(4)

where $\mathbf{Y}_{gt}$ is the ground truth sinogram, and it should be given when training.

Although input sinograms have been enhanced via the SRT module, directly learning from the concatenation of $\tilde{\mathbf{{X}}}_{1}$ and $\tilde{\mathbf{{X}}}_{2}$ leaves a drift between the optimization directions of SRT and RIRM. To compensate for the drift, we make use of a differentiable DuDo Consistency Layer ${\mathnormal{M}}_{DC}$ to back-propagate the gradients of RIRM. In this way, the optimization direction imposes the preferred sinogram characteristics to $\mathbf{\tilde{Y}}$, and vice versa. To be specific, given the input fan-beam sinogram $\mathbf{\tilde{Y}}$, the DuDo Consistency Layer first converts it into parallel-beam geometry, followed with Filtered Backprojection:

$$\tilde{\mathbf{{X}}}_{2}={\mathnormal{M}}_{DC}(\mathbf{\tilde{Y}}).$$

(5)

To additionally keep the restored sinograms consistent with the ground-truth CT image $\mathbf{X}_{gt}$, ${\mathcal{L}}_{DC}$ is proposed as follows:

$$\mathcal{L}_{DC}=\|\tilde{\mathbf{{X}}}_{2}-\mathbf{X}_{gt}\|_{2}.$$

(6)

As a long-standing clinical problem, the final goal of CT image reconstruction is to recover a high-quality CT image for diagnosis. With the initially estimated low-quality images that help rectify the geometric deviation between the sinogram and image domains, we next employ Shallow Layer ${\mathnormal{M}}_{sl}$ to obtain shallow features of input low-quality image $\mathbf{\tilde{X}}$:

$$\mathbf{F}_{sl}={\mathnormal{M}}_{sl}([\tilde{\mathbf{{X}}}_{1},\tilde{\mathbf{{X}}}_{2}]).$$

(7)

Then a series of Deep Feature Extraction Layers ${\{\mathnormal{M}}_{df}^{i}\}_{i=1}^{n}$ are introduced to extract deep features:

$$\mathbf{F}_{df}^{i}=\mathnormal{M}_{df}^{i}(\mathbf{F}_{df}^{i-1}),~{}i=1,2,\ldots,n,$$

(8)

where $\mathbf{F}_{df}^{0}=\mathbf{F}_{sl}$. Finally, we utilize a Recon Layer ${\mathnormal{M}}_{re}$ to predict the clean CT image with residual learning:

$$\mathbf{\tilde{X}}={\mathnormal{M}}_{re}(\mathbf{F}_{df}^{n})+\tilde{\mathbf{{X}}}_{1}.$$

(9)

To supervise our network optimization, the below ${\mathcal{L}}_{RIRM}$ loss is used for this module:

$$\mathcal{L}_{RIRM}=\|\mathbf{\tilde{X}}-\mathbf{X}_{gt}\|_{2}.$$

(10)

The full objective of our model is:

$$\mathcal{L}=\mathcal{L}_{SRT}+{\lambda}_{1}\mathcal{L}_{DC}+{\lambda}_{2}\mathcal{L}_{RIRM},$$

(11)

where ${\lambda}_{1}$ and ${\lambda}_{2}$ are blending coefficients, which are both empirically set as 1 in experiments.

Note that the intermediate convolutional layers are used to communicate between image space $\mathcal{R}^{{H_{I}}\times{W_{I}}}$ and patch-based feature space $\mathcal{R}^{\frac{{H_{I}}}{w}\times\frac{{W_{I}}}{w}\times{w}^{2}}$. Further, by dynamically tuning the depth $m$ and width $n$, SRT modules are flexible in practice depending on the balance between memory and performance. We will explore this balancing issue in later experiments.

We next prove the effectiveness of our proposed SRT module and exhaust the best structure for DuDoTrans. Firstly, we conduct the experiments with the five models: (a) FBPConvNet [15], (b) DuDoNet [21], (c) FBPConvNet+SRT, which combines (a) with our proposed SRT, (d) ImgTrans, which replaces the image-domain model in (a) with Swin-Transformer [22], and (e) our DuDoTrans. The experimental settings are by default with ${\alpha}_{max}$ = 96, and the results are shown in Table 1.
The Effectiveness of SRT. Comparing models (a) and (c) in Table 1, the performance is improved 0.66 dB, which confirms that the SRT module output $\tilde{\mathbf{Y}}$ indeed provides useful information for the image-domain reconstruction.
The exploration of RIRM. Inspired by the success of Swin-Transformer in low-level vision tasks, we simply replace the post-precessing module of FBPConvNet with Swin-Transformer, named ImgTrans. Comparing it with the baseline model (a), the achieved 1 dB improvement confirms that Transformer is skilled at characterizing deep features of images, and a thorough exploration is worthy.
The effectiveness of DuDoTrans. When comparing (d) and (e), the boosted 0.18 dB proves SRT effectiveness again. Further, comparing (b) and (e), both dual-domain architectures with corresponding CNNs and Transformer, the improvement demonstrates that Transformer is very suitable in CT reconstruction.

We then investigate the impact of each sub-module on the performance of DuDoTrans:
RIRM depth and width. Similar to the SRT structure, the RIRM depth represents the number of sub-modules of RIRM, and the RIRM width describes the number of successive Swin-Transformers in each sub-module. Results in Fig 3 (a) and (b) show the corresponding effect of the RIRM width and RIRM depth on the reconstruction performance. When increasing the RIRM depth (with fixed RIRM width 2), the performance is improved quickly when RIRM depth is smaller than 4. Then the PSNR improvement slows down, while the introduced computational cost is increased. Then we fix RIRM depth equal to 3 (blue) and 4 (yellow) and increase RIRM width, and find that the performance is improved fast till RIRM width = 3. After balancing the computation cost and performance improvement, we set RIRM width and depth to 4 and 2, respectively, which is a small model with similar FLOPs to FBPConvNet.
The SRT size. With a similar procedure, we explore the most suitable architecture for the SRT module. As Fig 3 (c) shows, with fixed RIRM depth and width. the performance is not influenced when we enlarge the SRT size (depth $n$ and width $m$ as introduced in Section 3.1). Specifically, we test five paired models whose RIRM depth and width are set to {(2, 1), ( 3, 1), (3, 2), (4, 2), (5, 2)}, respectively. Then we increase ($m$,$n$) from (3, 1) to (4, 2), but the PSNR is even reduced sometimes. Therefore, we set SRT depth and with to (3, 1) as default in later experiments.

Further, we analyze the convergence, robustness, and effect of the training dataset scale.
Convergence. In Fig. 3 (d), we plot the convergence curve of FBPConvNet, ImgTrans, and DuDoTrans. Evidently, the introduction of Transformer Structure not only improves the final results, but also stabilizes the training process. Besides, our Dual-Domain design achieves consistently better results, compared with ImgTrans.
Robustness. In practice, the photon noise in the imaging process influences the reconstructed images, therefore the robustness to such noise is important for application usage. Here, we simulate such noise with mixed noise (Gaussian & Poisson noise). Specifically, we train models with default noise level and test them with varied Poisson noise levels (with fixed Gaussian noise), whose intensity correspond to [1${e}^{\text{5}}$, 5${e}^{\text{5}}$, 1${e}^{\text{6}}$, 5${e}^{\text{6}}$], and show the results in Fig. 3 (e). Evidently, our models achieve better performances except when the intensity is 1${e}^{\text{5}}$, which noise is extremely hard to be suppressed, while DuDoTrans is still better than CNN-based methods, which confirms its robustness.
Training dataset scale. Vision Transformers need large-scale data to exhibit performance, and thus limits its development in medical imaging. To investigate it, we train FBPConvNet, ImgTrans, and DuDoTrans with [20%, 40%, 60%, 80%, 100%] of our original training dataset, and show the performance in Fig. 3 (f). Obviously, reconstruction performance of DuDoTrans is very stable till the training dataset decreases to 20%, in which case training data is too less for all models to perform well and DuDoTrans still achieves the best performance.

We next conduct thorough experiments to test the performance of DuDoTrans on various sparse-view scenarios. Specifically, we first train models when ${\alpha}_{max}$ is 24, 72, 96, 144, respectively. The results are shown in Table 3, and DuDoTrans have achieved consistently better results. Besides, we observe that ImgTrans and DuDoTrans are more stable in training, and the learned parameters are both extremely small, compared with CNN-based models. Furthermore, the improvement of DuDoTrans over ImgTrans becomes larger when the ${\alpha}_{max}$ increases, which confirms the usefulness of restored sinograms in reconstruction.
Qualitative comparison. We also visualize the reconstructed images of these methods in Fig. 4 with ${\alpha}_{max}$ = [72, 96 ,144] (See more visualizations in Appendix). In all three rows, our DuDoTrans shows better detail recovery, and sparse-view artifacts are suppressed. Further, when decreasing ${\alpha}_{max}$, where raw sinograms are too messy to be restored and low-quality images from FBP are too hard to capture global features, Transformer-based models exhibit reduced performance. The phenomena suggests that we should design suitable structures with the Transformer and CNNs, facing with different cases.

As a practical problem, reconstruction speed is necessary when deployed in modern CT machines. Therefore, we compare the parameters and FLOPs versus performances in Fig 1 and Fig. 6, respectively. We find that Transformer-based methods have achieved better performances with fewer parameters, and our DuDoTrans exceeds ImgTrans with only a few additional parameters. As known, the patch-based operations and the attention mechanism are computationally expensive, which limits their application usage. Therefore, we further compare the FLOPs of these methods. As shown, light versions (DuDoTrans-L1, DuDoTrans-L2) have achieved 0.8-1 dB improvement with fewer FLOPs, and DuDoTrans-N with default size has enlarged the improvement to 1.2 dB. Besides, we report the inferring time in each Table 2 3 4, and computation time is very similar to CNN-based methods, whose additional consumption is because of the patch-based operations.

As in Table 1, we have shown the effectiveness of SRT with FBPConvNet [15] and ImgTrans [22], which are two post-processing methods. Recently deep-unrolling methods have attracted much attention in reconstruction. To concretely verify the SRT module’s effectiveness, we further combine it with PDNet [1], which is a deep-unrolling method. Results of involved three paired models (w/ v.s. w/o SRT) are shown in Table 5 with default experimental settings when ${\alpha}_{max}$ = 96 (See other cases when ${\alpha}_{max}$ = [24, 72, 144] in Appendix). All these three reconstruction methods have been improved by the use of the SRT module. Furthermore, our DuDoTrans still performs the best without any unrolling design. Thus, our SRT is flexible and powerful probably in any existing reconstruction framework.