U-DuDoNet: Unpaired dual-domain network for CT metal artifact reduction

Computed tomography (CT) reveals the underlying anatomical structure within the human body. However, when a metallic object is present, metal artifacts appear in the image because of beam hardening, scatters, photon starvation, etc. [2, 3, 19], degrading the image quality and limiting its diagnostic value.

With the success of deep learning in medical image processing [27, 28], deep learning has been used for metal artifact reduction (MAR). Single-domain networks [5, 21, 25] have been proposed to address MAR with success. Lin et al. are the first to introduce dual-domain network (DuDoNet) to reduce metal artifacts in the sinogram and image domain jointly and DuDoNet shows further advantages over single-domain networks and traditional approaches [4, 10, 12, 13, 19]. Following this work, variants of the dual-domain architecture [18, 22, 24] have been designed. However, all the above-mentioned networks are supervised and rely on paired clean and metal-affected images. Since such clinical data is hard to acquire, simulation data are widely used in practice. Thus, supervised models may over-fit to simulation and do not generalize well to real clinical data.

Learning from unpaired, real data is thus of interest. To this, Liao et al. propose ADN [14, 16], which separates content and artifact in the latent spaces with multiple encoders and decoders and induces unsupervised learning via various forms of image generation and specialized loss functions. An artifact consistency loss is introduced to retain anatomical preciseness during MAR. The loss is based on the assumption that metal artifacts are additive. Later on, Zhao et al. [26] design a simple reused convolutional network (RCN) of encoders and decoders to recurrently generating both artifact and non-artifact images. RCN also adopts the additive metal artifacts assumption. However, neither of the works has theoretically proved the property. Moreover, without the aid of processing in sinogram domain, both methods have limited effect on removing strong metal artifacts, such as the dark and bright bands around metal objects.

In this work, we analytically derive the additive property associated with metal artifacts and propose an unpaired dual-domain MAR network (U-DuDoNet). Without using complicated encoders and decoders, our network directly estimates the additive component of metal artifacts, jointly using two U-Nets on two domains: a sinogram-based estimation net (S-Net) and an image-based estimation net (I-Net). S-Net first restores sinogram data and I-net removes additional streaky artifacts. Unpaired learning is achieved with cyclic artifact reduction and synthesis processes. Strong metal artifacts can be reduced in the sinogram domain with prior knowledge. Specifically, sinogram enhancement is guided by a self-learned sinogram completion network (P-Net) with clean images. Both simulation and clinical data show our method outperforms competing unsupervised approaches and has better generalizability than supervised approaches.

Here, we prove metal artifacts are inherently additive up to mild assumptions. The CT image intensity represents the attenuation coefficient. Let $X^{c}(E)$ be a normal attenuation coefficient image at energy level $E$. In a polychromatic x-ray system, the ideal projection data (sinogram) $S^{c}$ can be expressed as,

where $\mathcal{P}$ and $\eta(E)$ denote forward projection (FP) operator and fractional energy at $E$. Comparing with metal, the attenuation coefficient of normal body tissue is almost constant with respect to $E$, thus we have $X^{c}=X^{c}(E)$ and $S^{c}=\mathcal{P}(X^{c})$. Without metal, filtered back projection (FBP) operator $\mathcal{P}^{*}$ provides a clean CT image $I^{c}$ as a good estimation of $X^{c}$, $I^{c}=\mathcal{P}^{*}(S^{c})=\mathcal{P}^{*}(\mathcal{P}(X^{c}))$.

Metal artifacts appear mainly because of beam hardening. An attenuation coefficient image with metal $X^{a}(E)$ can be split into a relatively constant image without metal $X^{ac}$ and a metal-only image $X^{m}(E)$ varies rapidly against $E$, $X^{a}(E)=X^{ac}+X^{m}(E)$. Often $X^{m}(E)$ is locally constrained. The contaminated sinogram $S^{a}$ can be given as,

And the reconstructed metal-affected CT image $I^{a}$ is,

Here, $\mathcal{P}^{*}(\mathcal{P}(X^{ac}))$ is the MAR image $I^{ac}$ and the second term introduces streaky and band artifacts, which is a function of $X^{m}(E)$. Since metal artifacts are caused only by $X^{m}(E)$, we can create a plausible artifact-affected CT $I^{ca}$ by adding the artifact term to an arbitrary, clean CT image: $I^{ca}=I^{c}+F(X^{m}(E))$.

Fig. 1b shows the proposed cyclical MAR framework. In Phase I, our framework first estimates artifact components $a_{S}$ and $a_{I}$ through U-DuDoNet from $I^{a}$, see Fig. 1a. In Phase II, based on the additive metal artifact property (Section 2), plausible clean image $I^{ac}$ and metal-affected image $I^{ca}$ could be generated, $I^{ac}=I^{a}-a_{S}-a_{I}$, $I^{ca}=I^{c}+a_{S}+a_{I}$. Then, the artifact components should be removable from $I^{ca}$ by U-DuDoNet, resulting in $a_{S}^{\prime}$ and $a_{I}^{\prime}$. In the end, reconstructed images $I^{aca}$, $I^{cac}$ can be obtained through subtracting or adding the artifact components, $I^{aca}=I^{ac}+a_{S}^{\prime}+a_{I}^{\prime}$ , $I^{cac}=I^{ca}-a_{S}^{\prime}-a_{I}^{\prime}$.

In this paper, we present an unpaired dual-domain network (U-DuDoNet) that exploits the additive property of artifact modeling for metal artifact reduction. In particular, we first remove the strong metal artifacts in sinogram domain and then suppress the streaks in image domain. Unsupervised learning is achieved via cyclic additive artifact modeling, i.e. we try to remove the same artifact after inducing artifact in an unpaired clean image. We also apply prior knowledge to guide data restoration. Qualitative evaluations and visual comparisons demonstrate that our model yields better MAR performance than competing methods. Moreover, our model shows great potential when applied to clinical images.