Multi-view X-ray Image Synthesis with Multiple Domain Disentanglement from CT Scans

The synthesis of X-ray images from volume data aims to obtain realistic X-ray images with consistent structures and styles, thereby reducing patient exposure to X-ray radiation. Mainstream methods are reached by modelling X-ray imaging procedures to generate DRRs (Dhont et al., 2020). Early research primarily employed ray-casting for synthesizing X-ray images (Kügler et al., 2020). More sophisticated methods explored the Monte Carlo (MC) (Csébfalvi and Szirmay-Kalos, 2003; Andreu Badal and Badano, 2009) method to accurately simulate the interaction between X-ray particles and tissue organs. Li et al. (Li et al., 2006) devised an adaptive MC volume Rendering algorithm, partitioning the volume into multiple sub-domains for sampling, thus accelerating the MC simulation.

Recent advancements in deep learning have significantly promoted X-ray image synthesis performance. Methods such as DeepDRR (Mathias Unberath et al., 2018) utilize convolutional neural networks for tissue segmentation in CT scans, alongside ray-casting and MC methods to compute tissue absorption rates. Gopalakrishnan et al. (Vivek Gopalakrishnan and Polina Golland, 2022) accelerated the synthesis of DRRs by reformulating the ray-casting algorithm as a series of vectorized tensor operations, facilitating DRRs interoperable with gradient-based optimization and deep learning frameworks. It is noteworthy that existing methods heavily rely on the segmentation results from the CT scans. Fixed absorption coefficients are assigned to different structures to calculate X-ray beam attenuation rates. Recent advancements in X-ray projections synthesis have revealed that deep learning models trained on simulated DRRs struggle to generalize to actual X-ray images (Mathias Unberath et al., 2018; Ying et al., 2019). However, recent progress in image translation has demonstrated that utilizing condition features extracted from representation learning models can markedly reduce the disparity between synthesized DRRs and real X-ray images (Dhont et al., 2020). Inspired by this, our model diverges from traditional approaches to modelling of imaging procedures. Instead, it employs style-based generative models, offering a robust approach for disentangled synthesis, thus improving the performance and image quality of X-ray synthesis.

In recent years, generative adversarial networks (GANs) have achie-ved remarkable success in image synthesis (Zhang et al., 2018b), image translation (Isola et al., 2017), and image editing (Alaluf et al., 2021). Building upon progressive GAN (Karras et al., 2017), Karras et al. proposed StyleGAN (Karras et al., 2019), which enhanced the quality of the generated image and allowed the network to decouple different features, providing a more controllable synthesis strategy. StyleGAN2 (Karras et al., 2020) redesigned the instance normalization scheme to remove the water droplet-like artefacts existing in StyleGAN, leading to higher-quality outputs. Some methods have attempted to control generated images by exploring the latent space of GANs (Collins et al., 2020; Shen et al., 2020; Roich et al., 2022). Despite their remarkable process, most deviations of GANs still concentrate on data augmentation in medical imaging, with limited research conducted on synthesizing cross-dimension medical images (Iglesias et al., 2023). One of the primary challenges inhibiting GANs from the CT2X task is the synthesis of multi-view results, as GANs have limited awareness of pose information (Iglesias et al., 2023; Liao et al., 2024).

Recently, there has been a trend to incorporate 3D representations into GANs, enabling them to capture pose information in 3D space (Liao et al., 2024). Such approaches, known as 3D-aware GANs, facilitate multi-view image synthesis. Methods that introduced explicit 3D representations, such as tree-GAN (Zhang et al., 2020), MeshGAN (Cheng et al., 2019), and SDF-StyleGAN (Zheng et al., 2022), are capable of synthesizing the 3D structure of the target object, which allows them to explicitly define poses in 3D space. However, the use of explicit 3D representations has constrained their resolution. $\pi$-GAN (Chan et al., 2021) and EG3D (Chan et al., 2022) introduced radiance fields to endow networks with the capability of pose awareness. Nonetheless, the high computational cost of stochastic sampling required for radiance fields introduces training complexity and may lead to noise (Kerbl et al., 2023). In contrast, our approach aims to improve the capability of the network to perceive spatial poses by involving the projections at the target view angles.

The purpose of X-ray image synthesis is to predict the X-ray image at any arbitrary view angle from the 3D CT scan. Let us denote the input CT scan with $V\in\mathbb{R}^{H\times W\times D}$ , the X-ray image for styles with $I_{X}\in\mathbb{R}^{H\times W}$ , the referenced DRR for structures with $I_{DRR}\in\mathbb{R}^{H\times W}$ . $H$, $W$ and $D$ represent the height, width and depth, respectively. $p\in\mathbb{R}^{1\times 25}$ represents the camera pose of the target view angle. Hence, A mapping function from CT scan to X-ray images can be expressed as follows:

In this study, two main challenges in X-ray image synthesis will be tackled: (1) How to conduct end-to-end synthesis using unpaired CT and X-ray data? (2) How to realize X-ray image synthesis from multiple view angles? To address them, we train the network using images from three domains, including CT scans, X-ray images, and DRR images. In Section 2.3, we utilize the CT encoder to extract anatomical information from CT data, while the disentangled encoder processes X-ray images to extract X-ray style features. Our generator seamlessly integrates unpaired information for synthesis. Moreover, we leverage the style features extracted from DRRs to produce style reconstructed syntheses and hence provide supervision. The integration of information from three domains enables the tackling of end-to-end training with unpaired data. Additionally, a consistency regularization is employed in Section 2.4 to further constrain the training of the decoupling encoder, ensuring comprehensive extraction of domain-specific style information. A PAM is introduced to focus the content code on the target view angle, facilitating multi-view synthesis, as depicted in section 2.5. In Section 2.6, we employ a pose-aware adversarial training strategy by feeding the corresponding DRRs into the discriminator, further improving the quality of multi-view synthesis.

CT2X-GAN synthesizes multi-view X-ray images by incorporating images from three domains: CT scans $V$, X-ray images $I_{X}$, and DRRs $I_{DRR}$ as inputs during training, as depicted in Figure 2. The CT scan is employed to compute the CT content code $f_{c}^{CT}$ through a CT encoder $E_{CT}$, whereas the X-ray image is sent to a style decoupling encoder, denoted as $E_{sty}$, to extract the X-ray style code $f_{sty}^{X}$ and X-ray content code $f_{c}^{X}$. The CT content code and X-ray style code are then fed into distinct layers of the generator $G$ to conduct the X-ray image synthesis $\hat{I}_{X}$. The synthesized X-ray results lack corresponding ground truth (GT). To provide supervision to the network training, we utilize the DeepDRR (Mathias Unberath et al., 2018) framework to generate target view angle DRRs $I_{DRR}$ from CT scans, which also serve as the GT DRRs. These DRR images are then sent into the style decoupling encoder to obtain DRR style code $f_{sty}^{DRR}$ and DRR content code $f_{c}^{DRR}$. Forwarding the DRR style code and the CT content code into the generator produces a DRR stylized image $\hat{I}_{DRR}$. Reconstruction loss can then be computed between the reconstructed and the GT DRR for supervision. A consistency regularization is next introduced to constrain the training and improve the disentanglement by minimizing the discrepancy between the style code and content code between the real and synthesized images. Furthermore, to improve the perception of CT content code to the information of the target pose $p$, a PAM is utilized to modify the CT content code $f_{c}^{w^{+}}$ by the maximum intensity projection at the target pose.

To address the challenge of limited paired CT and X-ray data, we propose a style decoupling encoder $E_{sty}$ to separate images into content and style code. This allows the generator to control the style feature in the synthesized images. Specifically, for X-ray reference images $I_{X}$, the style decoupling encoder consists of two branches, including an X-ray style branch and a content branch. The two branches separate the X-ray image into style code $f_{sty}^{X}$ and content code $f_{c}^{X}$ as follows:

where $E_{sty}^{s_{X}}$ and $E_{sty}^{c}$ denote the X-ray style branch and content branch of the style decoupling encoder, respectively.

The X-ray style code $f_{sty}^{X}$ describes the low-level style and intensity information of the X-ray image, while the modified content code $f_{c}^{w^{+}}$ contains the high-level anatomical information. The generator $G$ synthesizes the X-ray images by combining the X-ray style code with the content code as follows:

To provide auxiliary supervision to the network training, we employ the DRRs $I_{DRR}$ computed from the input CT scan as the GT. The reconstructed images $\hat{I}_{DRR}$ synthesized by the style code from such DRRs and the content code from CT scans should be consistent with $I_{DRR}$. To realize this, we introduce a DRR style branch into the style decoupling encoder to be distinct from the X-ray style branch, thus enabling the style decoupling encoder. The style decoupling encoder is utilized to extract style code $f_{sty}^{DRR}$ and content code $f_{c}^{DRR}$ from DRRs as follows:

where $E_{sty}^{s_{DRR}}$ denotes the DRR style branch of the style decoupling encoder.

The reconstructed DRRs $\hat{I}_{DRR}$ can be obtained by feeding the DRR style code and the modified CT content code into the generator:

To improve the preservation of anatomical structures and improve the quality of image synthesis, we compute the supervised reconstruction loss between the reconstructed DRRs $\hat{I}_{DRR}$ and GT DRRs $I_{DRR}$ as follows:

Here, $\mathcal{L}_{mae}$ is the mean absolute error (mae) loss, $\mathcal{L}_{lpips}$ denotes the learned perceptual image patch similarity (lpips) (Zhang et al., 2018a). $\lambda_{mae}$ and $\lambda_{lpips}$ are the hyperparameters controlling the weight of the loss items.

In decoupling the style information, it is challenging and unstable to supervise and train the decoupling encoder due to the absence of paired X-ray images. Hence, to extract domain-specific style features, we propose a novel consistency regularization approach by combining disentanglement learning and consistency to constrain the decoupling encoder. To ensure that the synthesized images obtain style information only from the style code while preserving the original structure, the decoupling encoder should thoroughly disentangle style and content information from the style image as much as possible. This implies that style branches $E_{sty}^{s_{*}}$ should extract all essential style information, while structural information can only be extracted by content branch $E_{sty}^{c}$. We compel the network to extract consistent style codes from the generated image $\hat{I}_{*}$ and input style image $I_{*}$, simultaneously ensuring consistency of content codes $f_{c}^{\hat{X}}$ and $f_{c}^{\hat{DRR}}$. Therefore, we formulate the following consistency constraints:

where $\mathcal{L}_{cc}$ describes the content consistency and $\mathcal{L}_{sc}$ represents style consistency. $\lambda_{cc}$ and $\lambda_{sc}$ control the weight of the regularization term.

For the purpose of ensuring that the style encoders can thoroughly disentangle domain-specific style information, we incorporate zero loss (Benaim et al., 2019) as an auxiliary constraint during training. As discussed in Section 2.3, we have designed two domain-specific style branches within the decoupling encoder: one for capturing features of X-ray style and the other for capturing DRR style features. Ideally, the X-ray branch should extract appearance features specific to the real X-ray images, while the DRR branch should extract appearance features specific to the DRRs. Therefore, when a style reference image is inputted into the corresponding branch that does not pertain to the domain, the objective is to minimize the errors extracted by the style encoder.

To ensure that each branch extracts style-specific features effectively, the extracted style information should be zero when the input style image belongs to a different domain.

While CT scans contain rich spatial information in 3D space, encoding it into latent space can incur notable information loss. Fully exploiting the 3D inherent in CT scans can lead to a more accurate anatomical structure in the synthesis. Therefore, we introduce a PAM, which is capable of accentuating distribution information of a specific view angle in the CT code $f_{c}^{CT}$, thereby obtaining an modified content code $f_{c}^{w^{+}}$.

Given that the CT scan encompasses comprehensive anatomical information about the human body, the information within the generated X-ray image at a specific view angle should encompass the position, tissue size and morphology of anatomical structures that appeared in the CT scans. Thus, we can project the CT scans to the imaging plane of a target view angle and utilize the projection as auxiliary information to modify the content code through attention mechanisms. Initially, we encode the input CT scan $V$ into a CT code $f_{c}^{CT}$ using an encoder. Next, the camera parameters $p$ are fed into a multi-layer perceptron (MLP) and concatenated with $f_{c}^{CT}$ as conditional input. Subsequently, the conditional information is merged with the CT code using an MLP and denoted as $\mathbf{Q}$. We then rotate the input volume $V$ in 3D space to align with the target pose $p$ for maximum intensity projection. The resulting projection is encoded via an MLP and utilized as $\mathbf{K}$ and $\mathbf{V}$. Thus, the PAM can be represented as follows:

where $Concat$ refers to the concatenation, $Rot$ denotes the 3D rotation operation, $Proj$ represents the maximum intensity projection and $\tau$ is a small-valued constant preventing the extreme magnitude.

Overall, our proposed PAM enhances the generator capability of robust multi-view synthesis. By intensifying specific pose features, the PAM facilitates high-quality multi-view synthesis, empowering the 2D network with enhanced 3D perceptual capabilities without the need for 3D representations in the network.

To understand the role of each component in CT2X-GAN, we conduct a series of ablation studies. Table 2 presents the quantitative results under all configurations, while Figure 5 illustrates the visualization examples of the outcomes.

We run a user study by randomly selecting 20 instances from the synthesized images and inviting 47 users to score them. Specifically, the participants are required to rate the synthesized images produced by various methods on a scale of 1-5, where higher scores signify greater resemblance to actual X-ray images. Subsequently, we compute the average score as the X-ray image similarity measure. The results are depicted in Table 4. The table reveals that the images synthesized by $\pi$-GAN and EG3D are notably distorted, as users predominantly perceive them to deviate substantially from authentic X-ray data. While superior to the 3D-aware methods, the DRRs still fall short compared to our method, which further demonstrates that our approach yields results more closely resembling to real X-ray images.