UDCR: Unsupervised Aortic DSA/CTA Rigid Registration Using Deep Reinforcement Learning and Overlap Degree Calculation

In 2D/3D registration, the objective is to identify an optimal spatial transformation $\mathbf{T_{opt}}$ that adjusts the volume $\mathbf{V}$ based on observed X-ray projections $\mathbf{I}_{f}$, ensuring minimal misalignment between the overlaid images [4]. The mathematical formulation is:

where $\mathcal{F}$ is the objective function to measure the misalignment between two images. $\mathcal{S}(\mathbf{V},\mathbf{T})$ is a function representing the application of transformation $\mathbf{T}$ to volume $\mathbf{V}$.

In the aortic DSA/CTA rigid registration, $\mathbf{I}_{f}$ represents the intraoperative aortic DSA, while $\mathbf{V}$ corresponds to the aortic CTA acquired preoperatively. During interventional procedures, the operator manipulates the C-arm by translating and rotating it to capture the patient’s aortic DSA from the optimal viewing angle, facilitating the conduct of interventional surgical treatment. Consequently, the spatial transformation operation $\mathcal{S}$ includes 3D rotation and translation in three directions, as well as projection.

Before registration, it is necessary to preprocess the paired aortic DSA and CTA images. We begin by reading the pixel array from the DICOM files of the DSA. A Minimum Intensity Projection (MinIP) is applied by taking the minimum value of each pixel across the tamporal dimension in DSA. This process serves to preserve the contrast agent information to the greatest extent, thereby converting the DSA sequence into a single 2D DSA image that highlights the lumen of the vessels by depicting the areas with the least intensity, which usually correspond to the contrast-filled regions. This preprocessing step is crucial for enhancing the visibility of the vascular structures and facilitating the subsequent registration with the CTA images. Considering the calculation method of the registration reward function described in section 2.3, we convert the black edges in the MinIP to white. Moreover, the actual pixel spacing $S_{\text{DSA}}$ of the DSA images is calculated by multiplying the pixel spacing with an estimated radiographic magnification factor from the DSA.

To extract aorta from CTA images, we employ a hybrid architecture, PHTrans [6], under the nn-UNet [3] framework for the training of aortic segmentation on the AVT dataset [10]. The trained model is subsequently applied to segment the aorta in CTA images prepared for registration. Furthermore, the aortic segmentation volume is resampled using nearest neighbor interpolation to adjust its sizes. This step ensures that the spatial resolution of the segmented volume matches the actual pixel spacing of the DSA, laying the foundation for precise registration. The sizes of resampling are calculated based on $S_{\text{CTA}}\times D_{\text{CTA}}/S_{\text{DSA}}$, where $S_{\text{CTA}}$ and $D_{\text{CTA}}$ is the original pixel spacing and dimensions of the CTA. Additionally, rotation angle information extracted from the DSA is employed to initialize the rotational transformation of CTA before registration. This preparatory step is essential for setting up an accurate starting point for the rotational alignment of the DSA and CTA, facilitating a more efficient and precise registration process.

Using DRL for DSA/CTA registration involves training a DRL agent to find the optimal spatial alignment between 2D DSA and 3D CTA. The agent iteratively adjusts the spatial transformation of the CTA to maximize alignment with the DSA image, guided by a reward function that quantifies the quality of the registration. This process leverages state representations that include transformation parameters and potentially image-derived features, exploring a defined action space of rotations and translations. The objective is to accurately overlay vascular structures from DSA onto the detailed anatomical context provided by CTA, with the learning algorithm refining the agent’s policy based on feedback to converge on the most effective registration policy.

The data for the registration of aortic DSA/CTA originated from *** Hospital, comprising endovascular interventional DSA images for aortic aneurysms and aortic dissection, as well as preoperative and postoperative CTA examinations, totaling 61 paired cases. We developed a manual registration interactive program, enabling vascular surgeons to perform manual registration on all data preprocessed as described in section 2.2. The spatial transformation parameters obtained through this process serve as the ground truth for assessing the accuracy of the registration method. We select 40 pairs of data as the training set and the remaining 21 pairs as the test set.

We performed aortic DSA/CTA registration experiments utilizing Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC), which represent state-of-the-art on-policy and off-policy reinforcement learning algorithms, respectively. The experiments were conducted using PyTorch on a single GeForce RTX A100 GPU, and for the implementation of the SAC and PPO, we employed stable baselines [11]. In the experiments, the learning rate was set to 3e-4, the batch size to 64, and the total number of timesteps to 10000. The maximum translation and rotation angle for the agent at each step are 5 pixels and 0.5^∘, respectively. We employed two types of policy networks: one is the CnnPolicy from the stable baselines library, and the other (PCM) combines a pre-trained CNN with a Multilayer Perceptron (MLP). During the training phase, the pre-trained CNN component is frozen, allowing only the MLP part to be updated. This approach leverages the powerful representational capabilities of CNN for visual tasks while focusing on the optimization of policy decision-making through the MLP.

The metric for evaluating registration accuracy included reward values and the Mean Absolute Error (MAE) between the spatial transformation parameters relative to the starting point and the manually set ground truth. Considering the advantage of not relying on ground truth, the UDCR registration method exhibits considerable flexibility. We can employ UDCR for pre-training on the training dataset, where the refined policy is utilized for real-time registration. Moreover, UDCR can be directly employed for online learning. We conducted experiments on these two modes, configuring them through different approaches including observation, DRL algorithms, and policy networks. The spatial transformation parameters that yield the maximum reward are selected for evaluation.

As illustrated in Table 1, the strong negative correlation between the reward and the MAE of rotation and translation proves the efficacy of the overlap calculation reward function. Compared to the pre-training mode, the registration performance of online learning is superior. Online learning allows the model to continually refine its policy and update the highest reward. The registration result is determined by selecting the spatial transformation parameters that achieve the highest reward value across all iterations. However, online learning requires a significant amount of time. Table 2 presents the hyperparameter experiments on timesteps for OL+SAC+Fuse+PCM. Optimal performance was observed when timesteps were set at 10,000, yet the duration for online learning extends to 378 seconds. In comparison, loading a pre-trained model for direct registration takes only 10 seconds.

Leveraging the encouragement of exploration through maximum entropy, The registration performance of SAC in online learning surpasses that of PPO. Meanwhile, PPO demonstrates more robust generalization capabilities, consistently outperforming SAC across various metrics in pre-training model. It is noteworthy that within the same mode and using the same RL algorithm, the configuration of Fuse+PCM achieve the optimal results. We have visualized the results of the PPO experiments, as shown in Fig. 2. The first two rows correspond to cases of aortic dissection, while the last row pertains to a case of aortic aneurysm. The final column showcases the registration results under the online learning mode of PPO+Fuse+PCM, where the red denotes the binary vascular map obtained through spatial transformation and parallel projection of the 3D CTA segmentation volume, essentially achieving comprehensive overlay of the DSA angiography area. The MAE for rotation and translation is relatively small. In comparison, other methods exhibit suboptimal registration performance in certain cases, particularly with larger absolute errors in $T_{y}$.

Despite the presence of artifacts and incomplete visualization of the aorta in DSA, the experimental results demonstrate that UDCR achieves accurate aortic DSA/CTA registration. In this work, we did not develop a new RL algorithm. Instead, we constructed a cross-dimensional registration environment based on the imaging principles and characteristics of DSA and CTA. It is noteworthy that this method is independent of ground truth or synthetic data and does not necessitate segmentation or centerline extraction of DSA. Only the segmentation of CTA requires additional annotated data. Furthermore, the advantage of UDCR based on preoperative 3D image segmentation is that it is not limited to data modalities, 3D rotational DSA and magnetic resonance angiography, along with other 3D imaging modalities, can also be registered with 2D DSA using UDCR. Furthermore, the results of the experiments is based on calculations performed with the current computing equipment and does not incorporate additional data acceleration or parallel processing techniques. Therefore, there is significant potential for improvement to reduce the algorithm’s runtime.