TSI-Net: A Timing Sequence Image Segmentation Network for Intracranial Artery Segmentation in Digital Subtraction Angiography

The TSI-Net architecture consists of an encoder-decoder network based on the U-shape design. The encoder of TSI-Net receives preprocessed IA DSA sequences as input. In contrast to the original U-Net architecture, the DSA sequences are fed to the network frame by frame. The bi-directional ConvGRU module (BCM) is utilized to incorporate temporal information features, where each group of Down-Conv shares one weights. The encoder consists of four IA down-sampling layers, and each frame of one sequence undergoes two convolution in the same layer and then connect into BCM block frame by frame. Following the bi-directional ConvGRU module (BCM), the output sequence undergoes IA down-sampling through maximum pooling with a step size of 2, and the output of the final ConvGRU cell is connected to the decoder using a skip connection.

Each IA down-sampling doubles the number of channels in the feature map, which increases the receptive field of the deep features and ensures their global nature to some extent. The decoder is symmetric to the encoder and consists of 4 up-sampling layers. Connected the up-sampling feature maps with the same resolution feature maps from encoder can reduce the loss of edge features because of up-sampling. The features of the encoder have higher resolution, while they extracted by down-sampling, this information is not retrievable at the time of up-sampling. Finally, a 1x1 fully connected layer is used to classify the background and blood vessels in the image to output a segmented image of IA. The number of channels start from 32. An important improvement over U-Net is the use of 2D+T DSA sequences that contain temporal information instead of a single 2D image as an input.

It is essential to depict the sequential evolution of blood flow contrast information in the temporal domain, Since a DSA sequence presents a series of IA information in chronological order. Our proposal involves utilizing the BCM to simulate the temporal progression of blood flow appearance in the sequence. ConvGRU[10] is an extension of the traditional fully connected GRU, with convolutional structures in both the input-to-state and state-to-state connections. Define the given sequence with t-frames as S = ($f_{1}$,$f_{2}$,…,$f_{t}$), and $h_{t}$ represents the hidden state at time t. A ConvGRU cell consists of a reset gate and a update gate are expressed as $R_{t}$ and $U_{t}$. Through these gates, ConvGRU cells can not only achieve selective memory but also forgetting. Based on the above definition, the overall update process at the ConvGRU cell for different time steps in a DSA sequence is as follows:

where ‘*’ denotes the convolutional operation, ‘$\circ$’ represents the hadamard product, $\sigma(.)$ denotes sigmoid function, $\omega$, $\omega^{\prime}$ means weights that can be learned. $\tilde{h}$ is a candidate activation which is computed in a manner similart to that of a traditional recurrent unit in an Recurrent Neural Network. The formula simplifies the expression and ignores the bias term.

We input the DSA sequences frame-by-frame into encoder for feature extraction. Inspired by[21], we stack two sets of ConvGRU to compose BCM in forward and backward propagation, respectively, to enhance the exchange of spatio-temporal information between them. In this way, the BGM not only remembers past sequences but also predicts future sequences. This can be expressed as:

where ${h}_{t}^{f}$ and ${h}_{t}^{b}$ represent the hidden state from forward and backward ConvGRU cells, respectively. $l{h}_{t}$ represents the final output of BCM.

It should be noted that all blocks in the BCM share the same weights. The last layer of each BCM aggregates the features of all frames in the DSA sequence. This operation is designed to allow the network to learn the relationships between frames rather than treating them as individual frames for processing.

Data imbalance is a common problem in medical image segmentation. The region of inerest in a medical image is usually much less than entire image. Moreover the proportion of voxels in the regions to be segmented is smaller than the rest of the image. If the data is unbalanced, the learning process may converge to a suboptimal local minimum, and the final prediction results could be biased towards non-relevant features. This data imbalance issue is particularly severe in vessel segmentation.

Consequently, we used sensitive detail branch (SDB) for adjunctive monitoring of the thin vessels. In this branch, we use the skeletonization operation to perform centerline extraction of the original ground truth (GT) as a new label. In this module for assisted supervision, we calculate the mean square error values between the actual and predicted values one by one. The aim is to keep reducing the gap between the two, making the network more sensitive to fine vessels and some details, and maintaining a higher connectivity between the vessels.It can be defined as:

where $pre_{i}\in[0,1]$ represent predicted probability of network output, $gt_{i}\in[0,1]$ denotes GT, $\phi(gt_{i})$ is the centerline lifting method using the skeletonization operation.

On the one hand, the most often used loss function is the cross-entropy loss, which examines each pixel individually and compares the class prediction vectors to the GT. It can be defined as:

where N is the total pixel number.

On the other hand, the dice coefficient is a measure of the overlap area of the picture difference region and is used to compare the predicted map to the GT. It is a superior metric for minor goals. The dice loss ($\mathcal{L}_{dice}$) can be defined as:

The final loss function is defined as:

where $\lambda_{1}$ and $\lambda_{2}$ denotes the weight of $\mathcal{L}_{dice}$ and $\mathcal{L}_{SDB}$ respectively.

Auxiliary supervised mean square error loss improves the sensitivity of the model. Sensitivity measures how many true positives the model correctly detects among all actual positives. As a result, it allows for better segmentation of the end vessels and fine vessels of the vascular tree, improving segmentation accuracy and vascular connectivity.

To validate the performance of our proposed TSI-Net on the IA segmentation task, we conducted experiments on the DIAS dataset[7].

The DIAS dataset contains 60 one-on-one pixel-level labeled sequences. In the same way as the data are divided in [7], 30 as training samples, 10 as validation samples, and 20 as test samples. The DSA sequences have dimensions of $800\times 800\times 8$. While $800\times 800$ is the resolution of a single frame of the image in the sequence, and 8 is the number of frames in a single sequence.

We compare the predicted segmentation results with the corresponding ground truth to calculate the dice similarity coefficient (Dice) and evaluate the accuracy (Acc), sensitivity (Sen), specificity (Spe), and intersection over union (IOU) of the binary segmentation maps obtained using thresholding. Their definitions are as follows:

where true positives (TP) and true negatives (TN) denote the numbers of correctly segmented vascular pixels and nonvascular pixels, respectively; false positives (FP) and false negatives (FN) denote the numbers of incorrectly segmented vascular pixels and nonvascular pixels, respectively.

Additionally, we introduced the vascular connectivity (VC) metric[7]. This metric calculates the ratio of connected components in the segmentation image to those in the ground truth, providing a measure of vessel connectivity.

Firstly, all DSA sequences were z-score normalized. We randomly crop the input sequences into patches of size $8\times 64\times 64$. We randomly crop the input image into patches of size $8\times 64\times 64$. Randomized horizontal, vertical and 90-degree flips were used for data enhancement. The preprocessed sequences are used for training with an epoch size of 64. We optimize the model parameters using AdamW, with an initial learning rate of 5e-4. The learning rate is gradually reduced using the cosine annealing algorithm for more than 100 iterations. All experiments are conducted using PyTorch with constant hyperparameters and on a single GeForce RTX 3090 GPU.

We conducted IA segmentation experiments on the most popular networks, including U-Net[11], Res-UNet[22], UNet++[23] and CS²Net[24]. To address the limitation of network processing only individual 2D images, a dimension reduction module (DRM)[7] is incorporated to handle the DSA sequence. Specifically, given a DSA sequence $s\in\mathbb{R}^{C\times F\times H\times W}$, where $C$, $F$, $H$ and $W$ denote the number of channels, frame, height, and width, respectively. After DRM compression, the initial channel number is changed to 1. The compressed input is then fed into the 2D model, where property F replaces C as the input channel:

where, $Squ$ is dimensionality squeeze operation and $\mathcal{M}$ is a 2D model.

Fig. 3 shows the segmentation results. We perform minimizing operations on the entire sequence for each pixel along the sequence dimension to obtain minimum intensity projection (MIP) images. It can be seen that all models have comparable segmentation ability for large blood vessels, while for small blood vessels, there is a significant reduction of red pixels in the segmentation results of TSI-Net, which indicates that our network is more powerful for fine vessel segmentation.

The qualitative segmentation results for the DIAS dataset are presented in Table I. Overall, TSI-Net demonstrates the best performance on this dataset, exhibiting an approximately 2% improvement in Dice coefficient, which is outperforming to state-of-the-art methods.

Notably, TSI-Net exhibited a noticeable 3% improvement in the Sen score and a significant reduction in the VC compared to other methods, while only experiencing a marginal 0.1% decrease in Auc. These enhancements imply an improved ability of the model to accurately segment fine vessels and ensure improved vessel connectivity with fewer disruptions. Precisely capturing vessel distribution in interventional procedures serves as a crucial benchmark, and both metrics hold particular significance in scenarios characterized by highly imbalanced background and target distributions.

However, in contrast to the other composite evaluation metrics, the Spe is relatively low. Sen and Spe represent the proportions of correctly predicted positive and negative pixels, respectively. During the extraction of weakly vascularized pixels, TSI-Net introduces some false positives, indicating incorrect predictions of non-vascular pixels. We believe that the model further improves its ability to extract weak blood vessels while naturally increasing the acceptance of false positives.

To validate the effectiveness of the components, we conducted ablation experiments and evaluated the impact of each component on the results on the DIAS dataset. Using UNet as the baseline for the experiments, We incrementally incorporated Uni-directional ConvGRU Module (UCM) ,BCM and SDM to the baseline of the ablation study. All experiments were conducted using the same hyperparameter configuration. In our ablation study, all competitors were run on the same computing environment with the same data augmentation to ensure a fair comparison. Again since UNet cannot handle DSA sequences, the initial number of channels is set to 1 and compressed by DRM. The compressed input is then fed into the 2D model where ”F” replaces ”C” as the input channel. The results are shown in the Table II.

The inclusion of ConvGRU resulted in significant improvements in all model scores compared to the baseline, with bi-directional connectivity demonstrating superior performance over uni-directional connectivity across multiple metrics. When compared to baseline + BCM, the addition of SDB further enhanced the Sen score by 2.4%, achieving a maximum score of 0.7971. Auc and Spe exhibited only a marginal decrease of approximately 0.4%. Considering the degree of imbalance between IA vascular and background and practical medical application scenarios, the noted improvement in Sen and VC indicates that the model facilitates enhanced preservation of fine vascular information and vascular topology, ultimately enhancing the performance of sequence segmentation.

The ablation results were further visualized, as depicted in Fig 4, which includes an IA sequence featuring numerous fine vessels. Consequently, we zoomed in on specific image details to facilitate clearer visualization, focusing on the highlighted rectangular area. Within the zoomed subfigure, the thin vessels in the original image exhibit low contrast, appearing blurred and challenging to discern by the human eye. The expert has labeled the blood vessels in the GT, and based on the visualization of the segmentation results, it can be observed that Baseline, Baseline+UCM, and Baseline+BCM show incomplete extraction of the thin blood vessels in this area. However, the inclusion of Baseline+BCM+SDB resolves this issue by introducing fewer false-negative pixels, thereby visibly enhancing vascular connectivity.