\jyear

2021

[1,2]\fnmYun \surGu

1]\orgdivInstitute of Medical Robotics, \orgnameShanghai Jiao Tong University, \orgaddress\cityShanghai, \countryChina

2]\orgdivInstitute of Image Processing and Pattern Recognition, \orgnameShanghai Jiao Tong University, \orgaddress\cityShanghai, \countryChina

LTSP: Long-Term Slice Propagation for Accurate Airway Segmentation

\fnmYangqian \surWu [email protected] \fnmMinghui \surZhang [email protected] \fnmWeihao \surYu [email protected] \fnmHao \surZheng [email protected] \fnmJiasheng \surXu [email protected] [email protected] [ [

Abstract

Purpose: Bronchoscopic intervention is a widely-used clinical technique for pulmonary diseases, which requires an accurate and topological complete airway map for its localization and guidance. The airway map could be extracted from chest computed tomography (CT) scans automatically by airway segmentation methods. Due to the complex tree-like structure of the airway, preserving its topology completeness while maintaining the segmentation accuracy is a challenging task.

Methods: In this paper, a long-term slice propagation (LTSP) method is proposed for accurate airway segmentation from pathological CT scans. We also design a two-stage end-to-end segmentation framework utilizing the LTSP method in the decoding process. Stage 1 is used to generate a coarse feature map by an encoder-decoder architecture. Stage 2 is to adopt the proposed LTSP method for exploiting the continuity information and enhancing the weak airway features in the coarse feature map. The final segmentation result is predicted from the refined feature map.

Results: Extensive experiments were conducted to evaluate the performance of the proposed method on 70 clinical CT scans. The results demonstrate the considerable improvements of the proposed method compared to some state-of-the-art methods as most breakages are eliminated and more tiny bronchi are detected. The ablation studies further confirm the effectiveness of the constituents of the proposed method.

Conclusion: Slice continuity information is beneficial to accurate airway segmentation. Furthermore, by propagating the long-term slice feature, the airway topology connectivity is preserved with overall segmentation accuracy maintained.

keywords:

Airway segmentation, Slice propagation, Long-term feature

1 Introduction

Bronchoscopic intervention higgins2015multimodal ; mehta2018evolutional ; shen2019context is widely used in clinical practice since it could achieve minimally invasive access techniques for pulmonary diseases. To improve the efficacy of the bronchoscopic intervention, an accurate 3D airway map extracted from chest computed tomography (CT) is required, which is essential to the endoscopic tips localization and intraoperative guidance. However, due to the complex tree-like structure, manual segmentation of the airway from chest CT scans is time-consuming and requires expert knowledge. Therefore, to relieve the burden of clinicians, automatic airway segmentation methods are proposed to extract airways accurately.

Traditional airway segmentation methods are always designed based on region growing fabijanska2009two ; chen2018automatic and morphology operation aykac2003segmentation , which are very sensitive to the manually designed features. Thus, their performance will be degraded once noises appear. Besides, owing to the similarity between bronchi and surrounding human tissues, these methods will cause severe leakages and breakages. As shown in Fig. 1, in bronchoscopic intervention, leakages in the airway will cause misleading destinations while breakages will lead to interrupted trajectories. Recently, convolutional neural networks(CNNs)-based methods juarez2018automatic ; qin2020airwaynet ; qin2020learning ; zhang2021fda are increasingly developed for airway extraction. U-Net ronneberger2015u or 3D U-Net cciccek20163d is widely used as the backbone to acquire a coarse airway segmentation result, which is then refined by utilizing prior knowledge like neighborhood connectivity qin2019airwaynet or global information juarez2019joint .

Refer to caption — Figure 1: The impact of leakages and breakages in the airway for bronchoscopic intervention. (a) describes the principle of bronchoscopic intervention referred to shen2019context . (b) demonstrates that airway leakages will provide an incorrect destination and mislead the trajectory. (c) illustrates that airway breakages will interrupt the planning trajectory.

However, due to the complex tree-like structure of the airway, acquiring accurate and fine-grained segmentation results is difficult. The CNNs-based airway segmentation methods remain the following challenges. First, the intensity distribution is different in the main trachea region and bronchi region. In the main trachea region, the intensity contrast between the airway lumen and the wall is distinct and the features are easy to learn, while the ambiguity of intensity contrast in the bronchi region is hard to identify for the CNN models. Second, the encoder-decoder architecture is widely used in the CNN models like U-Net, where several pooling operations are utilized. Since some bronchi only have a diameter of 2-5 voxels, these features will be vanished by pooling operations and are hard to reconstruct in the decoder. Furthermore, these two challenges will result in weak airway features in the decoding process. Since adjacent slices in chest CT scans have similar airway shape and lumen position, one intuition for weak feature enhancement is to use this airway continuity prior knowledge. The prior knowledge could be achieved by applying slice feature propagation, so as to strengthen the weak feature and impose the continuity constraints.

To address the above problems and preserve the topological connectivity of the airway, we propose a two-stage end-to-end framework for accurate airway segmentation using long-term slice propagation (LTSP). In the proposed method, the slice relationship is considered in the decoding process. The slice feature is transferred in the proper direction to fully recover the continuity information destroyed in the encoder. Although slice propagation could alleviate the breakage phenomenon by enhancing the airway features, there remains a problem that slice features are hard to transfer among too long distances. In our method, to solve the above gradient vanish problem, we further design the LTSP cell in the decoding process where continuity information is extracted and densely propagates to other slices. Furthermore, we compare the proposed airway segmentation method to some state-of-the-art methods in 70 clinical CT scans. Extensive experiments show that our method achieves superior performance in extracting topological complete airway while maintaining the competitive overall segmentation accuracy.

2 Methodology

The overview of the proposed airway segmentation method is illustrated in Fig. 2, which is a two-stage end-to-end framework. The first stage is used to acquire a coarse feature map by utilizing an encoder-decoder architecture. The second stage is to predict a refined and well-connected airway segmentation result by: (1) propagating slice information of the coarse feature map using the long-term slice propagation (LTSP) method, (2) decoding the airway segmentation result from the refined feature map with skip connection.

2.1 Stage 1: Coarse feature map generation using encoder-decoder architecture

Stage 1 in the framework is to provide a coarse feature map by using an encoder and a coarse decoder. The feature map is extracted for subsequent slice propagation and airway refinement.

In this stage, we employ an encoder-decoder architecture to extract the airway feature and then recover from the deep information. A part of 3D U-Net is used as the backbone, which contains three down-samplings and two up-samplings with skip connections. In each down-sampling module, two convolution layers (Conv) with batch normalization (BN) and rectified linear unit (ReLU) are followed by a max-pooling layer. In each up-sampling module, the up-sampled feature map and corresponding feature map in the encoder are concatenated together and then passed to a convolution layer with BN and ReLU.

Given a 3D CT cube input $X_{0}$ with size of $1\times S_{0}\times H_{0}\times W_{0}$ , a coarse feature map $X_{coarse}$ with size of $C\times S\times H\times W$ is generated by stage 1 as

X_{coarse}=F_{1}(X_{0}),

(1)

where $F_{1}(\cdot)$ denotes the feature extraction process in stage 1.

2.2 Stage 2: Segmentation refinement using long-term slice propagation

Stage 2 in the framework is used to densely propagate the slice features in the coarse feature map extracted in stage 1. The proposed LTSP method is designed to transfer the long-term slice features for the refined feature map generation. The final airway segmentation result is then predicted from the refined feature map by a fine decoder.

2.2.1 Long-term slice propagation

Taking the airway’s continuity prior knowledge into consideration, we utilize the slice propagation method into the decoder, which aims to enhance the weak airway features in each slice by integrating the adjacent slice features. Additionally, we improve the previous spatial CNN pan2018spatial method and propose a long-term slice propagation (LTSP) method for effective feature transferring.

The original method proposed in pan2018spatial uses the spatial CNN to propagate the neighboring message in 2D images. However, as it only transfers the feature from a slice to its nearest neighbor, the propagating airway feature will be rapidly dismissed and the continuity information will be lost. To improve the propagation efficiency and expand the receptive field of slice propagation, we propose the long-term slice propagation (LTSP) method.

Fig. 3 gives the comparison of spatial CNN method and LTSP method. As illustrated in Fig. 3 (a), in spatial CNN, one slice only receives the feature from its upper slice by directly utilizing convolution operation. For the complicated tree-like airway structure, there are several limitations in spatial CNN: (1) Insufficient feature propagation. Some breakages in the airway are dependent on the feature far from it, which needs to use features in several slices to recover it. As one slice only can receive the feature from its nearest neighbor, the information is not sufficient for its recovery. (2) Short propagation distance. Spatial CNN only uses convolution to extract the information from a neighbor slice, leading to the feature disappearance when the propagation distance is too long. Besides, a short propagation distance will result in a narrow receptive field and great propagation deficiency.

To address the problems, we propose the LTSP method to develop the effectiveness of slice propagation. As illustrated in Fig. 3 (b), the LTSP method consists of two parts: information extraction and feature propagation.

In the information extraction part, the LTSP cell inspired by the long short-term memory (LSTM) hochreiter1997long is designed to extract airway continuity information. Fig. 3 (c) gives the propagating mechanism of the designed LTSP cell. Given a coarse feature map $X_{coarse}$ with the size of $C\times S\times H\times W$ , each split slice $X_{i}$ with the size of $C\times H\times W$ is acquired by

\displaystyle X_{i}=X_{coarse}[i].

(2)

Together with the previous LTSP cell output $H_{i-1}$ and previous cell state $C_{i-1}$ (with each size of $C\times H\times W$ ), the forward calculation includes the following three steps. First, the stacked gates $G$ is acquired by using 2D convolution operation to double the channel number of the concatenation of $X_{i}$ and $H_{i-1}$ , where $G$ is computed by

\displaystyle G=Conv(Cat(X_{i},H_{i-1})).

(3)

Then, the convolution result $G$ is chunked into $G_{1}$ , $G_{2}$ , $G_{3}$ and $G_{4}$ , with each size of $C\times H\times W$ . Pass them into different activation function to calculate forget gate $G_{f}$ , new cell state $\tilde{C_{i}}$ , input gate $G_{i}$ , and output gate $G_{o}$ , respectively. The calculation process could be described as

		$\displaystyle G_{f}=\sigma(G_{1}),\qquad\quad\tilde{C_{i}}=\tanh(G_{2}),$		(4)
		$\displaystyle G_{i}=\sigma(G_{3}),\qquad\quad\ G_{o}=\sigma(G_{4}).$		(4)

Finally, previous cell state $C_{i-1}$ and new cell state $\tilde{C_{i}}$ are used to calculate the updated output $H_{i}$ and final cell state $C_{i}$ by

		$\displaystyle C_{i}=G_{f}\circ C_{i-1}+G_{i}\circ\tilde{C_{i}},$		(5)
		$\displaystyle H_{i}=G_{o}\circ tanh(C_{i}).$		(5)

In the feature propagation part, the original slice $X_{i}$ is updated by propagating information maps extracted by LTSP cells. The forward slice propagation could be described as

X^{{}^{\prime}}_{i}=\begin{cases}X_{i},\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\quad\;i=1,\\ X_{i}+f\big{(}{H_{i-1}*W_{1}}),\qquad\qquad\qquad\qquad\qquad i=2,\\ X_{i}+f\big{(}H_{i-1}*W_{1}+H_{i-2}*W_{2}),\qquad\qquad\ {i=3,...,S},\end{cases}

(6)

where $W_{1}$ and $W_{2}$ denote the convolution weights for 1-distance feature and 2-distance feature, respectively. And $f(\cdot)$ denotes the nonlinear activation function like ReLU.

In the end, all updated slices $X^{{}^{\prime}}_{i}$ are stacked into a new refined feature map $X_{fine}$ with the size of $C\times S\times H\times W$ by

X_{fine}=[X_{1}^{{}^{\prime}},X_{2}^{{}^{\prime}},...,X_{S}^{{}^{\prime}}].

(7)

2.2.2 Segmentation prediction and optimization

To acquire the airway segmentation from refined feature map $X_{fine}$ , a fine decoder with skip connection is used to get the probability map of the airway $X_{prob}$ . An argmax operation is then utilized to predict the final airway segmentation result $X_{seg}$ . The prediction process could be described as

		$\displaystyle X_{prob}=F_{2}(X_{fine}),$		(8)
		$\displaystyle X_{seg}=argmax(X_{prob}),$		(8)

where $F_{2}(\cdot)$ denotes the up-sample operation in fine decoder.

In the training process, soft dice loss milletari2016v is used for the airway segmentation tasks. Given prediction $p(x)$ and corresponding binary label $y(x)$ for each voxel $x$ in segmentation result volume $X$ , the segmentation loss could be calculated by

\displaystyle\mathcal{L}_{seg}=1-\frac{2\sum_{x\in X}p(x)y(x)+\epsilon}{\sum_{x\in X}(p(x)+y(x))+\epsilon},

(9)

where smoothing parameter $\epsilon$ is used to avoid division by zero.

3 Experiments and results

We evaluate the proposed method in LIDC dataset qin2019airwaynet , where 50 chest CT scans are randomly chosen for training and the remaining 20 scans are used for testing. Furthermore, ablation studies are conducted to confirm the effectiveness of our method.

3.1 Datasets and implementation details

The experiment dataset contains 70 clinical chest CT scans, where the pixel spatial resolution ranges from 0.5 to 0.781mm and slice thickness varies from 0.45 to 1.0mm. The model is trained on 50 randomly chosen scans and tested on the remaining 20 scans.

To improve the model’s generalization ability, the HU value of each scan is truncated into [-1000, 600] and linearly mapped into [0, 1]. Besides, data augmentation is performed to each cropped cube including horizontal flipping and slight rotation. For training strategy, we adopt Adam optimizer kingma2014adam ( $\beta_{1}=0.9$ , $\beta_{2}=0.999$ ) with learning rate set as 0.002. Our method is implemented in PyTorch 1.7 with NVIDIA GeForce RTX 3090. The training process converges in 30 epochs.

Additionally, we utilize the center crop method instead of the random crop method to acquire more representative cropped cubes as the model’s inputs. The center crop method is used to guarantee that the cropped cubes are able to contain sufficient airway features. Given a CT scan with a size of $S^{{}^{\prime}}\times H^{{}^{\prime}}\times W^{{}^{\prime}}$ , we first search for the minimum and maximum index for airway region in x, y, and z directions. We denote these index pairs as $(X_{min},X_{max}),(Y_{min},Y_{max}),(Z_{min},Z_{max})$ and randomly choose a voxel $P=(x,y,z)$ from these spans as the cropped center. The input cube and corresponding binary label with a size of $S_{0}\times H_{0}\times W_{0}$ are cropped from original CT scans and ground truth maps based on the cropped center. Then the cropped cubes are fed into the model for training. Furthermore, we utilize the sliding window technique in the testing process to acquire each cube’s prediction and combine them to form the whole airway segmentation result.

3.2 Evaluation metrics and results

To evaluate our method, we adopt three metrics to assess the topological completeness and segmentation accuracy of airway prediction results: (1) Branches detected (BD), (2) Tree-length detected (TD), and (3) Dice coefficient (DSC). The definition of three metrics could be referred to lo2012extraction .

In our method, 3D U-Net is adopted to be the backbone of the segmentation framework. Therefore, we compare our method with original 3D U-Net cciccek20163d and other related state-of-the-art methods like Wang et al. wang2019tubular and Juarez et al. juarez2019joint . Wang et al. wang2019tubular develop a radial distance loss for detecting more tiny airway tubular structures. Juarez et al. juarez2019joint introduce graph neural network (GNN) module into the deepest level of 3D U-Net to improve airway connectivity. These methods are implemented by ourselves and fine-tuned on the experiment dataset. Table 1 gives the comparison results which show that our method achieves the best performance in BD and TD with compelling DSC. Compared to others, our method increases the BD and TD by over 3 % and 2 %, respectively. Since these two metrics could reflect the topological completeness of airway segmentation, the comparison results also demonstrate that our method outperforms the others in detecting more small bronchi and improving the connectivity of airway segmentation.

Table 1: Results (%) of the proposed framework compared to state-of-the-art methods (Mean

\pm

Standard deviation).

Method	BD	TD	DSC
3D U-Net cciccek20163d	$86.06\pm 11.72$	$83.10\pm 9.62$	$93.35\pm 1.74$
Wang et al. wang2019tubular	$87.97\pm 9.49$	$84.61\pm 8.54$	$93.12\pm 2.00$
Juarez et al. juarez2019joint	$87.40\pm 10.15$	$84.60\pm 8.94$	$\mathbf{93.51\pm 1.57}$
Our proposed	$\mathbf{90.83\pm 9.38}$	$\mathbf{87.59\pm 8.71}$	$92.95\pm 1.61$
\botrule

Qualitative comparison of airway segmentation shown in Fig. 4 also demonstrates the effectiveness of our method. Compared to other methods, more small branches and tiny bronchi are reconstructed in our method, which results in the great improvement of the airway topological connectivity. The preservation of airway topological completeness is significant for clinical practice like bronchoscopic intervention and surgical navigation. In Fig. 4 (b), a great number of breakages (false negatives voxels in green color) are eliminated in our method while maintaining a high overall segmentation accuracy.

3.3 Ablation study

We also conduct ablation studies to further investigate the effect of each component in our method. Since we use dense structures in the LTSP method, the density of this structure should be considered in our framework. Therefore, experiment comparisons are made by utilizing modules with different propagation distances in a single pass. Furthermore, to validate the effectiveness of the LTSP cell in slice propagation, experiments with or without LTSP cells are also conducted.

Table 2: Results (%) of ablation study on the testing set (Mean

\pm

Standard deviation).

Propagation distance	BD	TD	DSC
No propagation module	$86.06\pm 11.72$	$83.10\pm 9.62$	$\mathbf{93.35\pm 1.74}$
One-slice module	$87.97\pm 9.49$	$84.61\pm 8.54$	$93.12\pm 2.00$
One-slice module + LTSP cells	$90.11\pm 10.28$	$87.20\pm 8.96$	$93.15\pm 1.79$
Two-slices module	$89.22\pm 9.41$	$86.39\pm 7.84$	$92.42\pm 2.79$
Two-slices module + LTSP cells	$\mathbf{90.83\pm 9.38}$	$\mathbf{87.59\pm 8.71}$	$92.95\pm 1.61$
\botrule

As shown in Table 2, the LTSP cells and more densely-connected structure both can achieve higher BD and TD with DSC maintained. For the comparison of propagation distance, the two-slices module outperforms the no propagation module and one-slice module since it enables the slice feature to propagate to more slices directly. Furthermore, experiment results show that the designed LTSP cell could effectively improve the segmentation accuracy and topological completeness by extracting more valuable continuity information.

4 Conclusion

This paper proposed an effective long-term slice propagation (LTSP) method for accurate airway segmentation. Focusing on alleviating breakage phenomenon and improving topological connectivity, LTSP cell and dense slice propagation were designed to fully exploit the slice continuity relationship. Extensive experiments showed that the proposed method outperformed some state-of-the-art methods by detecting more tiny bronchi and reconstructing essential branches, which further validates the effectiveness of its constituents. The proposed method is beneficial to extracting topological complete airway segmentation for bronchoscopic intervention.

References

\bibcommenthead
(1) Higgins, W.E., Cheirsilp, R., Zang, X., Byrnes, P.: Multimodal system for the planning and guidance of bronchoscopy. In: Medical Imaging 2015: Image-Guided Procedures, Robotic Interventions, and Modeling, vol. 9415, p. 941508 (2015). International Society for Optics and Photonics
(2) Mehta, A.C., Hood, K.L., Schwarz, Y., Solomon, S.B.: The evolutional history of electromagnetic navigation bronchoscopy: state of the art. Chest 154(4), 935–947 (2018)
(3) Shen, M., Gu, Y., Liu, N., Yang, G.-Z.: Context-aware depth and pose estimation for bronchoscopic navigation. IEEE Robotics and Automation Letters 4(2), 732–739 (2019)
(4) Fabijańska, A.: Two-pass region growing algorithm for segmenting airway tree from mdct chest scans. Computerized Medical Imaging and Graphics 33(7), 537–546 (2009)
(5) Chen, F., Gao, J.: An automatic multiscale region growing segmentation in medical image based on slic and 2d otsu. In: Proceedings of the 3rd International Conference on Intelligent Information Processing, pp. 58–62 (2018)
(6) Aykac, D., Hoffman, E.A., McLennan, G., Reinhardt, J.M.: Segmentation and analysis of the human airway tree from three-dimensional x-ray ct images. IEEE transactions on medical imaging 22(8), 940–950 (2003)
(7) Juarez, A.G.-U., Tiddens, H.A., de Bruijne, M.: Automatic airway segmentation in chest ct using convolutional neural networks. In: Image Analysis for Moving Organ, Breast, and Thoracic Images, pp. 238–250 (2018). Springer
(8) Qin, Y., Gu, Y., Zheng, H., Chen, M., Yang, J., Zhu, Y.-M.: Airwaynet-se: A simple-yet-effective approach to improve airway segmentation using context scale fusion. In: 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), pp. 809–813 (2020). IEEE
(9) Qin, Y., Zheng, H., Gu, Y., Huang, X., Yang, J., Wang, L., Zhu, Y.-M.: Learning bronchiole-sensitive airway segmentation cnns by feature recalibration and attention distillation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 221–231 (2020). Springer
(10) Zhang, M., Yu, X., Zhang, H., Zheng, H., Yu, W., Pan, H., Cai, X., Gu, Y.: Fda: Feature decomposition and aggregation for robust airway segmentation. In: Domain Adaptation and Representation Transfer, and Affordable Healthcare and AI for Resource Diverse Global Health, pp. 25–34 (2021). Springer
(11) Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-assisted Intervention, pp. 234–241 (2015). Springer
(12) Çiçek, Ö., Abdulkadir, A., Lienkamp, S.S., Brox, T., Ronneberger, O.: 3d u-net: learning dense volumetric segmentation from sparse annotation. In: International Conference on Medical Image Computing and Computer-assisted Intervention, pp. 424–432 (2016). Springer
(13) Qin, Y., Chen, M., Zheng, H., Gu, Y., Shen, M., Yang, J., Huang, X., Zhu, Y.-M., Yang, G.-Z.: Airwaynet: a voxel-connectivity aware approach for accurate airway segmentation using convolutional neural networks. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 212–220 (2019). Springer
(14) Juarez, A.G.-U., Selvan, R., Saghir, Z., de Bruijne, M.: A joint 3d unet-graph neural network-based method for airway segmentation from chest cts. In: International Workshop on Machine Learning in Medical Imaging, pp. 583–591 (2019). Springer
(15) Pan, X., Shi, J., Luo, P., Wang, X., Tang, X.: Spatial as deep: Spatial cnn for traffic scene understanding. In: Thirty-Second AAAI Conference on Artificial Intelligence (2018)
(16) Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural computation 9(8), 1735–1780 (1997)
(17) Milletari, F., Navab, N., Ahmadi, S.-A.: V-net: Fully convolutional neural networks for volumetric medical image segmentation. In: 2016 Fourth International Conference on 3D Vision (3DV), pp. 565–571 (2016). IEEE
(18) Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
(19) Lo, P., Van Ginneken, B., Reinhardt, J.M., Yavarna, T., De Jong, P.A., Irving, B., Fetita, C., Ortner, M., Pinho, R., Sijbers, J., et al.: Extraction of airways from ct (exact’09). IEEE Transactions on Medical Imaging 31(11), 2093–2107 (2012)
(20) Wang, C., Hayashi, Y., Oda, M., Itoh, H., Kitasaka, T., Frangi, A.F., Mori, K.: Tubular structure segmentation using spatial fully connected network with radial distance loss for 3d medical images. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 348–356 (2019). Springer