DD-VNB: A Depth-based Dual-Loop Framework for Real-time Visually Navigated Bronchoscopy

The training strategy of the proposed depth estimation module is shown in Fig. 3. Training of our depth estimation network employs an unpaired image-to-image translation framework based on cycle consistency [26]. The depth estimation aims to map a bronchoscopic frame $x\in X$ to its depth space by $G_{{depth}}:X\rightarrow Z$, generating its corresponding depth map $\hat{z}_{t}=G_{{depth}}\left(x_{t}\right)$. The cycle is completed by reconstructing $\hat{z}_{t}$ back to the domain $X$ using $G_{{image}}:Z\rightarrow X$. The translation from $Z$ to $X$ is similarly performed. The model enforces cycle consistency for $G_{{depth}}$ and $G_{{image}}$. Finally, discriminators $D_{{depth}}$ and $D_{{image}}$ distinguish between generated/real depth maps and video frames, respectively.

We use the LS-GAN loss [27] as adversarial loss $L_{adv}$ and L1 losses for $L_{cyc}$ and $L_{iden}$ to guide the networks to learn domain transferring as preserving important structure.

View Consistency Loss: To enforce the network to learn depth estimation with absolute scale, virtual camera poses of input depth maps are also collected from our simulator to impose view consistency between generated bronchoscopic video frames.

Taking depth map $z_{t-n}$ and $z_{t}$ as input, network $G_{{image}}$ generates bronchoscopic images $\hat{x}_{t-n}$ and $\hat{x}_{t}$. With ground truth 6 DOF relative camera pose $P_{t-n,t}$ between depth map $z_{t-n}$ and $z_{t}$, along with camera intrinsic $K$, a homogeneous pixel $p_{t-n}\in\hat{x}_{t-n}$ can be warped to $\hat{p}_{t}\in\hat{x}_{t}$ according to view synthesis:

Differentiable bilinear sampling is applied to $\hat{x}_{t-n}$ according to continuous coordinates $\hat{p}_{t}$, mapping $\hat{x}_{t-n}$ to warped bronchoscopic image $w\left(\hat{x}_{t-n}\right)$, which is supposed to be in consistent with $\hat{x}_{t}$. Therefore, we propose to minimize the pixelwise inconsistency between warped frame $w\left(\hat{x}_{t-n}\right)$ and generated frame $\hat{x}_{t}$. For a pair of generated frames $\hat{x}_{t-n}$ and $\hat{x}_{t}$, the view consistency loss is defined as

To further constrain the learning of mapping $G_{{depth}}:X\rightarrow Z$, view synthesis is applied to input bronchoscopic frames $x_{t-n}$ and $x_{t}$ with their generated depth maps. Although ground truth camera motion between $x_{t-n}$ and $x_{t}$ is not accessible, as the estimated depth maps $\hat{z}_{t-n}$ and $\hat{z}_{t}$ are obtained, the inferred relative pose $\hat{P}_{t-n,t}$ could be calculated by ego-motion estimation, which will be discussed in detail in Section III.B. Thus, a satisfying depth map estimation should be informative to recover the camera motion and preserve object structure, which is formulated as a view consistency loss of:

Combining (2) and (3), the complete view consistency loss is defined as:

Geometry Consistency Loss: For generated consecutive depth map $\hat{z}_{t-n}$ and $\hat{z}_{t}$, suppose they conform the same 3D scene structure, the difference of their 3D depth attributes should be minimized. Following [28], we enforce geometry consistency loss on the predicted depth maps so that scale of a depth estimation sequence should all agree with each other, and as a result further constrain the learning of mapping $G_{{depth}}:X\rightarrow Z$.

For generated consecutive depth map $\hat{z}_{t-n}$ and $\hat{z}_{t}$, the depth inconsistency map $z_{{diff}}$ is defined in [28] as:

Combining the above loss terms, the overall loss is as follows:

Our localization module integrates an ego-motion estimation network into the loop of depth registration for real-time bronchoscope tracking, enhancing accuracy by using relative pose changes as initial guesses in depth registration against the pre-operative 3D model.

Depth Based Ego-motion Inference. As depth map and corresponding camera poses are accessible in virtual bronchoscopy, the learning objective is to minimize the difference between predicted transformation $\widehat{P}_{t-n,t}$ and ground truth transformation $P_{t-n,t}$. Therefore, the relative camera pose inference loss is defined as:

We use FlownetC encoder [29] to extract image features. Then, five convolutional blocks are utilized to regress a pose vector from extracted features. By using virtual depth maps and poses as training data, our ego-motion network can be deployed directly to estimated depth map of incoming bronchoscopic frames during test time.

Depth Map Registration. Relying on ego-motion estimation network alone for tracking yields accumulative error, as errors of previous inferences will propagate over time to current estimation in incremental localization. Thus, refining previous results becomes necessary. Registration between estimated depth maps and pre-operative bronchi model provides absolute position of bronchoscope, eliminating accumulative error from previous relative pose inference.

With depth map registration, after generating depth map $\hat{z}_{t}$ from input bronchoscopic frame $x_{t}$, camera pose $P_{t}$ is estimated by minimizing the difference between $\hat{z}_{t}$ and rendered depth $z\left(P_{t}^{\prime}\right)$ in pre-operative airway model at pose $P_{t}^{\prime}$. The optimization process is described as:

Embedding ego-motion inference into registration, we introduce an embedded-loop localization diagram, where DD-VNB estimates the relative pose change of the bronchoscope in high frequency while the depth map registration against the pre-operative $3\mathrm{D}$ model periodically provides absolute location for error correction in a lower speed.

Taking one registration loop as example, denote $ref$ as the reference time point. At time $ref$, iterative registration of the estimated depth map $\hat{z}_{{ref}}$ against airway model begins. As the registration running, $\hat{z}_{{ref}}$ together with estimated depth map $\hat{z}_{{ref+i}}$ are taken as a pair of input to the ego-motion loop, inferring relative camera position $P_{{ref,ref+i}}$ at time $ref+i$ in real-time, where $i\in[1,m]$. When $P_{{ref,ref+m}}$ has been estimated, registration for $P_{{ref}}$ has completed. A more accurate pose inference $\dot{P}_{ref+m}$ for $ref+m$ is obtained by concatenating $P_{{ref}}$ and $P_{{ref,ref+m}}$. For the next iteration, $ref+m$ serves as the reference time point, and $\dot{P}_{ref+m}$ is considered as the initial value for registration. The selection of the hyperparameter $m$ is made with the aim of aligning the computational frame rate with the video capture speed.

Our experiments span phantom and patient datasets, training and testing our framework and benchmarks on each.

Phantom Data: Collected from a robotized bronchoscope, we have 13 video clips (640x480 res., 30fps), totaling 1000-3000 frames per clip. Training uses 8 right lung clips; testing uses 5 left lung clips. A high-precision CT scan of the lung phantom facilitates 3D reconstruction and segmentation for depth estimation and localization training.

Patient Data: Includes nine cases captured with an Olympus BF-6C260 bronchoscope by 10$\sim$15fps, supplemented by checkerboard videos for camera calibration. CT scans precede operations, with airway segmentation performed for model reconstruction. Training involves six cases with comprehensive airway videos (approx. 1500 frames each); testing involves three cases with trachea to lobar bronchus videos (150-200 frames each). Real and virtual bronchoscopy frames are manually aligned to ensure ground truth accuracy.

Virtual Bronchoscopy Data: Generated using the SOFA framework simulator, comprising 54 video clips (640x480 res., 400-1000 frames per case) with corresponding camera poses and depth maps for motion inference network training.

Our training leverages the Pytorch framework on an NVIDIA RTX3090 GPU. We train the depth estimation network on phantom and patient datasets seperately. Initially, we maintain the learning rate at 0.0001 without enforcing consistency ($\tau_{1}=\tau_{2}=\eta=0$) to accommodate early-stage training variability. Parameters adjust after 10 epochs to $\tau_{1}=0.3$, $\tau_{2}=5$ and $\eta=5$ for refined training over 100 epochs using the Adam optimizer. $\gamma$ and $\delta$ are set to 10, 5 and 1 respectively as in [26] during the whole training process. The ego-motion network, trained with virtual data reflecting different camera intrinsics, undergoes 300 epochs at a learning rate of $1\mathrm{e-5}$ with $\omega$ set to 100. Depth map registration employs Powell’s algorithm with set error tolerance and convergence criteria of 0.01.

As no ground truth camera movement and depth map is available during training process, monocular depth estimation methods of EndoSLAM [17] and CycleGAN [26] are employed as baselines for comparison. CycleGAN is the most commonly used depth estimation method in learning based VNB. EndoSLAM jointly trains unsupervised monocular depth estimation and camera pose networks by adopting view consistency and geometry consistency from [28].

Results are shown in Table I. To further validate the scale-awareness of our depth estimation, we add scale drift in the measurements, which is defined by:

To address scale ambiguity in EndoSLAM depth maps, we align the first frame’s average depth with ground truth, excluding scale drifts from our evaluation. Our method sets a new standard in depth estimation, with a notable low RMSE of $5.727\pm 0.885$ mm for phantom data and $7.631\pm 3.217$ mm for patient data, outperforming CycleGAN by 18.6% and 42.5% in RMSE reduction for phantom and patient data, respectively. This improvement, especially against CycleGAN’s baseline, underscores our method’s effective use of knowledge embedding for accurate depth estimation within absolute scale. We selected several typical images from patient data for the qualitative evaluation of depth estimation methods in Fig. 4.

All methods are trained and evaluated on phantom data and in-vivo patient data respectively. Our method and CycleGAN additionally necessitate the inclusion of virtual data captured with corresponding intrinsic parameters.

Due to the variations in video capturing frequencies, $m$ is selected to be 10 for phantom data and 3 for patient data, in order to ensure sufficient co-visibility between input frames for ego-motion estimation. We also notice enormous error when testing EndoSLAM localization performance in phantom data, due to lack of parallax between consecutive frames and the accumulative error during incremental pose inference. Instead, we test EndoSLAM by sampling one out of every ten frames from phantom data as input frames.

Results in phantom and patient data are shown in Table II and Table III respectively. Evaluation metrics follows existing standards [13, 31]. According to the tests in terms of ATE on data in phantom and patients, our method outperforms SOTA. The average ATE of our proposed framework is $4.7\pm 3.17$ mm in phantom and $6.49\pm 3.88$ mm in patient data, far exceeds the performance of EndoSLAM and CycleGAN with depth registration, which is the mostly reported in depth based bronchoscopic localization literature. Our method also achieved the highest SR-5 and SR-10, adding further proof for its superiority. We observe minor performance deterioration of our method from phantom data to patient data.

We provide an example case (Case-1) in patient data, showing the located view in virtual bronchoscope in Fig. 5. Fig. 6 shows the located position by different frameworks in Case-3 against their ground truth.

Our overall localization framework reaches average update frequency 33.9 Hz for phantom data and 12.2 Hz for patient data, due to different $m$ value. Reported learning based VNB algorithms that exceed our computation speed including [20], [10] and [9], set out to track camera position by a single localization network, which contributes to their high computation speed. However, their methods are difficult, if not impossible, to generalize to unseen airways or branches.

In order for our depth estimation network to obtain scale perception, we have integrated view consistency losses and geometry consistency loss during CycleGAN-style network training. By those consistency constraints, we are expecting to achieve lower scale drifts in our depth estimation, thereby reducing the estimation error. Defining $L_{rec_{-}\hat{x}}$ as the view consistency loss for generated bronchoscopic frame, $L_{rec_{-}x}$ as the view consistency loss for real bronchoscopic frame, and $L_{gc}$ as geometry consistency loss, here we specifically investigate the following cases:

1. 1.

   our depth estimation with $L_{rec_{-}\hat{x}}$ , $L_{rec_{-}x}$ and $L_{gc}$,

2. 2.

   our depth estimation with $L_{rec_{-}x}$ and $L_{gc}$, without $L_{rec_{-}\hat{x}}$,

3. 3.

   our depth estimation with $L_{gc}$, without $L_{rec_{-}\hat{x}}$ and $L_{rec_{-}x}$,

4. 4.

   our depth estimation without $L_{rec}$ and $L_{gc}$, deteriorates as CycleGAN.

The results for depth estimation are given in Table I. As seen from quantitative ablation analysis, the consistency losses make our depth estimation more accurate in both phantom and patient data. View consistency loss which utilizes camera movement for bronchoscopic frame view synthesis, contributes the most with regard to scale perception, and in turn reduces the estimation error.

We conduct ablation studies to assess various depth estimation and pose inference combinations within our localization framework. Table II and Table III shows the results on phantom and patient datasets respectively. Initially, we evaluate our depth estimation technique against CycleGAN, highlighting its impact on bronchoscopic localization. We further examine the role of ego-motion estimation by comparing localization accuracy with and without it, ensuring consistent searching spaces and hyperparameters across tests. Our findings underscore the significance of integrating accurate depth estimation and ego-motion inference for enhanced localization performance, particularly noting ego-motion’s sensitivity to depth estimation scale drift. Results on both phantom and patient datasets confirm our framework’s effectiveness, with ego-motion inference proving crucial for avoiding local minima in depth map registration.

All of the methods are tested on a workstation with a 12th Gen Intel® Core ™ i7-12700 CPU and a NVIDIA RTX3090 GPU. Input images and depth maps are all cropped and resized to 256 × 256-res. Results are presented in Table II and Table III. Runtime variance of our localization framework between phantom and patient data lies on the different $m$ value. Depth map rendering takes up the most time in registration methods. Rendering one frame takes about 1.7ms in our implementation. Rendering acceleration should further speed up our overall framework.