Enhanced Scale-aware Depth Estimation
for Monocular Endoscopic Scenes
with Geometric Modeling

Fig. 1 shows an overview of our proposed scale-aware monocular depth estimation framework which consists of three parts. First, we fuse estimations at different image resolutions, and use a relative depth estimation network to generate high-quality depth maps. Second, we propose a geometry modeling method to track the poses of the surgical instruments in the image using only geometric primitives from the endoscopic images. Third, the 3D poses of the instruments and the depth map are united to recover the scale between relative depth and real-world values. Thus, the absolute depth of the monocular endoscopic scenes can be estimated via the scale recovery based on surgical instruments geometry.

Our relative depth estimation network utilizes Monodepth2 [5] as its backbone. In the training phase, the network is learned by minimizing the photo-metric loss between the input images and the corresponding synthetic frames generated through novel view synthesis. However, since the inputs images are cropped to a relatively low resolution and then fed to the network for training, many details are lost while performing depth estimation. To address this issue, we develop an enhancing depth module that fuses two estimates for the same endoscopic image at different resolutions to improve the quality of relative depth maps.

Our depth model is capable of handling arbitrary image sizes for depth estimation. As shown in Section A of supplementary material, when input with different resolutions is fed into the network, we can observe a specific trend in the depth maps. In low resolutions that are close to the training data, the estimated depth consistently captures the overall structure but may lack high-frequency details. When the same image is input to the model in higher resolution, more details are captured while the structure consistency of the result gradually degrades. Further, the second observation is that when a fusion operation is performed to combine relative depth estimates with different resolutions, the details in the depth estimated from higher-resolution input are transferred to the lower-resolution depth map while maintaining structural consistency. Following these observations, we present a module to improve the depth maps in low resolution.

In our enhancing module shown in Fig. 1 (a), we generate two depth estimations of the same endoscopic image. One depth map estimated from the network with a smaller-res input has a consistent structure but suffers from fine details. Another depth predicted with a higher-res input contains more details. Thus, we adopt a low pass fusing filter [6] with low-res depth as a guide and apply it to the high-res depth to generate a higher-quality relative depth with good structural consistency and details. Based on the above fusing process, the enhanced relative depth map $\mathbf{D}_{i}$ for each monocular image is predicted.

As illustrated in Fig. 1(b), most of surgical instruments have a cylindrical shaft with a constant radius $r_{s}$. Hereby, given an endoscopic frame $i$, we can calculate the 3D pose ${}^{e}\mathbf{T}_{i}\in\mathbb{R}^{3\times 4}$ of the tool based on its geometrical primitives (i.e., boundaries and tip), where $e$ denotes the endoscopic image coordinate. The axis of the tool at pose ${}^{e}\mathbf{T}_{i}$ under the endoscope frame can be represented with the Plücker coordinates $\left({}^{e}\mathbf{s}_{i},{}^{e}\mathbf{m}_{i}\right)$, with ${}^{e}\mathbf{s}_{i}\in\mathbb{R}^{3\times 1}$ being a unit vector denoting the direction of the tool’s shaft and ${}^{e}\mathbf{m}_{i}\in\mathbb{R}^{3\times 1}$ being the moment of the tool. Further, assume ${}^{e}\mathbf{c}_{j}$ is a 3D point on the tool’s surface and ${}^{e}\widetilde{\mathbf{c}}_{j}=[{}^{e}\mathbf{c}_{j}\;1]^{\mathsf{T}}$ is its homogeneous coordinate. Based on the geometrical modeling [2] of cylindrical objects, the following relationship can be derived:

According to the perspective projection theory [2], the Plücker coordinates of the tool’s axis can be associated with the edge boundaries $\left(\mathbf{l}_{i}^{-}\in\mathbb{R}^{3},\mathbf{l}_{i}^{+}\in\mathbb{R}^{3}\right)$ of the shaft in the image plane as follows:

with $\mathbf{K}$ is the camera intrinsic and $\alpha=\frac{r_{s}}{\sqrt{\|{}^{e}\mathbf{m}_{i}\|^{2}-r_{s}^{2}}}$. Please refer to Section B of supplementary material for detailed derivation. Besides, based on the shaft mask predicted by the tool segmentor, we can compute the edge boundaries from the mask contour. After that, the $\left({}^{e}\mathbf{s}_{i},{}^{e}\mathbf{m}_{i}\right)$ can be calculated via combining Eq. 2 and the boundaries equations from the 2D mask. In this regard, we can obtain the 3D pose ${}^{e}\mathbf{T}_{i}$ of the surgical instrument from the Plücker coordinates $[{}^{e}\mathbf{s}_{i},{}^{e}\mathbf{m}_{i}]$.

In this section, we first calculate the 3D points on the surface of the shaft on the basis of the calculated pose. By jointly taking these points and corresponding depth information from the estimated relative depth map in Section 2.2, we can compute the transformation scale parameters between the relative depth and the real world. First, the $\left({}^{e}\mathbf{s}_{i},{}^{e}\mathbf{m}_{i}\right)$ can be further transformed to the Plücker matrix $\mathbf{L}_{i}$ for the representation of the axis:

The projection of the axis of the tool in the image plane is calculated as $\mathbf{l}_{s}=\mathbf{K}^{-\mathsf{T}}\cdot{}^{e}\mathbf{m}_{i}$. As shown in Fig. 1 (b), the shaft tip $\mathbf{p}_{s0}=[u_{s0}\;v_{s0}\;1]^{\mathsf{T}}$ along $\mathbf{l}_{s}$ is firstly predicted from the endoscopic image, where $(u_{s0},v_{s0})$ denotes the image pixel. The corresponding 3D point $\mathbf{c}_{s0}$ along the axis of shaft is developed as follows:

where $\mathbf{w}$ is an vector calculated by $\mathbf{L}_{i}$ and camera intrinsic. So the depth of the point $\mathbf{c}_{f0}$ on the surface of the shaft is derived (see Section C of supplementary material). Based on the above process, the depth values $\left(z(\mathbf{c}_{f0}),\cdots,z(\mathbf{c}_{fn})\right)$ of 3D points re-projected by the pixel $\left(\mathbf{p}_{s0},\cdots,\mathbf{p}_{sn}\right)$ along $\mathbf{l}_{s}$ can be computed.

Then, we obtain the relative depth values of $\left(\mathbf{p}_{s0},\cdots,\mathbf{p}_{sn}\right)$ from the depth map $\mathbf{D}_{i}$, which are defined as $\left(\mathbf{D}_{i}[\mathbf{p}_{s0}],\cdots,\mathbf{D}_{i}[\mathbf{p}_{sn}]\right)$. Therefore, the transformation scale parameters between the relative depth and the real world can be solved in closed form as a standard least-squares problem:

Afterward, as shown in Fig. 1 (c), we can recover the absolute depth $\mathbf{A}_{i}$ of the monocular endoscopic image utilizing the parameters $\left(\eta,\gamma\right)$.

Evaluation on Scale-Aware Depth Estimation. We present the quantitative depth comparison results on our in-house surgery data in Table 1, which rescale the results using the ground truth median scaling method. Apart from depth metrics, we calculate the means and standard errors of the re-scaling factors to showcase the scale-awareness capability. Our proposed depth model demonstrates the best performance in terms of up-to-scale accuracy across all metrics. Specifically, the state-of-the-art baseline method only achieves an MAE error of 8.564 mm, an RMSE of 11.032 mm, and an accuracy of 76.9%. In contrast, our method demonstrates superior performance with an MAE error of 6.347 mm, an RMSE of 8.637 mm, and an accuracy of 88.0%. Notably, our model also achieves near-perfect accuracy in terms of absolute scale. These quantitative results show that the proposed method indeed extracts the absolute scale with geometric modeling, resulting in fine scale-aware depth estimation. Furthermore, four typical images are selected for qualitative depth comparison. As shown in Fig. 2, our method can predict the scale-aware depth with smaller errors, sharp boundaries and fine-grained details compared to other approaches. Moreover, the runtime of the proposed method is around 66 ms per frame.