SuPer Deep: A Surgical Perception Framework for Robotic Tissue Manipulation using Deep Learning for Feature Extraction

Our surgical perception framework, as shown in Fig. 1, consists of a deformable tissue tracker and a surgical tool tracker to perceive the entire surgical scene, including the the deforming environment and robotic agent. Two deep neural networks are embedded into our framework for specific feature extractions: DNN(1) finds and matches features from stereo images to generate a depth map for tissue tracking, and DNN(2) extracts point features for surgical tool tracking.

To track the tissue deformation, we update tissue model by fusing the estimated depth maps using the previously developed model-free tissue tracker [25]. The pose of the surgical tool is estimated using a model-based tracker that utilizes a kinematic prior and fusing the encoder readings with the 2D detection from the images, which will be described in Section II-B. In order to separate the tools from the deformable tissue model, a mask of the surgical tool is generated by rendering the 3D CAD model into the endoscopic camera view and is removed from the depth map fed to the deformable tissue tracker. Finally, we combine the tissue point cloud and surgical tools into the camera frame to reconstruct the entire surgical scene.

To localize the surgical tools on image frames, we use DeepLabCut [27], which employs DeeperCut [28] as the backbone for point feature detection. The DNN consists of variations of Deep Residual Neural Networks (ResNet) [29] for feature extraction and deconvolutional layers to up-sample the feature maps and produce spatial probability densities. The output estimation for each point feature is represented as a tuple $(\mathbf{h}^{i},\rho^{i})$, where $\mathbf{h}^{i}\in\mathbb{R}^{2}$ is the image coordinate of the $i$-th feature and $\rho^{i}\in\mathbb{R}$ is the corresponding confidence score. The DNN was fine-tuned with few training samples to adapt to surgical tool tracking by minimizing the cross-entropy loss. The samples were hand-labeled using the open-source DLC toolbox [30]. Fig. 2 shows examples of point features that were detected on surgical instruments.

To estimate the pose of the surgical tool in 3D space, the 2D detections are combined with the encoder readings from the surgical robot, and a particle filter is applied for estimation. Previous work has shown low-profile, cable driven surgical robots [31, 32] have inaccurate in joint angles [33] and challenges calibrating the transform between the camera and robotic base, also known as hand-eye [34]. To localize the surgical tool and overcome these uncertainties, we utilize our previously developed formulation and track the Lumped Error. [35]. Let the Lumped Error be defined as $\mathbf{T}^{b-}_{b}(\boldsymbol{\omega},\mathbf{b})\in SE(3)$, which is parameterized by an axis-angle vector $\boldsymbol{\omega}\in\mathbb{R}^{3}$, and a translational vector $\mathbf{b}\in\mathbb{R}^{3}$. The Lumped Error compensates for both error in joint angles and hand-eye, for more details refer to our previous work [35]. To estimate $\boldsymbol{\omega}$ and $\mathbf{b}$ at time $t+1$ given time $t$, a zero-mean Gaussian noise is assumed to model the uncertainty. Therefore, the motion model for the particle filter is defined as:

where $\mathbf{\Sigma_{\boldsymbol{\omega},b}}$ is the covariance matrix.

Using this Lumped Error formulation, a detected feature point can be projected to the image plane by being first transformed by the kinematic chain and then the corrected hand-eye. More specifically, feature point $i$ on the $j_{i}$-th link $\mathbf{p}^{j_{i}}\in\mathbb{R}^{3}$ is projected onto the image plane by:

where $\mathbf{T}_{i}^{i-1}(\theta_{i})\in SE(3)$ is the $i$-th joint transform generated by joint angle $\theta_{i}$, $\mathbf{T}^{c}_{b-}$ is the initial hand-eye transform from the set-up joints or calibration, $\mathbf{K}$ is the intrinsic camera matrix, and $s$ is the scaling factor that constraints the point on the image plane. Note that $\overline{\cdot}$ denotes the homogeneous representation of a point (e.g. $\overline{\mathbf{p}}=[\mathbf{p},1]^{T}$).

From here, an observation model can be defined by relating the predicted feature location with the detected features. Given a list of $N$ observations, $(\mathbf{h}_{t+1},\boldsymbol{\rho}_{t+1})$, the observation model is defined to be:

where $\gamma$ is a tuning parameter.

Since the deep neural network performs end-to-end 2D feature detection, the only parameters that need to be tuned for tool tracking are $\gamma$ and $\mathbf{\Sigma_{\boldsymbol{\omega},b}}$. The direct association of the key-point features from the neural network also eliminates the need for explicit data association - a significant improvement for tool tracking over the previous method [25]. Furthermore, the feature detection from the deep neural network does not rely on the tracked states, which is a common technique in surgical tool tracking [14] and can lead to detrimental results when the tracking begins to fail.

Deformable tissue tracking relies heavily on the quality of depth estimation, as the deformable tracker uses the depth maps as the observation [20], [25]. To estimate the depth, a stereo matching algorithm is used to compute the disparity, and then inverted to obtain pixel-wise depth.

Traditional stereo matching algorithms, like [19] and [36], typically take a pair of rectified stereo images $I_{l}$ and $I_{r}$ as input, and estimate the disparity by matching image patches or features between $I_{l}$ and $I_{r}$. Due to the complexity of the surgical setting, the image quality is not the sharpest, which makes finding pixel-level correspondence extremely challenging. Deep-learning-based algorithms, such as [37], [38], and [39], use a weight-sharing feature extractor to obtain feature maps $F_{l}$ and $F_{r}$ from $I_{l}$ and $I_{r}$, respectively. Then a 4D matching cost volume $C_{d}$ is formed by concatenating the $F_{l}$ and $F_{r}$, such that $C_{d}(i,j,d,:)$ is the concatenation of $F_{l}(i,j)$ and $F_{r}(i,j+d)$, where $(i,j)$ is the pixel location and $d$ is the disparity. The cost volume is regularized using 3D convolutional layers and reducing the dimension of the 4-th channel to 1. Finally, the resulting 3D tensor $S_{d}$ is used to estimate the disparity for each pixel as

where $D_{max}$ is the max disparity and $\sigma(\cdot)$ denotes the softmax function. An investigation between these stereo matching algorithms will be presented in the Section III. After estimating the disparity $\hat{d}$, the depth value $z$ is obtained by the following transform

where $b$ is the horizontal offset (i.e., baseline) between the two cameras, and $f$ is the focal length, which can be obtained from camera calibration.

Various stereo-matching algorithms can be substituted in for DNN(1) of our framework shown in Fig. 1. To find the best one, we investigated several stereo matching algorithms by combining with the previously developed tissue tracker [25] for deformable tissue tracking. The state-of-the-art algorithm, Guided Aggregation Network (GA-Net) [39], was finally chosen for our framework. In comparison to previous works on deformable tissue tracking, which require a substantial amount of spatial and temporal filtering on the depth image [20], [25], our method employs a deep stereo matching network for accurate and dense depth estimation, which requires no post-processing on the depth image.

The Surgical Perception (SuPer) dataset^***https://sites.google.com/ucsd.edu/super-framework/home mentioned in [25] is a recording of a repeated tissue manipulation experiment using the da Vinci Research Kit (dVRK) [32]. The dataset consists of a raw stereo endoscopic video stream and encoder readings from the surgical robot with ground-truth labels for the tool tracking tasks, which consists 50 hand-labeled surgical tool masks. The tool tracking performance was evaluated by calculating the Intersection-Over-Union (IoU, Jaccard Index) for the rendered tool masks, which are based on estimated tool poses.

The Hamlyn Centre Video Dataset [40] was used to evaluate the performance of deformable tissue tracking. It includes two video sequences of silicone heart phantom deforming with cardiac motion and consists of ex-vivo endoscopic stereo videos (resolution: 360$\times$288) with depth information generated from CT scans. The re-projected depth maps of the reconstructed tissue model are evaluated, which is the projection of the entire reconstructed point cloud to the image plane, with each pixel containing a depth value. We calculated the per-pixel root-mean-square (RMS) error of the depth map for every image:

where $i,j$ is the pixel position, $\hat{d}$ is the estimated depth value, $d$ is the ground truth depth value, and $N_{p}$ is the total number of pixels for each image. We also reported the percentage of the valid (non-zero) pixels of the depth map.

The da Vinci tool tracking dataset used in [14] consists of a stereo video stream and the corresponding kinematic information of the da Vinci® surgical robot. The dataset is used to evaluate surgical tool feature detection and pose estimation. Note that the SuPer dataset has painted markers, and hence this additional experiment ensures that surgical tool feature detector learns surgical tool point features and is not dependent on colored markers. The performance of feature detection is evaluated by calculating the $L_{2}$ norm of the error in pixels, for the $i$-th feature:

where $N$ is the total number of test images, $\mathbf{h}^{i}_{n}$ is the predicted feature point location, and $\mathbf{t}^{i}_{n}$ is the ground truth feature point location in the $n$-th image. We experiment with varying amounts of hand-labeled training data to illustrate the data efficiency of the proposed surgical tool feature detection method.. Due to lack of ground-truth data for pose estimation, we only provide qualitative results for surgical robotic tool tracking on this dataset.