Situation Awareness for Automated Surgical Check-listing in AI-Assisted Operating Room

The various surgical tasks involved in the data collection process include

• •

  Laparoscopic cholecystectomy using a standardised porcine model.

• •

  Body torso laparoscopic box trainer with prepositioned trocars. Possibly a haptic box trainer.

• •

  Basic laparoscopic stack and standard surgical equipment (hook diathermy, Maryland graspers, crocodile graspers, 10mm and 5mm trocars, 5/10mm Endoclip applicator, Bert bag/Endocatch, 30-degree camera).

This subsection provides a brief description of the various surgical phases involved in the data collection process.

Retraction of the gallbladder: The surgeon passes the instrument(s) to the assistant surgeon, who holds the flaps up, in order for the surgeon to have a better view of the regions of interest, which are the gallbladder, liver, and cystic duct.

Dissection of critical view of safety: The surgeon uses hook diathermy and a Maryland grasper to dissect the hepatocystic/calot’s triangle.

Clipping and division of the cystic duct: The surgeon uses endoclip and laparoscopic scissors to clip the region of interest, which in this case is the cystic duct.

Clipping and division of the cystic artery: The surgeon uses endoclips and laparoscopic scissors to clip the region of interest, which in this case is the cystic artery.

Dissection of the gallbladder from the liver bed or cystic plate: The surgeon uses a crocodile grasper and hook diathermy to dissect the gallbladder from the liver bed. This causes the presence of smoke in the surgical scene.

Removal of the gallbladder: The surgeon uses a Bert bag or Endocatch to remove the gallbladder from the surgical scene.

It is worth mentioning that the participants involved in the data collection process were high surgical trainees specializing in general surgery, recruited from the Department of Surgery and Cancer, Imperial College London. The participants had previous exposure to basic laparoscopic surgical training and simulated and clinical exposure to laparoscopic surgical/cholecystectomy training.

For the object detection task, the endoscope videos were manually annotated by drawing bounding boxes around the tips of the surgical instruments of interest in each image frame and entering the corresponding class label associated with each bounding box. This is what we term the ”ground truth labels.” For this annotation task, we annotated eight different surgical instruments: Crocodile grasper, Johan grasper, hook diathermy, Maryland grasper, clipper, scissors, bag holder, and trocar.

VoTT: The virtual object tagging tool (VoTT), a free and open-source data annotation tool from Microsoft, was used to annotate our endoscope video dataset. VoTT was used to draw bounding boxes around the tips of the surgical instruments of interest in the dataset. Figure 1 depicts a screenshot of its graphical user interface.

LabelMe: LabelMe is a free open-source data annotation tool developed by MIT for manual image annotation in object detection, classification, and segmentation tasks. LabelMe was created in Python with the goal of collecting a large collection of images with ground-truth labels. LabelMe is simple to use and allows polygons, circles, rectangles, lines, line strips, and points to be drawn, among other shapes. Fig. 2 provides an overview of the use of LabelMe for image annotation.

To prepare our novel endoscope dataset for training, we performed a series of data preprocessing steps. We developed a Python script to convert our novel endoscopic videos into image frames. We implemented the Microsoft VoTT annotation tool to annotate the dataset and performed an adaptive image-scaling operation to obtain a standard image size of 640×640 for training. The Microsoft VoTT annotation tool provides the annotation of our dataset in the MSCOCO data format, which is not an acceptable data format for YOLO algorithms. A Python script was developed to convert the annotation from the MSCOCO data format to YOLO data format.

The unmodified YOLOv5 model uses CSPDarknet as its backbone. In this paper, we replaced the CSPDarknet backbone with a modified VGG-11 backbone by removing the fully connected layers in the original VGG-11 architecture and adding the SPP layer, which is an additional layer between the unmodified YOLOv5 backbone and neck. VGG, which represents the virtual geometry group, was first proposed by A. Zisserman and K. Simonyan at the University of Oxford. They investigated the influence of convolutional neural network depth on accuracy in the context of large-scale image recognition [15]. Their main contribution was to increase the depth of the model by replacing the large kernel-size filters with very small 3 × 3 convolution filters. This achieved significant improvement over the prior art, such as AlexNet [7] and almost 92.7% top-5 test accuracy on the ImageNet dataset.

The neck in YOLOv5 is a series of layers between the backbone and the head. The unmodified YOLOv5 model uses a path aggregation network(PANet) as its neck architecture. In this paper, we experimented with two other neck architectures: a feature pyramid network (FPN) and a bi-directional feature pyramid network (Bi-FPN). The architecture is shown in Fig. 4.

Feature Pyramid Network (FPN) is not an object detector. This is a feature extractor architecture incorporated within an object detector. The feature pyramid network extracts scaled feature maps at multiple layers from a single-scale input image of any size and passes these extracted feature maps to the head of the object detector (for example, YOLOv3 head), which performs the detection task. This process is not influenced by the backbone of the object detector. A feature pyramid network is composed of single bottom-up and top-down information pathways. The bottom-up information pathway is the backbone of the object detector, which generates feature maps of varying sizes using a size-two scaling step. The top-down information pathway uses upsampling techniques (with two nearest neighbors) on the previous layer, which is then linked with feature maps from the final layer of each stage in the bottom-up information pathway using a skip connection. Each connection between feature maps from the bottom-up information pathway to the top-down information pathway has the same spatial size [8].

Bi-Feature Pyramid Network (BiFPN) also known as Weighted Bi-directional Feature Pyramid Network, is an improved Feature Pyramid Network (FPN) developed by the Google Research Brain Team [16]. BiFPN integrates the notion of multi-level feature fusion from feature pyramid networks (FPN), path aggregation networks (PANet), and neural architecture search-feature pyramid networks (NAS-FPN). Hence, it enables simple and rapid multi-scale feature integration. Major modifications integrated into the bidirectional feature pyramid network are (i) removal of nodes with only one input. Nodes with one input edge and no feature integration contribute less to feature networks seeking to fuse distinct features. (ii) Additional edges from the original input node to the output node of the same grade to integrate more features without increasing the computational complexity [16].

Although the YOLOv3 [12] object detection model achieved state-of-the-art performance with respect to its speed and accuracy, it still had a short fall with scale variation, which needed improvements since its multi-scale object detection capabilities are linked to its network’s receptive fields. YOLOv3-SPP [6] was proposed to ameliorate this problem by introducing a spatial pyramid pooling layer into the YOLOv3 network to efficiently tackle the scale variation problem and integrate multi-scale features effectively. This equips the network with the ability to learn the various objects’ features properly. This approach achieved an improved object detection performance (mAP) over YOLOv3 when experimented on the UA-DETRAC dataset. which is our justification for selecting it as one of our benchmark models.

The Scaled-YOLOv4 object detection model designed by Chien-Yao Wang, Alexey Bochkovskiy, and Hong-Yuan Mark [18] propelled the YOLOv4 model forward by introducing a network scaling strategy which doesn’t only scale the depth, breadth, and resolution of the network but also scales the structure of the network. Their strategy outperformed the previous state-of-the-art object detection model, EfficientDet [16], developed by the Google Brain Research team, on both ends of the speed versus accuracy frontier when tested on the MSCOCO [9] dataset.

YOLOR [20] which stands for “You Only Learn One Representation”, is a state-of-the-art object detection model which is a different variant in the YOLO series (i.e. YOLOv1 to YOLOV7) because it utilises a unified framework to concurrently encode implicit and explicit knowledge. This unified network can produce a unified representation to fulfil several jobs at the same time. The idea behind the YOLOR framework was adopted from humans’ ability to implicitly (learning subconsciously) or explicitly (through regular learning) comprehend their surroundings through perception, listening, touch, and memory. The authors also claimed that YOLOR can perform kernel space alignment, prediction refinement, and multi-task learning in a Convolutional neural network. Results from using the YOLOR network suggest that the inclusion of implicit knowledge into the network model improves the performance of the network on various tasks when experimented on the MSCOCO [9] dataset, YOLOR achieved comparable accuracy with Scale-YOLOv4 [18] but an inference speed 88% faster than Scale-YOLOv4.

YOLOv7 [19] which is the most recent object detection model within the YOLO family, was developed by Alexey Bochkovskiy, who took up the management of the YOLO algorithm after the original author, Joseph Redmon, stopped computer vision research due to ethical issues, and Kin-Yiu, Wong who joined the computer vision research domain with the innovation of cross-spatial networks, allowing YOLOv4 and YOLOv5 to construct more scalable backbone networks. YOLOv7 achieved state-of-the-art performance on the MSCOCO dataset [9] when compared with its peers. In order to achieve this height, the authors’ major contributions to the YOLOv7 model are: (i) introduction of an extended version of the efficient layer aggregation (ELAN) computational block, which they termed E-ELAN as their final aggregation layer. (ii) introduction of a novel model scaling technique, which can scale both the network depth and breadth simultaneously while concatenating layers together. (iii) addition of an auxiliary head network that is used to supervise the detection head during training and a model re-parameterization technique in order to make the model more robust and generalize well on new data.

The dataset consists of Pointclouds that were extracted after calibrating and combining eight Azure Kinect RGB-D cameras, each having a 1 MP time-of-flight (Microsoft Team, n.d.). We used eight cameras to cover all angles of the operating scene. The collected dataset consists of four sessions, each has 4000 Pointclouds. For the first training, we used 150 Pointclouds. We present one of the Pointcloud samples shown in Fig. 6.

We manually annotated the Pointclouds by drawing a 3D bounding box. The objects of interest were humans with the class name ‘Human’. We used an online annotation platform called Supervisely (shown in Fig. 7) (Supervisely: Unified OS for Computer Vision, n.d.), which has the advantage of copying bounding boxes across frames which, in turn, allowed faster-annotating procedure. Furthermore, it has a taskforce environment that splits the annotation work among annotators to supervise and validate the annotations by reviewers or administrators.

The data are preprocessing following the steps as listed below:

• 1.

  We downsampled the combined Pointclouds to end up with Pointclouds of lower size, which will also take less time to process than the original Ponitclouds.

• 2.

  Then we converted the Pointclouds from .ply format to .pcd format which is the form that Supervisley accepts.

• 3.

  After annotating the Pointcloud, we downloaded them along with their annotations. Then both the Pointclouds and their annotations were converted from .pcd and .json formats to .bin and .txt formats, respectively. This is because most of the 3D detection models were trained and tested on Autonomous Driving datasets like KITTI [3], nuScenes [2], and Waymo(Addanki et al., 2021), which have their Pointclouds in .bin format. We provide the samples of Pointclouds for pcd and bin format in Fig. 8(a) and 8(b), respectively.

We used an off-the-shelf 3D detector called Point-Voxel-RCNN (PV-RCNN) [14]. 3D detection models can be categorised into two main categories, Point-based and Voxel-based models. PV-RCNN has integrated the advantages of both categories in one model. The structure of PV-RCNN is shown in Fig. 9.

The model starts with voxelizing the input that is dividing it into smaller chunks of Pointclouds. The voxels are then fed into the 3D sparse convolutional encoder to extract multi-scale semantic features and generate 3D object proposal. Using a novel voxel set abstraction module, the extracted voxel-level features at multiple neural layers are summarised and aggregated to form a set of key points. Then the keypoint features are aggregated to the ROI-grid points to extract proposal specific features. Finally, a two-head fully connected network refines the proposed boxes and predicts a confidence score.

As mentioned earlier, most 3D detectors (including PV-RCNN) are designed to work outdoors, specifically for autonomous driving applications/detection. Figure 10 shows a significant difference between the AV and MAESTRO Pointclouds. In MAESTRO, Humans are represented by a high number of points in a dense configuration (because they are close to the sensor) and the dimensions of humans are large compared to the size of the environment (operating room). In AV applications, humans are represented by a low number of relatively sparse pints and the dimensions of humans are relatively small compared to the environment (outdoor). Consequently, we had to modify many parameters to cope with the inherent differences in human representations and environments.

• 1.

  Create a new python class and yaml file for a custom dataset (MAESTRO).

• 2.

  The classes of interest, currently we are interested in detecting humans only.

• 3.

  Point Cloud Range: The x, y, z ranges of interested. Those were in order of tens of meters in the original AV dataset, however, for MAESTRO, the range is limited to the size of the operating room which is in terms of a few meters.

• 4.

  Voxel Size: Same rationale of the previous point, in the original AV datasets, the outdoor environment is large so the voxles are large. In the MAESTRO dataset, the environment is significantly smaller, so we minimised the voxel size.

• 5.

  Anchor sizes: The starting point of the proposed bounding box. This can be decided based on our prior knowledge about the size of human, we changed to the average dimension of human in the MAESTRO Pointcloud.

• 6.

  Other dimensions and parameters so that the dimensions of the tensor within the model are consistent (the dimension of the output of a certain layer equal to the dimension of the input of the next layer).

Our novel endoscope dataset was extracted from two endoscope videos, referred to as Pilot 1 and Pilot 2. From Pilot 1, we annotated seven hundred and seventy-six image frames, and two thousand, six hundred and eighty-nine image frames were annotated from Pilot 2 (see Fig. 11). Each image frame has a corresponding text file that contains the coordinates of the ground-truth annotation for each image. This file is saved in a different folder but in the same directory. To split our novel endoscope dataset into training and testing sets, we designed a Python script to randomly extract 70% of Pilots 1 and 2 as the training sets, and the remaining 30% of Pilots 1 and 2 were used as the test sets. This resulted in our training set consisting of 2425 image frames and their corresponding annotations, and our test set consisting of 1040 image frames and their corresponding annotations. Figure 12 displays the distribution of annotations per surgical instrument category for Pilot 1 and Pilot 2.

All experiments were trained and tested on the Visual Artificial Intelligence Laboratory beta-Mars server with an Ubuntu 20.04 operating system and PyTorch framework. The server has four Nvidia GTX 1080 GPUs, each of which has 12 GB VRAM. The training time for each experiment took an average of six hours to complete. In Fig. 13, we demonstrate the visualization of surgical equipment detection using our model, where different colored bounding boxes correspond to various class categories in our dataset.

In deep learning problems, such as this one, hyperparameter setting is a fundamental task that is required to improve model performance. The grid search approach for selecting the best hyperparameter value for a model is often used during model training. However, a major disadvantage of this approach is that it requires a significant amount of runtime to train across all possible hyperparameter values, thereby increasing the computational complexity of the model. Owing to time constraints and limited access to computing resources, only a few hyperparameters were manually adjusted to select the best-performing model. Table 1 displays the final values of the selected hyperparameters used for training all the models.

In order to illustrate the efficacy of our proposed refinements to the YOLOv5 algorithm, we conducted an ablation study to demonstrate the effectiveness of each modification in an incremental manner using our novel endoscope dataset and YOLOv5l as our baseline model. F1 score, [email protected], [email protected]:0.95, and inference are used as our evaluation metrics. The results of all experiments are summarized in Table 2.

Model A: To begin with, we build our baseline model using the original design of the YOLOv5 algorithm, which has predefined anchors. This resulted in mAP (98.1%), F1-score (90%), [email protected] (98.1%) and inference speed (9.8ms). The original YOLOv5 setting is described in Section 2.3.

Model B and C: Model B is the first refinement with a positive effect on the YOLOV5 algorithm, and it involves replacing the predefined anchors with three auto-anchor generators. Auto-anchors is a technique introduced in YOLOv5 which helps the algorithm automatically learn the best anchor box for any given dataset. This boosted performance from 98.1% mAP to 98.3% mAP. Model C refinement is similar to model B, except that we used five auto-anchor generators. The model resulted in 97.8% mAP, which is slightly lower than model B and our baseline model. Models B and C are slightly slower than model A.

Model D and E: In these models, the refinements made were replacing the baseline YOLOv5 neck architecture, PANet with BiFPN, and experimenting with three and five auto-anchor generations. Both models resulted in mAP (97.5% and 97.7%) which are lower than our baseline model but achieve a higher F1-score when compared to our baseline model. Both models also achieved higher inference speeds when compared to our baseline model.

Model F and G: The refinements made in these models are similar to those made in models D and E except that we used a different neck architecture, FPN. While model F achieved a slightly lower mAP (98.0%) compared to our baseline model, model G achieved a slightly higher mAP value (98.2%) than our baseline model, which makes it the second refinement with a positive effect on the YOLO algorithm. Both models F and G achieve higher inference speed than our baseline model.

Model H and I: The refinements made in models H and I involved replacing the backbone of our baseline model with our light-weight backbone inspired by VGG and experimenting with three and five auto-anchor generation. Both models achieved the same mAP value 0.98% which is slightly lower than our baseline model but had a higher inference speed and F1-score when compared to our baseline model.

Model J and K: The refinements made to models J and K are similar to models H and I except that we replace the PANet neck architecture in models H and I with an FPN neck architecture, experimenting with three and five auto-anchor generations. While both models J and K achieved mAP values (97.8% and 97.6%) slightly lower than our baseline model, both models were faster than our baseline model and also achieved an F1-score higher than that of our baseline model.

In this section, we compare the performances of the top four models from our ablation studies in Table 2 with four other state-of-the-art object detection models that can be used for surgical instrument detection tasks. Table 3 summarizes the comparison results, whereas Table 4 displays the performance of all the different experiments conducted in terms of their mean average precision (mAP) at a 50% IOU threshold for each surgical instrument in our novel endoscope dataset. The results in Table 3 demonstrate that model B, which is the best-performing model from our ablation studies in Table 4, outperformed all our benchmark models except for YOLOv3-SPP, which achieved equal model performance in terms of the mAP value.

We trained the model using a 12 GB VRAM Nvidia GTX 1080 GPU. The training time was 3 hours on average.

Most deep-learning-based models are sensitive to hyperparameters, and the accuracy of the results depends heavily on the right choice of hyperparameter values. For training, we used the values presented in Table 6.

Fig. 18 shows a sample of the 3D detection results. We can see that the model detected two humans out of 4 in each image. This is due to the low number of training samples. Usually, 3D detection models are trained with ten times more samples than that used for the preliminary experiment. Therefore, the following training batch will contain at least five times the number used for this batch.