Segment Anything Model for Medical Images?

Two studies focused on testing SAM’s performance under the Everything mode in segmenting the camouflaged objects [97, 42]. Results showed that its performance was poor in these scenes, e.g., camouflaged animals that are visually hidden in their natural surroundings. The authors found that SAM failed to detect concealed defects in industrial scenes [42]. Ji et al. [43] explored three testing methods of the SAM (points, boxes, and everything) for various applications. Specifically, their tasks covered natural images (salient/camouflaged/transparent object segmentation, and shadow detection), agriculture (crop segmentation and pest and leaf disease monitoring), manufacturing (anomaly and surface defect detection), and remote sensing (building and road extraction). They concluded that, although SAM can achieve good performance in some scenarios, such as salient object segmentation and agricultural analysis, it produces poor results in other applications. They also validated that human prompts can effectively refine the segmentation results compared to the automatic Everything approach.

Ji et al. [42] assessed the SAM under the Everything mode in segmenting the lesion regions in various anatomical structures (e.g., brain, lung, and liver) and modalities (computerized tomography (CT) and magnetic resonance imaging (MRI)). The experimental results indicated that SAM was relatively proficient in segmenting organ regions with clear boundaries but may struggle to accurately identify amorphous lesion regions. Another study then evaluated the performance of SAM in some healthcare subfields (optical disc and cup, polyp, and skin lesion segmentation) using both automatic Everything and two manual Prompt (points and boxes) strategies [43]. The authors found that SAM required substantial human prior knowledge (i.e., points) to obtain relatively accurate results on these tasks. Otherwise, SAM resulted in wrong segmentation, especially when no prompts are given. In the brain extraction task using MRI, Mohapatra et al. [75] compared SAM with the brain extraction tool (BET) of the FMRIB Software Library. Quantitative results showed that the segmentation results of SAM were better than those of BET, demonstrating the potential of SAM for application in brain extraction tasks. Deng et al. [22] assessed the performance of SAM in digital pathology segmentation tasks, including tumor, non-tumor tissue and cell nuclei segmentation on whole-slide imaging. The results suggested that SAM delivers outstanding segmentation results for large connected objects. Nevertheless, it may not consistently achieve satisfactory performance for dense instance object segmentation, even with prompts of all the target boxes or 20 points per image. Zhou et al. [109] applied SAM to the polyp segmentation task using five benchmark datasets under the Everything setting. The results showed that although SAM can accurately segment the polyps in some cases, a large gap exists between SAM and the state-of-the-art methods. Additionally, Liu et al. [65] equipped the 3D Slicer software [79] with SAM to assist in the development, assessment, and utilization of SAM on medical images.

Most recently, several studies tested the SAM on $\geq$10 public MIS datasets or tasks [27, 73, 69, 103]. In [27], it was concluded that SAM’s zero-shot segmentation performance is considerably inferior to that of traditional deep learning-based methods. In Mazurowski et al. [73], the authors evaluated SAM’s performance using different numbers of point prompts. They observed that as the number of points increases, the performance of SAM converges. They also noticed that SAM’s performance is 1) overall moderate and 2) extremely unstable across different datasets and cases. Ma and Wang [69] validated that the original SAM may fail on lots of medical datasets with the mean DICE score of 58.52%. They then finetuned the SAM using medical images and found that the proposed MedSAM achieved a 22.51% improvement on DICE compared with the SAM. Wu et al. [103] adopted the Adapter technique to finetune the SAM and enhance its medical ability. Experiments validated that their proposed Medical SAM Adapter can outperform the state-of-the-art (SOTA) MIS methods (e.g., nnUnet [37]).

Although the above works investigated the performance of SAM in MIS, they had at least one of the following limitations:

1. 1.

   Small datasets. Previous studies have only evaluated SAM’s performance in modalities, such as MRI, CT, and digital pathology. They included a limited number of segmented objects. However, medical images contain multiple modalities and numerous anatomical structures or other objects requiring segmentation. This has limited the comprehensive analysis in the field of MIS of the above studies [42, 43, 75, 22, 109].

2. 2.

   Single SAM testing strategy. Most previous studies [42, 73, 109] evaluated SAM with limited or even only one type of testing mode/strategy. However, different medical objects often exhibit different characteristics and thus may have their own suitable modes for testing. The limited testing strategy may lead to inaccurate and incomplete analysis for SAM.

3. 3.

   Lack of comprehensive and in-depth assessments. Some of the existing works [43] only assessed the SAM via the visualization results provided by the online demo⁵⁵5https://segment-anything.com/demo. Additionally, some studies focused only on limited metrics (e.g., DICE or IOU) to evaluate the performance of SAM [22]. Most of the studies did not investigate SAM’s perception of medical objects. Thus, the correlation between SAM’s segmentation performance and medical objects’ attributes has not been conducted carefully [27, 73, 69].

The analysis of medical object perception is crucial. It can help the community better understand the factors that influence SAM segmentation performance (i.e., the ability to perceive medical objects), to better develop a new generation of general medical segmentation models.

In this report, we build a large medical image dataset named COSMOS 1050K, including $1050K$ images with 18 different modalities (see Fig. 1) and 84 objects (e.g., anatomical structures, lesions, cells, tools, etc.), to cover the entire body (see Fig. 2). This can help us comprehensively analyze and evaluate SAM’s performance on medical images. We then fully explore the different testing strategies of SAM and provide rich quantitative and qualitative experimental results to show SAM’s perception of medical objects. Finally, we deeply evaluate the correlation between SAM’s performance and the characteristics (e.g., complexity, contrast, and size) of the objects. We hope that this comprehensive report can provide the community with some insights into the future development of medical SAM.

Medical images cover a wide range of object types, such as brain organs and tumors [93, 92, 4, 95, 80], lungs and hearts [7, 39, 99, 88], abdomen [93, 107, 71, 45, 68], spine [87, 66, 77], cells [54, 70], and polyps [41, 32]. Table 1 presents a detailed list of the collected MIS datasets and Fig. 3 (a) shows the amount of each dataset after preprocessing. To be compatible with different modes of evaluating SAM, we employed the following exclusion criteria: 1) Exclude objects that are extremely small, such as the cochlea, shown in Fig. 4 (a), and ureter. This is due to the difficulty of automatically generating points or box prompts on extremely small objects. 2) Exclude objects in the 3D volume where their overall target became significantly separated as the slice was sequentially extracted, such as the intestine (seen in Fig. 4 (b)), mandible, and thyroid gland. We aim to avoid confusing the main object and generate unique boxes for each object. 3) Exclude objects with a relatively discrete overall structure, such as histopathological images of breast cancer (see Fig. 4 (c)), slices of lung trachea trees (see Fig. 4 (d)), renal arteries, and veins. Most of these objects are dispersed into multiple items in a 2D slice and embedded in other objects, resulting in a failure to sensibly employ SAM’s prompt mode on those objects to verify. According to the above criteria, COSMOS 1050K now comprises a total of 84 objects, with their numbers depicted in Fig. 3 (b). These objects are categorized only once in one image, without distinguishing locations or detailed divisions (e.g., ’Left Lung’ and ’Right Lung’ are grouped as ’Lung’, and various instruments are considered as ’Tool’). Further details can be found in the legend of Fig. 3. The histogram distributions of modalities and image resolutions are shown in Fig. 3 (c) and Fig. 3 (d), respectively. Given the significant variation of the same object across different modalities, including differences in gray distribution and texture features, we further divide them into 125 object-modality paired targets.

COSMOS 1050K contain different labels, modalities, formats, and shapes from one to another. Additionally, the original version of SAM only supported 2D input, and 2D format is fundamental and even a basic component for 3D/4D format. To standardize the data across different datasets, we applied the following preprocessing steps for each collected public dataset.

For 3D volumes, the whole procedure can be summarized as follows: 1) Extract slices along the main viewing plane owing to its higher resolution. In CT, it is usually a transverse plane, while in MRI, it may be a transverse plane, e.g., prostate, brain tumor, or a sagittal plane, e.g., spine and heart. 2) Retain slices with the sum of the pixel values of their labels greater than 50 for any 3D image and label volumes. This ensures that each slice has the corresponding correct label. 3) Normalize the extracted image intensities by min-max normalization: $I_{n}=255*(I-I_{min})/(I_{max}-I_{min})$, limiting the range to (0, 255). $I$ means the original extracted image, and $I_{n}$ represents the normalized image. $I_{min}$ and $I_{max}$ are the minimum and maximum intensity values of $I$. Simultaneously, we reset the pixel values of the mask according to the object’s category or location (e.g., left and right kidney have different pixel values). This is due to the fact that medical images may vary widely in the voxel or pixel values. Examples include MRI with an intensity range of (0, 800) and CT with an intensity range of (-2000, 2000), while other modalities may already be in the range of (0, 255) [9]. 4) Save images and labels in PNG format. For 4D data (N, W, H, D), we convert the data into N sets of 3D volume and then follow the 3D volume processing flow. Here, N represents the number of paired volumes within the 4D data.

For 2D images, the preprocessing is as follows: 1) Retain images with the sum of the pixel values of their labels greater than 50. 2) Reset the pixel value of labels according to the object category or location within the range of 1 to 255. For the CellSeg Challenge-NeurIPS 2022 [70], owing to the wide range of original label values (1-1600), we reconstruct each image and label into several sub-figures to ensure a uniform label range. 3) Convert the format of images and labels from BMP, JPG, TIF, etc. to PNG for achieving consistent data loading.

In total, COSMOS 1050K consists of 1,050,311 2D images or slices, with 1,003,809 slices originating from 8,653 3D volumes and 46,502 being standalone 2D images. Furthermore, the dataset incorporates 6,033,198 masks.

SAM diverges from traditional segmentation frameworks by introducing a novel promptable segmentation task, which is supported by a flexible prompting-enabled model architecture and vast and diverse sources of training data. A data engine was proposed to build a cyclical process that utilizes the model to facilitate data collection and subsequently leverages the newly collected data to enhance the model’s performance. Finally, SAM was trained on a massive dataset comprising over one billion masks from 11 million licensed 2D images.

As shown in Fig. 5, SAM mainly contains three components: an image encoder, a prompt encoder, and a mask decoder. The image encoder was, with the backbone of ViT, pre-trained by the masked autoencoder (MAE [26]) technique. It takes one image as input and outputs the image embedding for combinations with subsequent prompt encoding. The prompt encoder consists of dense (masks) and sparse (points, boxes, and text) branches. The dense branch encodes the mask prompts via a convolutional neural network (CNN). For the sparse one, the points and boxes can be represented by positional encoding [96], while the text is embedded by CLIP [85]. Finally, the mask decoder decodes all the embeddings and predicts the masks.

During testing, SAM supports both automatic Everything and manual Prompt modes. For the former, the users only have to input an image to SAM, and then all the predicted masks will be produced automatically. For the latter, the users manually provide some additional hints to SAM, including masks, boxes, points, and texts, to give SAM more information about the segmented objects. Details of these two modes are presented in the next sub-sections. It is noted that SAM can only find multiple targets in the image, without outputting their detailed categories (i.e., single-label: object or not).

In the official GitHub repository⁶⁶6https://github.com/facebookresearch/segment-anything, authors provide three types of pre-trained models with different backbone sizes, named ViT-B, ViT-L, and ViT-H. Their model parameters range from small to large. In Kirillov et al. [50], ViT-H shows substantial performance improvements over ViT-B. However, the former requires multiplied testing time due to its increased complexity.

While evaluating SAM in medical images, one study used six medical datasets and found that there is no obvious winner among ViT-B, ViT-L, and ViT-H [72]. In our study, we chose the smallest ViT-B (with 12 transformer layers and 91M parameters) and largest ViT-H (with 32 transformer layers and 636M parameters) as encoders to run all testing modes. We hope that a comprehensive evaluation of models of different sizes on large COSMOS 1050K can provide more inspirations for researchers.

In the Everything mode ($S_{1}$), SAM produces segmentation masks for all the potential objects in the whole image without any manual priors. The initial step of the process involves generating a grid of point prompts (i.e., grid sampling) that covers the entire image. Based on the uniformly sampled grid points, the prompt encoder will generate the point embedding and combine it with the image embedding. Then, the mask decoder will take the combination as input and output several potential masks for the whole image. Subsequently, a filtering mechanism is applied to remove duplicate and low-quality masks using the techniques of confidence score, stability evaluation based on threshold jitter, and non-maximal suppression (NMS).

In the prompt mode, SAM provides different types of prompts, including points, boxes, and texts. The point prompt covers both positive and negative points, which indicate the foreground and background of one object, respectively. The box prompt represents the spatial region of the object that needs to be segmented. Furthermore, the text prompt indicates one sentence (i.e., basic information in terms of position, color, size, etc.) to describe the object. Notably, the text prompt has not been released yet on the official GitHub repository${}^{\ref{fn:github}}$.

As shown in Fig. 5, our prompt mode contains five strategies, including one positive point ($S_{2}$), five positive points ($S_{3}$), five positive points with five negative points ($S_{4}$), one box ($S_{5}$), and one box with one positive point ($S_{6}$). We further established a unified rule for point selection to ensure randomness, repeatability, and accuracy. For the positive point selection, a) we first calculated the center of mass of the ground truth (GT) mask (red point in Fig. 5). b) If the center of mass was inside the GT mask, we took the center as the first positive point. c) Then, we directly flattened the GT mask to a one-dimensional vector and obtained the other positive points by adopting the uniform sampling method (green points in Fig. 5). d) If the center of mass was outside the GT mask, all the required positive points would be obtained by performing step c. For the negative point selection, we aimed to avoid selecting points that were too distant from the target region. Specifically, we first enlarged the bounding box of the GT by two times. The negative points were generated by even sampling in the non-GT region (yellow points in Fig. 5). Last, for the box selection, we directly adopted the bounding box of the GT mask without any additional operations. The above strategy can ensure the repeatability of the experiments. Besides, we tend to test the theoretical optimum performance of SAM via the selection of the center of mass and tight box. Since they may include the most representative features of the target. It is noted that SAM allows multiple prompts to be input into the network once. Thus, for a fair comparison, we test the one-round interaction performance of SAM under the above five prompt strategies ($S_{2}$-$S_{6}$).

We performed multiple tests ($n$) on an image using different strategies to obtain a final assessment (see Fig. 5). In SAM’s original code logic and design, the same encoding operation is required on one image $n$ times, which results in poor runtime efficiency in our multi-strategy test scenario. The situation becomes even worse when using high-resolution inputs. Based on this observation, we computed the embedding features of all input images in advance and saved them as intermediate files. Accordingly, the image embeddings could be reused to relieve the computational burden of the inference pipeline. Thereby, the overall efficiency of the SAM testing could be improved by nearly n times. The more testing strategies in SAM, the more time that could be saved. This can simply be extended to other multi-strategy testing scenarios of SAM.

SAM generated multiple binary masks for each input image, but not all of them included the corresponding object. Hence, we proposed a mask-matching mechanism to evaluate segmentation performance using SAM in each mode. Specifically, for an object (one of the foregrounds) in a given image, we calculated a set of dice scores {DICE_n}${}_{n=1}^{N}$ between N binary predicted masks {P_n}${}_{n=1}^{N}$ and the GT G. Then, the one with the highest dice score in the set was selected as the matched predicted mask P for subsequent segmentation evaluation. This process for obtaining P can be expressed as follows:

where N is the total number of predicted binary masks for an object in one image. The operations $(\cdot)$ and $max\{\}$ indicate computing a dice score between one predicted mask and the GT, while $max$ denotes obtaining the predicted mask with the highest dice score.

Code Implementation and Logits. In this study, we implemented the testing pipeline of SAM basically following the official GitHub repository${}^{\ref{fn:github}}$. For our multi-strategy testing scenario, we ran the SAM algorithm n times and extracted image embeddings n times. We observed that the process of image embedding extraction was time-consuming. However, since the same embedding could be reused for different testing strategies, we sought to optimize and accelerate this multi-extraction process. Thus, we refactored part of the code. For each test image, we only used the image encoder for feature extraction once and saved the embedded features as an npy file. When different testing strategies were applied, only the corresponding npy files needed to be loaded, which significantly improved the testing efficiency (approximately $\textit{n}\times$). Additionally, for the prompt testing, we calculated the required points and boxes once after image embedding and stored them as npz files. Thereby, all the prompt testing strategies could directly use the npz information without recalculation.

Package Versions and Features. We used multiple GPUs, including the NVIDIA GTX 2080Ti with 12GB, NVIDIA GTX 3090 with 24GB, and NVIDIA A40 with 48GB, for testing. We implemented the SAM with python (version 3.8.0), PyTorch (version 2.0.0) and torchvision (version 0.15.1). We further adopted: 1) torch.compile with the max-autotune mode to pack the model; 2) torch.cuda.amp to adaptively adjust the types of tensors to Float16 or Float32; 3) @torch.inference_mode to replace the common torch.no_grad to reduce GPU memory occupation and improve inference speed while keeping model calculation precision. We used Numpy package (version 1.24.2) to generate/calculate the prompts (boxes/points) for simulating the human interaction process (click, draw box, etc.) with the testing images. Besides, we used OpenCV (version 4.7.0.72) and Matplotlib (version 3.7.1) software packages for visualizing provided prompts and final results.

Testing Strategy Design. We designed different settings to fully explore the performance of SAM under various testing strategies. We first considered the Everything mode, which is the key feature of SAM. It can output all the predicted masks for a given image without any manual hints. Considering that MIS is an extremely challenging task, we progressively incorporated various manual prompts to aid in accurate segmentation. The manual prompts included 1) one positive point, 2) five positive points, 3) five positive and five negative points, 4) one box, and 5) one box and one positive point. The manually provided prompts can better guide the SAM to output accurate segmentation results. We summarize all the testing strategies and their abbreviations as:

• 1.

  $S_{1}$: automatic everything mode;

• 2.

  $S_{2}$: one positive point;

• 3.

  $S_{3}$: five positive points;

• 4.

  $S_{4}$: five positive and five negative points;

• 5.

  $S_{5}$: one box;

• 6.

  $S_{6}$: one box and one positive point.

To fully evaluate SAM’s segmentation performance, we used three common metrics, as shown below:

1. 1.

   DICE Coefficient (DICE, %) [23, 8]: A similarity measure to evaluate the overlap between the prediction and GT. Ranging from [0, 1], a higher value indicated a better performance of the model.

2. 2.

   Jaccard Similarity Coefficient (JAC, %) [38, 82]: Also called IOU, is used to measure the similarity between two masks. It is similar to DICE, but with a different calculation method. Specifically, for the predicted and GT masks, A and B, JAC calculates the intersection ($|A\cap B|$) over union ($|A\cup B|$). JAC ranges from 0 to 1, and higher values indicate better performance.

3. 3.

   Hausdorff Distance (HD, pixel) [18, 48]: A measure that evaluates the degree of similarity between two sets of points, which can reflect the distance between each point in the prediction to the points in the GT. It is more sensitive to the boundary than DICE.

In the following experiments, we will primarily analyze the performance in terms of DICE and HD. The results of another evaluation index for similarity measurement (i.e., JAC) are shown in the supplementary materials.

In this section, we tend to compare the segmentation performance between two models (ViT-B and ViT-H) under different strategies. It can be observed from Fig. 6 that, under Everything mode ($S_{1}$), ViT-H exceeds ViT-B with 7.47% at DICE and is 10.61 pixels lower than ViT-B on HD. For the one point prompt ($S_{2}$), ViT-H achieves slightly higher average performance than ViT-B. As the number of point prompts increases, the advantages of ViT-H will become more obvious. While for the resting strategies (box without/with one point, $S_{5}$-$S_{6}$), their performance are very close (differences in DICE: 0.37% and 0.06%). Compared to the point prompt, the box prompt contains more region information about the object. Thus, it can better guide SAM with different models to achieve better segmentation performance. DICE and HD performance for specific objects can be found in Table 2 and Table 3. We only present the detailed performance of 40 objects in the main text, and full results can be found in supplementary materials. Fig. 7 and Fig. 8 display the DICE distribution for the same objects evaluated under varying model sizes. It proves that ViT-H exhibits more stable performance with smaller standard deviations compared to ViT-B (see Kidney-CT, Prostate-MRI, and Polyp-Colonoscopy for typical examples). See Fig. 9 for prediction visualization. Additional comparison and visualization results can be found in the supplementary materials.

In this section, we tend to compare the segmentation performance among different strategies using different models (ViT-B and ViT-H). As shown in Fig. 6, we show the average DICE and HD performance of ViT-B and ViT-H under different strategies. For both ViT-B and ViT-H, the performance trends of different strategies are basically consistent. Everything ($S_{1}$) achieve the worst performance. For the point prompts ($S_{2}$-$S_{4}$), adding more points will bring stable performance improvement (ViT-B: DICE from 56.02% to 63.81%, ViT-H: DICE from 56.78% to 71.61%). SAM with a box prompt yields the best performance, while adding one point to the box will not bring obvious changes (ViT-B: DICE 0.49%$\downarrow$, ViT-H: DICE 0.06%$\downarrow$). Based on the experiments, we conclude that box prompts include more vital information compared to point prompts. Since the box actually tells the exact location of the target and also its potential intensity features given the limited region. However, points only represent the part features of the target, which can lead to confusion. Fig. 7, Fig. 8, Table 2 and Table 3 present specific segmentation accuracy results for partial targets under six testing strategies. See Fig. 9 for prediction visualization of the 5 manual prompting modes ($S_{2}$-$S_{6}$). For additional results, please refer to the supplementary materials.

In Fig. 10, we summarized SAM’s performance under box prompts for different modalities. SAM shows the highest average DICE performance on the X-ray modality for both models (ViT-B and ViT-H). In modalities including histopathology, colonoscopy, and DW MRI, SAM also achieved satisfactory DICE performance ($>$90%). Additionally, their standard deviations are relatively low. This proves that SAM performs well and stably in the objects of these modalities. There are 6 (ViT-B) and 7 (ViT-H) modalities with DICE performance between 80 and 90, but most of them have large standard deviations. SAM performs worse in the remaining modalities, and the results are also more unstable. Note that the same objects with different modalities may achieve slightly varying segmentation performance. See BrainTumor in Table 2 and 3, its performance ranges from 71.76% to 77.74% in DICE and 13.02 pixels to 15.35 pixels in HD. Although segmentation performance can be affected by various factors, we hope that Fig. 10 could provide basic guidance for researchers on how to use SAM appropriately for different modalities.

Inference time is an important factor in evaluating a model. In Table 4, we reported the average inference time in terms of embedding generation, prompt encoding, and mask decoding. All the testing is performed on one NVIDIA GTX 3090 GPU with 24G memory. Testing time may be influenced by lots of factors, including the image size (tiny difference in pre-processing time, i.e., upsampling the images with different sizes to 1024$\times$1024), and the number of targets (processing the prompts for each target in serial). Thus, we conducted a fair comparison by limiting the image size to 256$\times$256 and setting the number of targets to 1. It can be observed that ViT-H takes nearly four times the embedding time of ViT-B. Prompt encoding and mask decoding for Everything ($S_{1}$) is extremely time-consuming, as it requires processing hundreds of points sampled from the whole image, including heavy post-processing using NMS, etc. While for the manual prompt encoding and mask decoding ($S_{2}$-$S_{6}$), the reference times of different models and strategies are similar and less than 0.01s. We believe that the evaluation time of SAM in manual mode can meet the needs of real-time use.

As described above, in the Everything mode, a grid of point prompts (m$\times$m) will be generated. In default, m is set to 32. The number of points will have an impact on the final segmentation performance. Especially for images that have multiple targets with different sizes, improper parameter designs will lead to imperfect segmentation with some objects being unprompted. As shown in Table 5, we tested four datasets with multiple objects on one image. The results show that in these four datasets, as the number of points increases from $8^{2}$ to $256^{2}$, the DICE also increases gradually. Fig. 11 also shows that more points will bring more potential objects (shown in different colors). Additionally, too many points make SAM split an object into several pieces, destroying the integrity of the object. Increasing the number of points can also lead to a significant increase in test time. Hence, it is a trade-off between segmentation performance and test efficiency.

To verify the factors that affect the segmentation performance of SAM, we recorded the size, aspect ratio, intensity difference between foreground and background, modality, and boundary complexity of 191,779 anatomical structures. By analyzing these factors, we aim to better understand the correlation between anatomical structure characteristics and SAM’s segmentation performance, and further provide some useful insights into the development of medical SAM.

The size of the anatomical structure was computed as the pixel-level area of the corresponding mask. To determine the aspect ratio of a mask, we need to calculate the ratio (ranging from 0 to 1) between the short and long sides of its bounding box. The intensity difference was defined as the variation in mean intensity values between the structure and the surrounding area within an enlarged bounding box, excluding the structure itself. Specifically, to accommodate the varied dimensions of targets, we dynamically expanded the box outward by a preset ratio of $0.1$, instead of using fixed pixel values (e.g., extending by 10 pixels). Additionally, the modality of each anatomical structure was mapped to numerical values. Furthermore, we introduced Elliptical Fourier Descriptors (EFD) [51] to describe boundary complexity.

EFD encodes the contour of a mask into a Fourier series that represents different frequency components. As the Fourier order (FO) increases, the contour decoded from the Fourier series gets closer to the original contour (see Fig. 12), and the decoding process can be described as Eq. 2.

where $(x_{N}(t),y_{N}(t))$ is the coordinates of any point on the contour, $N$ is the number of Fourier series expansions, and $t\in[0,T]$ denotes the different sampling locations. $(L_{x},L_{y})$ indicates the coordinates of the contour’s center point, $(a_{n},b_{n})$ denotes the parameter obtained by Fourier coding of the x-coordinates, and $(c_{n},d_{n})$ denotes the encoding result in the y-direction. We can roughly estimate the complexity of the object boundary in terms of the FO. Specifically, the order is defined as the required number of accumulations when the contour from the decoded Fourier series reaches a certain degree of overlap (we use DICE to represent overlap) with the original contour. However, when using this approach as a quantitative measure, it is especially important to set an appropriate DICE threshold.

A low threshold cannot accurately distinguish the difference in complexity among various object boundaries. If the threshold is too high, the EFD may fail to fit the complex contours as required and get into an infinite calculation. Thus, we optimized the representation of FO to avoid the EFD program from getting stuck in endless accumulation (see cumulative terms in Eq. 2).

For different structures, we increased the FO from 1 and calculated the DICE between the decoded contour and the original contour at each step. Then we set two ways to end the process: 1) $DICE>97.0\%$; 2) the difference in the DICE between order $F_{(a-1)}$ and order $F_{(a)}$ is less than $0.1\%$. Consequently, we record the FO ($F_{(a)}$) and DICE after termination. Finally, we take $F_{final}=F_{a}+n\times 100\times(1-DICE),n=2$ as the final optimized FO.

We conducted an analysis of the partial correlation between the five attributes of the target objects mentioned above and the DICE score using Spearman’s rank partial correlation coefficient under different testing strategies. The statistical results are displayed in Table 6, while Fig. 13 illustrates the scatterplot for the $S_{5}$ strategies. Across most testing strategies, we observed that the DICE score exhibited a moderate correlation ($0.4\leq\rho<0.7$) with the FO and intensity difference, a weak correlation ($0.2\leq\rho<0.4$) with the size, and no correlation with the modality and aspect ratio. Thus, SAM can consistently segment medical targets with different modality and aspect ratios. The performance of SAM under box prompt might be affected by the size of anatomical structures. Furthermore, the performance of SAM tends to be poor under all testing strategies when dealing with objects characterized by complex boundaries or low contrast. To confirm these findings, we divided DICE calculated under $S_{5B}$ into ten levels on average (e.g., level-1 means DICE (%) belongs to (0,10]) and visualized the FO boxplot for different DICE levels in Fig. 14. The figure illustrates that as the DICE level increases, the FO distribution of the structures gradually shifts to a range with smaller values. Additionally, in Fig. 15, we presented visualizations of anatomical structures with various FO ranges. These visualizations reveal that the DICE score of anatomical structures tends to decrease as the FO increases. It further implies that shape and boundary complexity may have an impact on SAM’s segmentation performance.

In this section, we discuss whether the SAM can help doctors improve annotation time and quality. We randomly sampled 100 images with average DICE performance from COSMOS 1050K, to construct an evaluation subset comprising 55 objects and 620 masks across 9 modalities, including instances of the same object in different modalities. We then invited three doctors with 10 years of experience to evaluate whether SAM’s prediction under box prompts could improve annotation speed and quality. They were given tasks including 1) annotating all objects in the evaluation subset from scratch, 2) adjusting object labels based on SAM’s predictions, and 3) recording the time for both tasks. To evaluate annotation quality, we utilize the Human Correction Efforts (HCE) index [83], which estimates the human effort required to correct inaccurate predictions to meet specific accuracy (i.e., GT masks) requirements in real-world applications. The lower HCE index represents that the mask (annotation of human with/without SAM) is closer to GT, i.e., annotation is of higher quality. As shown in Table 7, with SAM’s help, it can attain higher annotation quality (HCE: 0.27$\downarrow$) and boost annotation speed by approximately 25%. Specifically, $\sim$1.31 minutes can be saved for annotating one image, and $\sim$0.2 minutes for one object (since one image contains $\sim$6.2 objects in the above task). The greater the number of anatomical structures that need to be labeled, the more obvious the advantage of SAM’s efficiency will be.

In the previous experiments, we fixed the box and point selection strategy for the repeatability of the experiments. We tested the theoretical optimum performance of SAM via the selection of the center of mass and tight box because they may include the most representative features of the target. However, it is not practical to click the exact center or draw the exact box of each object to evaluate SAM. Thus, we added different levels of randomness to the centers and boxes to simulate real-life human operation [35]. Besides, we believe that this can help us better discuss the robustness of SAM.

Specifically, we enlarge/move the boxes/points randomly in 0-10, 10-20, and 20-30 pixels. In Table 8, the random experiments (Random 1-3) were conducted three times, and the average results were calculated (Mean). DICE drop represents the average declining DICE value compared to the original results without shifting. For $S_{2}$ (single point), the DICE performance dropped by 2.67%, 7.38%, and 14.62% as the shifting level increased. With the increase in the number of point prompts ($S_{3}$ and $S_{4}$), the decline of DICE could be alleviated, and the stability of the model could be improved. SAM was severely affected by box offsets ($S_{5}$, 24.11% decrease in performance for shifts of 20-30 pixels), while this impact was even more pronounced when adding one point to the box ($S_{6}$, with a decrease of 29.93%).

In the previous sections, we input all the prompts to the SAM’s prompt encoder once for a fair comparison of its one-round performance. To mimic the real-life interactive segmentation procedures, we performed multi-round SAM. The point selection strategy is similar to the common interactive methods. Specifically, SAM first clicks on the center of the target, then the remaining clicks are based on false negative (FN) and false positive (FP) regions. We then compared SAM with two different strong interaction segmentation approaches, i.e., FocalClick [14] and SimpleClick [63]. Both of them are pretrained on the same number of images as SAM was.

We selected 10 typical organs/tumors, covering various modalities, shapes, sizes, and intensity distributions. Experimental results are shown in Fig. 16. Based on the DICE results, our conclusion is that: 1) SAM outperformed FocalClick and SimpleClick in the first interaction with a single point; 2) As iterations progressed, SAM’s performance increased slowly, or even declined, while the performance of the interactive method could be improved steadily; 3) Using 10 points, SAM performed worse than the interactive methods. Similar results can be found in the recently published MedIA paper [73]. We consider that the current SAM’s point-based multi-round iteration capability is weak on medical images. Future work should optimize iterative training strategies when training SAM or finetune it to enhance its ability to iterate multiple rounds [15].

SAM’s weak ability perception on most medical images/tasks is mainly due to the lack of training data. The training dataset of SAM, i.e., SA-1B⁷⁷7https://ai.meta.com/datasets/segment-anything/ contains 11 million photos, including natural locations, objects, and scenes, but without any medical images. Natural images are commonly different from medical images because they have color-encoding, relatively clear definitions and boundaries of objects, easier-to-distinguish foreground (objects) and background (non-objects), and relatively balanced size. However, most medical images are gray-scale, with unclear and complex object boundaries, similar back- and fore-ground, and wide-range image size (especially containing some very small objects).

Thus, we finetuned SAM using part of the COSMOS 1050K to improve SAM’s perception of medical objects. Specifically, we considered 45 common and typical objects for finetuning the SAM. Inspired by Ma and Wang [69], we only considered finetuning the SAM using box prompts. We fixed the image encoder to minimize computation costs and also kept the prompt encoder frozen because of its powerful capacity for encoding box positional information. Thus, only the parameters in the mask decoder were adjusted during finetuning. We set the total epoch as 20, and the learning rate and batch size are 1e-4 and 2.

The results demonstrate a general improvement in segmentation performance after finetuning for both ViT-B and ViT-H models, as shown in Fig. 17 and Fig. 18. Fig. 17 illustrates a shift towards higher DICE values across different correlating factors, indicating improved overall performance. Specifically, for ViT-B, 32 out of 45 objects exhibit performance enhancement, while ViT-H shows improvement in 37 out of 45 objects. This can prove the strong learning ability of ViT-H, because it has almost 7 times the parameters as ViT-B (636M vs. 91M). The performance of objects with small numbers, RGB color coding, etc., decreases. This reminds us that we may need to make more careful designs for task-specific finetuning. We will release all the finetuned models and source codes.