TongueSAM: An Universal Tongue Segmentation Model Based on SAM with Zero-Shot

Tongue segmentation refers to extract the tongue region from facial or oral images of patients and is a critical step in automated Traditional Chinese Medicine (TCM) tongue diagnosis[1]. The task of tongue segmentation presents certain difficulties due to the rich morphological and color features of the tongues, as well as the similarity in color between organs(lips,chin) and tongues. Several researches have employed deep neural network(DNN) based methods to address tongue segmentation problems and achieved remarkable experimental results [2, 3, 4, 5, 6]. However, these methods exhibit limited performance on datasets with distributions different from the training set[7]. This is largely due to the lack of large-scale tongue segmentation datasets, which results in pretrained models lacking applicability in changing scenarios. Meanwhile, it is hard to directly transfer models pretrained on other domain datasets to the task of tongue segmentation due to its unique nature. Consequently, many automated TCM tongue diagnosis methods still rely on color thresholding and contour-based segmentation methods for tongue segmentation[8, 9, 10], leaving room for improvement in segmentation accuracy. In a recent study by [7], a training framework called ”Iterative cross-domain tongue segmentation” was proposed. This framework aims to improve the performance of the model in the target domain by fine-tuning the model using samples from the target domain. However, this approach requires obtaining data from the target domain in advance, which can impose limitations in real-time. Therefore, this paper aim to develop a universal segmentation model that without additional training.

Recently, the ”Segmentation Anything Model” (SAM) has gained widespread attention[11] due to its versatile segmentation capabilities. Applying SAM to tongue segmentation can effectively leverage its strong zero-shot transfer capability. However, as an interactive segmentation model, the quality of segmentation results have related to manual prompts. Although SAM supports segmentation without prompts, prompt-free SAM performs lacks the advantages of segmentation accuracy and robustness. We discovered that by utilizing object detection methods, it is possible to generate box prompts that closely resemble the ones provided manually. This allows the SAM to achieve end-to-end high-quality segmentation without manual intervention. Based on the research background mentioned above, this paper proposes a tongue segmentation method called TongueSAM based on SAM, with the addition of an object detection based Prompt Generator. In the zero-shot scenario, TongueSAM demonstrates significantly performance compared to other tongue segmentation methods.

Tongue segmentation, as a medical image segmentation task, has received considerable attention. Over the past few years, numerous segmentation methods have utilized low-level features such as color and texture in tongues. In [12], a method combining dual elliptical deformable templates and active contour models was proposed, demonstrating improved stability and accuracy in tongue segmentation. Another approach proposed in [13] employed a combination of polar coordinate edge detectors and active contour models to achieve accurate segmentation of tongue contours. Additionally,[14] introduced a novel color tongue image segmentation algorithm based on the HSI model, which exhibited faster speed and superior performance.

With the rapid development of deep neural networks, increasing researches focused on leveraging deep semantic segmentation methods for tongue segmentation. [2] presented an end-to-end model called TongueNet, which employed multi-task learning for tongue segmentation. This model effectively leverages pixel-level prior information to achieve superior performance in tongue segmentation. Furthermore, [3] proposed a Dilated Encoder Network (DE-Net) that captured more advanced features and attained high-resolution outputs, outperforming contemporary methods in tongue segmentation. [4] introduced DeepTongue, significantly improved segmentation precision and efficiency for tongue segmentation without the need for preprocessing. In [5], an enhanced tongue segmentation method based on Deeplabv3+ and a loss function based on edge information was proposed, which improved the network’s ability to extract multi-scale and low-level information and encouraged the model to focus on separating tongue boundaries during training. Additionally, [6] presented a deep semantic enhancement network named DSE-Net, composed of lightweight feature extraction modules, efficient deep semantic enhancement modules, and decoders. Experimental results demonstrated that DSE-Net outperformed other mobile tongue segmentation methods on two different datasets.

The Segment Anything Model (SAM) has been proposed as a benchmark model for natural image segmentation [11], thanks to being trained on over 1.1 million natural images with over 1 billion ground truth segmentation masks. With the advent of SAM, researchers have explored its performance in various domains, with medical image segmentation garnering significant attention. In [15], SAM was successfully extended to medical image segmentation tasks, including adaptation of empirical benchmarks and methodologies, while discussing potential future directions of SAM in medical image segmentation. [16] explored SAM’s zero-shot performance in medical image segmentation by implementing eight different cueing strategies on six datasets from four imaging modalities. The study demonstrated that SAM’s zero-shot performance was comparable to, and in some cases even surpassed, current state-of-the-art methods. [17] designed SkinSAM, a model refined through training based on SAM, which exhibited outstanding segmentation performance in skin cancer segmentation. [18] reported quantitative and qualitative zero-shot segmentation results on nine medical image segmentation benchmarks. These benchmarks encompassed various imaging modalities and different application domains. Experimental results revealed that SAM’s zero-shot segmentation performance in medical images remained limited. [19] extensively evaluated SAM on 19 medical image datasets from multiple modalities and anatomical structures, drawing the conclusion that SAM exhibited impressive zero-shot segmentation performance on certain medical image datasets, but moderate to poor performance on others. Some recent studies suggest that although SAM has achieved remarkable results in natural image segmentation, there is still room for improvement in medical image segmentation[18, 19].

The proposed TongueSAM architecture as shown in the Fig. 1. TongueSAM consists primarily of two components: SAM and the Prompt Generator. For a given tongue image, TongueSAM first utilizes the pretrained Image Encoder in SAM for encoding. Meanwhile, the Prompt Generator generates bounding box prompt based on the tongue image. Finally, the image embedding and prompts are jointly fed into the Mask Decoder to generate the segmentation result. The entire segmentation process is end-to-end and does not require any additional manual prompts. The following sections will introduce different components of TongueSAM.

To thoroughly validate the effectiveness of TongueSAM, experiments were conducted on three datasets, the samples of these datasets are shown as Fig. 3. The first dataset, called TongueSet1, comprised 3192 tongue images gathered using TFDA-1 tongue diagnostic instruments in collaboration with Shanghai University of Traditional Chinese Medicine. Each patient’s tongue image was captured under standardized environmental and manual labeled using the labelme tool. The size of each tongue image were 400*400 pixels. This dataset, characterized by a relatively large amount of data and standardized capture conditions, served as the benchmark pretraining dataset for testing the model’s transferability.

The second dataset, called BioHit, was a widely-used public dataset for tongue segmentation[24]. BioHit was introduced by Harbin Institute of Technology and comprised 300 RGB tongue images, along with corresponding manual masks as ground truth. All tongue images were acquired in a semi-enclosed environment under stable lighting conditions, with the image size of 768*576 pixels. It was denoted as TongueSet2 in this study.

The third dataset was obtained from the webset[25]. It encompassed a diverse range of tongue images captured using various sources. A total of 1000 tongue images with varying sizes from this dataset were manual annotated using the labelme tool. This dataset effectively simulated tongue image segmentation tasks in real-world environments, enabling testing of the model’s robustness and adaptability.

In this study, we utilized three evaluation metrics to assess the performance of the model: mean Intersection over Union (mIoU), mean Pixel Accuracy (mPA), and Accuracy (Acc).

The hardware device cpu used in the experiment is Intel(R) Core(TM) i9-7940X, its memory is 16G, the gpu we used is GeForce RTX 2080 Ti. All the experiments were implemented using the PyTorch framework.

This paper presents TongueSAM, an universal tongue segmentation method based on the large-scale interactive segmentation model SAM. TongueSAM achieves superior segmentation results compared to other methods, particularly in zero-shot scenarios. TongueSAM demonstrates exceptional adaptability and robustness, enabling it to achieve optimal segmentation results even on challenging samples. In future work, we aim to enhance the efficiency of TongueSAM, enabling its flexible application in diverse scenarios.

This work was supported by Industry-University-Research Cooperation Project of Fujian Science and Technology Planning (No:2022H6012),Natural Science Foundation of Fujian Province of China (No.2021J011169, No.2022J011224), Science and Technology Research Project of Jiangxi Provincial Department of Education (No.GJJ2206003). Special thanks to my friends Geng Dong, Zhizhong Zhang, and his wife Xue Wang for their encouragement.