Feature-prompting GBMSeg: One-Shot Reference Guided Training-Free Prompt Engineering for Glomerular Basement Membrane Segmentation

To generate the prompt points of the GBM (or background) in the target image automatically, we need to build a patch-level correspondence matrix between the reference image $x_{r}$ and the target image $x_{t}$. Specifically, we first rely on the image encoder from DINOv2[12] to extract patch-level features for both $x_{t}$ and $x_{r}$ which are represented by $f_{t}$ and $f_{r}$, respectively. Patch-level correspondence matrix between the $f_{t}$ and $f_{r}$ is computed to discover the most similar regions of the GBM (or background) on the target image. We define a correspondence matrix $M_{s}$ as follows:

Here, $(M_{s})_{ij}$ represents the Euclidean distance between the $i$-th patch features $f_{r}^{i}$ from $f_{r}$ and the $j$-th patch features $f_{t}^{j}$ from $f_{t}$. A smaller value of $(M_{s})_{ij}$ indicates that the $j$-th patch is more similar to the $i$-th patch. Thus, we can obtain the patch from the target image that has the highest similarity to each patch in the reference image via $M_{s}$.

SAM serves as a robust foundational model for image segmentation, demonstrating the potential for zero-shot learning through carefully designed prompts. Similar to NLP, the efficacy of prompts profoundly influences SAM’s output results. Consequently, leveraging SAM’s impressive performance in class-agnostic segmentation, we transform the semantic segmentation task into an endeavor focused on automatically generating high-quality ptompt point. As shown in Fig. 2, we devise a series of prompt engineering strategies across the feature and physical space to systematically enhance the segmentation properties of the GBM, a procedural elucidation of which is expounded below.

Forward matching. Given a target image, our objective is to employ the $M_{s}$ for the automatic generation of prompt points by selecting the most similar patches from the reference image. We define the patches from $x_{r}$ whose centers are on the reference mask as $p_{r}^{p}$, and the patches whose centers are not on the reference mask as $p_{r}^{n}$. For each $x_{t}$, we can use $M_{s}$ in the feature space to obtain the most corresponding patch $p_{t}^{p}$ (or $p_{t}^{n}$) for each $p_{r}^{p}$ (or $p_{r}^{n}$), and set it center as a positive prompt point (or a negative prompt point).

Backward matching. Ideally, all patches from the GBM region (or background) in the reference image should be matched as $p_{t}^{p}$ (or $p_{t}^{n}$) via forward matching. However, in comparison to natural images, the background of TEM images is more complex, and the edges of the GBM are more blurred. Relying solely on forward matching will lead to imprecise and incomplete segmentation results attributed to numerous wrong prompt points. To eliminate incorrect prompt points, we perform backward matching on the prompt points $p_{t}^{p}$ (or $p_{t}^{n}$) obtained from forward matching to $x_{r}$ using $M_{s}$. For each $p_{t}^{p}$ (or $p_{t}^{n}$), if the most correspond patches are $p_{r}^{p}$ (or $p_{r}^{n}$), these prompt points are retained. Conversely, if the most corresponding patch is $p_{r}^{n}$ (or $p_{r}^{p}$), these points are excluded. In addition, these excluded points with correspondence exceeding the mean value would be employed as hard negative sampling prompt points in subsequent steps.

Exclusive sampling. Through forward and backward matching prompt engineering, we have substantiated the accuracy of generated prompt point in the feature space. To further optimize the prompt scheme, we employ exclusive sampling in the physical space. Specifically, by defining a hyperparameter $D_{ex}$, we exclude all negative prompt points within the range defined by each positive prompt point as the center of the circle and $D_{ex}$ as the search radius. This approach ensures the maximization of negative prompt points for errors in the physical space.

Sparse sampling. The rationale behind our point generation involves iterating over all $p_{r}^{p}$ and $p_{r}^{n}$ to generate prompt points, resulting in a substantial number of positive prompt points and negative prompt points for each $x_{t}$. However, the features of background or target exhibit significant diversity. Overemphasizing the same region can cause SAM to disregard other regions. For instance, when the features in the target image strongly correlate with those in the reference image, an abundance of prompt points may concentrate on that region, leading to excessive attention to that specific area and neglecting other regions. Therefore, we employ sparse sampling in the physical space to sparsify positive prompt points (or negative prompt points), aiming for a more balanced focus on the GBM region (or background) to enhance segmentation performance. Specifically, we introduce a hyperparameter $D_{sp}$ to conduct a search with all selected prompt points as the center of the circle and $D_{sp}$ as the search radius. If there exists a set of prompt points of the same class within the range, we compute the average distance between all prompt points and other classes of prompt points separately, retaining the prompt point with the largest distance. This sparsification of prompt points in physical space aims to enhance the subsequent segmentation performance of SAM.

Hard sampling. To minimize false positive segmentation of background regions similar to GBM regions, we employ a collection of hard negative prompt points. These prompt points are selected based on their correspondence value exceeding the mean value and are removed during the backward matching procedure. These hard negative sampling prompt points, which represent false positive prompt points, are then incorporated into the negative prompt points.

A total of 286 kidney biopsy samples from patients are collected at the Second Affiliated Hospital of Shanxi Medical University between 2020 and 2022. The dataset comprises a diverse range of chronic kidney diseases, such as membranous nephropathy, diabetic nephropathy, microscopic lesions, and others. From these kidney biopsy samples, a total of 2538 TEM images of glomeruli are digitalized using the JEM-1400 FLASH TEM, operating at 120 kV acceleration voltage, and magnification varied between 2500$\times$ and 15000$\times$. Ethical approval for data collection is obtained from the Ethics Committee on Human Research of the Second Affiliated Hospital of Shanxi Medical University (No. YX.026). All TEM images are labeled by three pathologists using Labelme[15] to annotate the GBM. Only one of these images served as the reference of GBMSeg, while the remaining are employed to evaluate the model’s performance.

Our experiments are carried out on a Linux server platform equipped with an NVIDIA Tesla V100. In the patch-level feature extraction stage, we utilize DINOv2 with a ViT-L/14 as the default image encoder. The SAM serves as the segmenter, incorporating ViT-H, ViT-L, and ViT-B, with their performances compared. Notably, our model is training-free, and to emphasize, any training is not employed for segmenting the GBM. During the testing phase, we employ the DSC to evaluate the segmentation performance of the proposed method.

We conduct ablation experiments to evaluate the segmentation performance of different model components. The quantitative segmentation results, compared to various prompt engineering schemes, are presented in Table 1. As the components in the automatic prompt engineering continue to be refined, multiple backbone architectures exhibit improved SAM segmentation performance.

The corresponding qualitative segmentation results are also displayed in Fig. 3. Specifically, forward matching alone leads to a significant number of erroneous and redundant prompt points, resulting in high false positives and false negatives. The introduction of backward matching reduces the occurrence of erroneous prompt points to a certain extent and enhances the recall of segmentation results. Exclusive sampling further corrects for false negative prompt points, contributing to a more comprehensive segmentation of the GBM. Meanwhile, sparse sampling helps refine the prompt points, improving segmentation accuracy by eliminating excessive prompt points in similar regions. Finally, hard sampling is employed to increase the number of challenging negative prompt points, negatively prompting the background area around the target and optimizing segmentation performance.

Additionally, we conducted hyperparameter selection experiments for exclusive sampling and sparse sampling. For exclusive sampling, the performance is optimal when $D_{ex}$ selects 25% of the image size. For sparse sampling, the best performance is achieved when $D_{sp}$ for positive sampling points is set to 0, and $D_{sp}$ for negative sampling points is set to 12.5% of the image size. This may be due to the fact that a dense distribution of negative prompts in regions with heterogeneous background features can degrade SAM segmentation performance. Conversely, in targets with homogeneous features, a dense positive prompt distribution does not degrade SAM segmentation performance and helps counteract the impact of some erroneous negative prompts.

To assess the effectiveness of the synergy between the proposed automatic prompt engineering and SAM, we compare GBMSeg with state-of-the-art few-shot or one-shot segmentation networks [11, 3], as well as recently introduced training-free segmentation frameworks [16, 20, 9]. The experimental results, presented in Table 2, demonstrate that GBMSeg not only outperforms the current state-of-the-art training-free methods but also excels over both the one-shot and few-shot training methods. Our model achieves optimal performance with minimal training resources by implementing an effective prompt scheme to assist the SAM in segmenting the GBM.