Progressive Vision-Language Prompt for Multi-Organ Multi-Class Cell Semantic Segmentation with Single Branch

The distribution of multi-class pathological cells provides crucial auxiliary diagnostic information for pathologists. A common approach for analyzing multi-class cell distribution is to divide the cell segmentation framework into multiple branches, such as performing object segmentation followed by classification [1, 44]. This approach helps eliminate background interference from original pathological images, leading to improved cell classification outcomes. Some research further adds branches to enhance contour or texture information of different cell types [19, 14]. For instance, Meta-MTL proposes a multi-task nuclei segmentation network with both contour detection and segmentation tasks, with a feature attention module to amplify shape information [16]. Similarly, AL-Net [52] introduces an attention-based learning network using multi-task learning strategy to enhance segmentation feature extraction by predicting nuclei boundaries. Beyond boundary detection, several studies focus on enhancing textual feature learning through semantic information analysis. For example, SMILE [33] leverages cell semantic segmentation and instantiation to generate a distance transformation map, improving the accuracy of multi-class cell instance segmentation. GSN-HVNET [53] employs an encoder-decoder framework to extract precise cell features for segmentation and classification tasks.

While multi-branch cell segmentation networks have shown strong performance in multi-class cell segmentation and classification, their increased computational complexity is a significant disadvantage. Therefore, designing a single-branch network that efficiently merges diverse features to achieve accurate multi-class cell segmentation remains an important research question.

Given the diverse structural and scalar details present across multiple cell types, multi-scale feature fusion is crucial for capturing information ranging from low-level textures and edges to high-level semantics and contextual details. Deep learning networks extensively used for image fusion are generally categorized into CNN-based and attention-based architectures. CNN-based methods leverage their architectural design to integrate features across various scales, e.g., FPN [15, 24] and skip connections [40, 49], etc. However, CNN-based feature fusion methods typically use static input features, which increases computational demands. Attention-based methods, on the other hand, provide nonlinear strategies for multi-scale feature fusion, effectively handling features of varying semantics and scales. These methods can adjust input features dynamically, leading to the proliferation of attention-based fusion networks [38, 43]. For example, MS-CAM [8] proposes a multi-scale channel attention module for feature fusion. HiFuse [22] introduces a three-branch hierarchical medical image classification network with a self-attention module to integrate local multi-scale features. In the field of medical image analysis, several attention-based multi-scale feature fusion networks have been proposed, including AF-Net[20], MAXFormer [27] and MSA-Net [41], etc. While these attention-based methods generally outperform CNN-based approaches in multi-scale feature integration, current cross-attention mechanisms still struggle to fully capture the complex relationships between features at different scales.

Most visual analysis research focuses on training a single-branch deep neural network using extensive annotated datasets, resulting in a laborious and time-consuming process [31, 50]. Recently, Vision-Language Models (VLMs) have gained significant attention for their ability to learn intricate correlations between visual and linguistic features using web-scale image-text pairs [51, 54]. Leveraging VLMs has had a substantial impact on computational pathology. To be more specific, VLMs have become particularly popular in computational pathology since the introduction of Contrastive Language-Image Pre-training (CLIP) [36]. For example, PLIP [21] introduces a novel approach to pathology by developing a language-image pre-training model capable of analyzing multimodal data. UNI [6] effectively adapts VLMs originally trained on natural images to various pathological tasks by leveraging large-scale pathological image-text pairs. CONCH [29] proposes a pioneering vision-language foundation model that employs contrastive learning on image-caption pairs, addressing several downstream tasks such as image analysis, text-to-image, and image-to-text retrieval.

While vision-language foundation models have shown promising results in various tasks, they primarily focus on organ-level analysis. However, subtle changes and interactions within cells can provide pathologists with more detailed structural and functional insights, allowing for the identification and examination of heterogeneity among cellular populations [17]. Therefore, further exploration of cell-level VLMs is a promising area of research.

Mathematically, given a WSI $\mathbf{X}_{w}\in\mathbb{R}^{W_{w}\times H_{w}\times 3}$, whose size is $W_{w}\times H_{w}$, a set of patches $\mathbf{X}\in\mathbb{R}^{W_{p}\times H_{p}\times 3}$ are cropped from the WSI. These images contain $C$ different cell types and are collected from multiple organs. Our goal is to predict the pixel-level label $\hat{\mathbf{Y}}=\{0,1,...,N\}\in\mathbb{R}^{W\times H}$, where $N$ is the number of cell types, based on both image and textual prompts.

To address this problem, we propose a novel framework for Multi-OrgaN multi-class multi-Cell segmentation with a single-brancH, named MONCH. As shown in Fig. 1, the proposed MONCH incorporates a coarse-to-fine visual feature extraction mechanism with a progressive vision language prompt decoder to efficiently fuse textural and multi-grained visual features. Specifically, we first generate cell attributes $\mathcal{T}$ utilizing GPT-4, which encompasses the background description and $N$ cell type descriptions. Among $N+1$ types, we describe each of them using three sentences. We then encode cell descriptions $\mathbf{S}$ and pathological cell image $\mathbf{X}_{p}$ using the image and text encoders inherited from a pre-trained VLM, thereby obtaining the textual feature $\mathbf{F}^{\mathcal{T}}$ and multi-grained image features $\mathbf{F_{X}}$. $\mathbf{F^{X}}$ is calculated as:

$$\small\mathbf{F^{X}}=\{\mathbf{F}^{c},\mathbf{F}^{m},\mathbf{F}^{f}\},$$

(1)

where $\mathbf{F}^{m}=\mathcal{G}_{VLM}(\mathbf{X},\mathbf{S})$, $\mathbf{F}^{c}=\mathcal{G}_{ds}(\mathbf{F}^{m})$, and $\mathbf{F}^{f}=\mathcal{G}_{us}(\mathbf{F}^{m})$. Here, $\mathcal{G}_{VLM}(\cdot)$ is the image encoder of the pre-trained VLM, $\mathcal{G}_{ds}(\cdot)$ is the downscale block, and $\mathcal{G}_{us}(\cdot)$ is the upscale block. The generated multi-grained features are middle-grained $\mathbf{F}^{m}\in\mathbb{R}^{B\times C\times W\times H}$, fine-grained $\mathbf{F}^{f}\in\mathbb{R}^{B\times C\times 2W\times 2H}$, and coarse-grained $\mathbf{F}^{c}\in\mathbb{R}^{B\times C\times W/2\times H/2}$, where $B$ is the batch size and $C$ is the channel size. Given the varying textures and scale information across different cell types, we introduce a coarse-to-fine feature extraction module to enhance the multi-grained features generated from the pre-trained image encoder.

Fine-grained features capture intricate details of pathological cells, which motivates the integration of a high-frequency information extraction module to enhance cell textual features. Coarse-grained features, while lacking in detailed information, retain essential cell distribution insights. To further enrich the structural and semantic content, we propose learning topological features. To preserve the original characteristics of the image, we incorporate a convolutional block to capture local and textual nuances.

The aforementioned comprehensive visual features, raging from coarse- to fine-grained, are then integrated with the embedded features derived from attribute prompts, resulting in enhanced image features that simultaneously capture visual and textural attributes of cells. To fully integrate these multimodal features, we design a progressive vision-language prompt decoder that iteratively adopts one feature set as the query for the subsequent feature set at a finer level. The coarse features are refined through fusing them with the multi-head self-attention mechanism, utilizing fine-grained features as the guiding query. Based on this progressive prompt learning approach, all features are effectively integrated, thereby facilitating superior performance in multi-class cell semantic segmentation.

To enhance the representation of pathological cells, we introduce a coarse-to-fine visual feature extraction module for multi-grained visual feature enhancement. Our approach utilizes text and image encoders from a pre-trained VLM to align textual and visual features. The initial medium-grained image feature, $\mathbf{F}^{m}$, is obtained from the pre-trained image encoder, capturing fundamental local details such as cell edges, textures, and essential structures We further process $\mathbf{F}^{m}$ by upscaling it to obtain a fine-grained feature $\mathbf{F}^{f}$ and downscaling it to derive a coarse-grained feature $\mathbf{F}^{c}$. These transformations provide multi-grained perspectives of the pathological cells. A convolutional module is then applied to adapt $\mathbf{F}^{m}$ for improved suitability to the new task, resulting in a convolutional feature $\mathbf{F}^{v}=\text{conv}(\mathbf{F}^{m})$. For fine-grained feature enhancement, we design a high-pass filter module that produces a refined high-frequency image feature $\mathbf{F}^{h}$. This process begins by transforming the fine-grained feature $\mathbf{F}^{f}$ at position $(x,y)$ into the frequency domain using a Fourier Transform $\mathcal{F}(\cdot)$:

$$\small\mathbf{F}^{f^{\prime}}(x,y)=\mathcal{F}(\mathbf{F}^{f}(x,y)).$$

(2)

We then extract the high-frequency feature by applying a high-pass filter $\mathcal{H}(\cdot)$, based on the resolution of $\mathbf{F}^{f}$:

$$\small\mathbf{F}^{f^{\prime\prime}}(x,y)=\mathcal{H}(\mathbf{F}^{f^{\prime}}(x,y)),$$

(3)

where $\mathbf{F}^{f^{\prime\prime}}$ represents the high-frequency component. The refined high-frequency feature $\mathbf{F}^{h}$ is obtained through an inverse Fourier transform $\mathcal{F}^{-1}(\cdot)$ and a residual operation:

$$\small\mathbf{F}^{h}(x,y)=\mathcal{F}^{-1}(\mathbf{F}^{f^{\prime\prime}}(x,y))+\mathbf{F}^{f}(x,y).$$

(4)

For the coarse-grained visual feature, we introduce a topological feature extraction module to better capture the intrinsic structural and shape information. Given the coarse-grained feature $\mathbf{F}^{c}$, we apply dilated KNN to calculate pixel-level pairwise distances, obtaining $k=9$ nearest neighbors for each point with dimensions $B\times C/2\times k\times k$. We first normalize $\mathbf{F}^{c}$ to reduce its channel dimension to get a blended feature with dimension of $B\times C/2\times W/2\times H/2$. For each image feature $\mathbf{F}^{c^{\prime}}$ in its spatial dimension, the pixel-level pairwise distance is calculated as:

$$\small\mathbf{D}=||{\mathbf{F}^{c^{\prime}}}||^{2},\hat{\mathbf{D}}=\mathbf{D}-2\times(\mathbf{F}^{c^{\prime}}\cdot{\mathbf{F}^{c^{\prime}}}^{\top})+\mathbf{D}^{\top},$$

(5)

where $\mathbf{D}$ is the pairwise distance of $\mathbf{F}^{c^{\prime}}$. The $k$ nearest neighbors for each spatial-level feature are then extracted based on the distance $\hat{\mathbf{D}}$, resulting in the topological structure $\mathbf{F}^{k}\in\mathbf{R}^{B\times C/2\times k\times\times k}$. The enhanced topological feature $\mathbf{F}^{t}$ is obtained through a residual connection:

$$\small\mathbf{F}^{t}=\mathbf{F}^{k}+\mathbf{F}^{c}.$$

(6)

To effectively harmonize the linguistic features with the multi-grained visual features, we introduce a progressive vision-language prompt decoder that generates a comprehensive representation for enhanced analysis, as illustrated in Fig. 2. In this framework, the higher-level features are sequentially used as queries for lower-level features to get a robust representation.

To reconcile knowledge differences between these modalities and extract additional textual information, we propose an attention mechanism that leverages the complementarity of visual and language features. Finer $query$ implies higher distinctiveness in attention mechanism, resulting in better differentiating between various parts of the input feature and capturing meaningful contextual information. Therefore, this attention based hierarchical approach ensures that the $query$ feature effectively captures the core requirements of the task, providing key details for refinement at each level. MONCH progressively sets the fine-grained feature as the $query$ for the coarser-grained feature from $\mathbf{F}^{h}$ to $\mathbf{F}^{\mathcal{T}}$, i.e. $\mathbf{F}^{h}\to\mathbf{F}^{v}\to\mathbf{F}^{t}\to\mathbf{F}^{\mathcal{T}}$. Specifically, using $\mathbf{F}^{h}$ as the $query$ for $\mathbf{F}^{v}$, the multi-head self-attention module is calculated as follows:

$\displaystyle\mathcal{G}_{ms}(\mathbf{F}^{h},\mathbf{F}^{v})=\mathtt{softmax}(\frac{g_{q}(\mathbf{F}^{h})\cdot{g_{k}(\mathbf{F}^{v})}^{\top}}{\sqrt{d_{k}}})\cdot g_{v}(\mathbf{F}^{v}),$

(7)

where $\mathcal{G}_{ms}(\cdot)$ represents the multi-head self-attention layer, $g_{q}(\cdot)$, $g_{k}(\cdot)$ and $g_{v}(\cdot)$ are projection functions, and $d_{k}$ is the dimension of the $key$. The subsequent multi-head self-attention modules are calculated in the same manner.

As shown in Fig. 2, after three-iteration attention architecture progressing from fine-grained to coarse-grained attention, The interactively learned features are ultimately merged with a blended feature, which is generated by fusing the three-scale features $\{\mathbf{F}^{c},\mathbf{F}^{m},\mathbf{F}^{f}\}$ obtained from the pre-trained image encoder using an FPN-like feature fusion block in Fig. 1. To enhance the detailed cell feature, we set the iteratively generated visual feature as the $query$ to this blended feature. Through this progressive prompt decoder strategy, we effectively harmonize multimodal and multi-granular features of pathological cells, ensuring accurate integration and representation of diverse characteristics. A final merge step is designed to enhance feature richness, resulting in a more robust representation.

The proposed MONCH builds upon a pre-trained VLM architecture, enhanced with several specifically designed adapters. Our network takes as input a combination of limited pathological cell descriptions and their associated images. To optimize learning, we freeze the text encoder while fine-tuning the image encoder, allowing better adaptation to pathological cell feature analysis. Following the progressive vision-language prompt decoder, we obtain a merged feature $\mathbf{F}^{m}$ that effectively integrates both textual and visual features. To further refine this representation, we introduce a forward propagation module that takes the textual feature $\mathbf{F}_{\mathcal{T}}$ and the merged feature $\mathbf{F}^{m}$ as inputs:

$$\small\mathbf{F}^{m}=\mathcal{G}_{conv}(\mathcal{G}_{reshape}(\mathcal{G}_{v}(\mathbf{F}^{m}),\mathcal{G}_{weight}(\mathcal{G}_{linear}(\mathbf{F}_{\mathcal{T}})))),$$

(8)

where $\mathcal{G}_{reshape}$ is operation that transforms a feature with a resolution of $B\times C\times W\times H$ into a matrix with dimensions $(1,B\times C,W,H)$. $\mathcal{G}_{v}$ represents a series of sequential convolution layers for processing visual features. $\mathcal{G}_{linear}$ is a linear transformation applied to the textual feature, and $\mathcal{G}_{weight}$ reshapes the weights for the subsequent convolution operation $\mathcal{G}_{conv}$.

For accurate multi-class cell semantic segmentation, we introduce a segmentation loss function $\mathcal{L}_{seg}$. We employ binary cross-entropy loss to facilitate precise segmentation of each cell class:

$$\small\mathcal{L}_{seg}(y,p)=-\frac{1}{N}\frac{1}{B}\sum_{n=1}^{N}\sum_{i=1}^{B}[y_{in}\log(p_{in})+(1-y_{in})\log(1-p_{in})],$$

(9)

where $y_{in}$ is the $i^{th}$ ground truth for cell class $n$, and $p_{in}$ is the $i^{th}$ predicted segmentation map for $n^{th}$ cell class.

Datasets. We adopt two well-known pathological cell segmentation datasets. PanNuke [11] consists of H&E-stained organ samples from 19 organs, with annotations for various cell types, including neoplastic, epithelial, inflammatory, connective, and dead cells. The dataset contains 189,744 annotated cells, each labeled with both class and shape information. It comprises 7,094 patches captured at 40x magnification, each with a resolution of 256$\times$256 pixels. As shown in Fig. 3, PanNuke is highly imbalanced in terms of both cell types and organ types, making it one of the most challenging datasets for cell semantic segmentation.

We evaluate MONCH against the state-of-the-art (SOTA) pathological cell segmentation methods that utilize visual inputs, including HoVer-Net [14], TSFD-Net [23], and CPP-Net [7]. As shown in Table 1, MONCH achieves superior performance across almost all evaluation metrics while maintaining a simpler single-branch architecture. Notably, our method demonstrates robust performance on PanNuke, as shown in Table 1, which exhibits significant class imbalance, particularly for epithelial and dead cell categories. The vision-language approach of MONCH proves especially effective in handling limited data scenarios compared to purely vision-based alternatives. A key advantage of MONCH lies in its architectural efficiency. While competing methods often employ multi-branch networks, our approach achieves comparable or superior results with a single branch. Its inference time is also comparable with those methods, as shown in Table 1. Given that MONCH leverages vision-language model fine-tuning, we also benchmark against SOTA fine-tuned medical image segmentation approaches, specifically Med-SA [42] and MA-SAM [4] in Table 1. This comparison provides insights into the effectiveness of our fine-tuning strategy within the medical imaging domain. MA-SAM is specifically designed for medical imaging through fine-tuning SAM, failing to represent pathological data with higher complexity. Med-SA performs much better due to its ability to represent multimodal medical images. Our method combines both textual and multi-grained visual information, resulting in its good characterization capabilities in diverse pathological cell samples. Fig. 4 illustrates that MONCH can segment the diverse cell dataset much better.

The multi-organ composition of PanNuke makes assessing organ-specific performance crucial for validating model robustness. Table 2 presents the organ-wise comparison between MONCH and SOTA methods. The results show that MONCH maintains consistent performance across diverse organ types while achieving comparable or superior metrics to existing approaches, indicating its strong generalization capability. Fig. 5 intuitively shows the F1 Score of the proposed MONCH against SOTA cell segmentation methods, demonstrating that MONCH can get the best-balanced evaluation results in all organ types.

We will provide feature maps of different enhancement modules and more visualized results in the supplementary.

We evaluate MONCH with three vision-language models (VLMs) as backbones: PLIP [21], CONCH [29], and CLIP [36]. Additionally, we replace the image encoder with three vision-based models, SAM [26] and UNI [6], as well as SAM2 [37], paired with a text encoder from pre-trained CLIP. Table 7 reports the F1 scores for each cell type, averaged across all organ types.

Among these, CLIP-based MONCH achieves the highest overall performance, followed by competitive results from SAM-based, CONCH-based, and SAM2-based implementations. PLIP-based MONCH shows a significantly lower performance, particularly for cell types with limited training samples. This gap can be attributed to PLIP’s architectural design, which was optimized for whole-slide image (WSI) classification. PLIP’s pre-trained image encoder generates feature maps with approximately half the resolution compared to the CLIP encoder, reducing the model’s ability to capture fine-grained cellular details. Furthermore, MONCH with UNI-based backbones demonstrates lower performance, likely due to the misalignment between text and image encoders.

The progressive prompt decoder block plays a crucial role in harmonizing linguistic and visual features. To demonstrate its effectiveness, we visualize the iteratively generated visual and multimodal features in Fig. 6. The high-frequency visual features, $\mathbf{F}^{h}$, effectively capture detailed information about pathological cells. By utilizing these fine-grained features as the $query$ for subsequent coarse features, the visual representations progressively refine, as shown by $\mathbf{F}^{h}\to\mathbf{F}^{v}\to\mathbf{F}^{t}$. These multi-grained visual features excel at preserving the semantic information of pathological cell textual features. Subsequently, the visual features are used as $query$ inputs to the text features, facilitating the capture of multimodal features that integrate linguistic and visual data. Finally, by merging visual and textual features, the multimodal features, $\mathbf{F}^{t}\to\mathbf{F}^{\mathcal{T}}$, effectively highlight features specific to different cell types.

Fig. 7 presents the cell segmentation results of MONCH compared to state-of-the-art cell segmentation methods and large-scale models. MONCH clearly outperforms competing approaches, leveraging both linguistic and visual features to effectively segment diverse cell data. Notably, MONCH delivers superior semantic segmentation performance compared to CPP-Net, the second-best performing method. By integrating linguistic information with visual learning, MONCH accurately identifies cell types, overcoming challenges faced by other methods. As shown in Fig. 7, cells marked with yellow arrows are misclassified by CPP-Net but are correctly segmented by MONCH. Furthermore, MONCH effectively segments rare instances, such as dead cells in PanNuke, despite limited data available. This underscores MONCH’s robustness in handling imbalanced datasets, making it a powerful solution for challenging segmentation tasks.