Cost-effective Instruction Learning for Pathology Vision and Language Analysis

The family of LLMs [1, 30, 31, 8] and visual language pre-training [26, 32] are two indispensable building blocks for vision-language foundation models. Visual language pre-training aligns the encoding space in vision and language. The visual features from the visual encoder can then be perceived by the LLMs. Examples of multi-modal foundation models include Flamingo [10], BLIP-2 [33], FROMAGe [11], mPLUG-Owl [12], and LLaVA [9]. In particular, Flamingo, BLIP-2, and FROMAGe all freeze LLMs and allow vision to adapt to language information through different modules, while mPLUG-Owl and LLaVA all train LLM to fuse visual and language information. Their success strongly benefits from leveraging large-scale natural image-text datasets. For instance, LLaVA leverages a GPT-based data generation to construct the instruction dataset. Despite the promise shown in general domains, these approaches face inherent challenges in the healthcare system. This is due to the medical complexity that significantly differ from generic domains, requiring models to capture specialized disease concepts [28, 29]. Therefore, extending these general models to the medical domain requires fine-tuning and the careful use of specific medical image-text pairs and question-answer data [1].

Building medical vision-language foundation models requires systematic efforts in both data curation and model inference in the healthcare system [1]. The large-scale resources from the literature (PMC-15M [22]), social media (PLIP [24] and Quilt-1M [23]), accessible textbooks [21] and private hospital data [25, 34] provide critical data support for domain-specific multi-modal models. Examples include PMC-LLaMA [20], LLaVA-Med [18], and Med-Flamingo [21]. LLaVA-Med is a representative work that aligns vision and language using 600K image-text pairs on the basis of LLaVA and fine-tunes the model using 60K dialogue-based instructions. Yet these instruction sets are generated by GPT-4 at a high cost. Similarly, in the pathological field, Quilt-LLaVA [19] uses Quilt-1M and GPT-4 to generate instruction data at a cost of up to $8,858. Towards a better utility of foundation models, instruction preparation and generation in the various forms of structure and scale require substantial research efforts. Meanwhile, during the fine-tuning stage, LLaVA-like methods [9] update the parameters of the LLM, resulting in a high training cost that can not be accommodated on consumer-grade GPUs. This pressing demand for high-performance training equipment becomes a daunting hurdle for researchers to extend general-purpose models into domain-specific vision-language models.

In this study, we propose a cost-effective learning framework for accurate pathology vision and language inference named as CLOVER (Fig. 1). To achieve fast domain tuning with low training resources, we adopt the BLIP-2 architecture [33] as a visual language pre-training using a lightweight trainable query transformer (Q-former), a frozen visual encoder, and a frozen LLM. We leverage the paired pathological image and text captures from the Quilt-1M [23] dataset to align the vision and language. For the instruction fine-tuning, we meticulously design prompts tailored for pathological question answering and use GPT-3.5 [7] for generating effective instruction dataset at low cost, referred as the “generation-based instructions”. The instructions generated by our PVLM-orient prompt show strong capabilities in the model tuning. Additionally, we construct instructions by matching template questions with the original text captions to enhance the model’s vision understanding ability, referred as the “template-based instructions”. Integrating these two types of instructions gives rise to the hybrid-form instructions, which remarkably enhance CLOVER’s conversational abilities in pathology. Using two benchmark datasets and one independent clinical dataset, we comprehensively validate the effectiveness of CLOVER and demonstrate its potential for assisting clinical pathological tasks by “talking to your pathology data” in resource-constrained settings.

Our study has made multifaceted contributions: First, we propose an effective PVLM training framework particularly at low computational resources and financial spending. Different from the LLaVA-like tuning (e.g. LLaVA-Med), we emphasize the use of BLIP-2 [33] to serve as an alternative choice for the cost-effective lightweight inference. We find that tuning the wide spectrum of LLM’s parameters is unnecessary for building a sufficiently usable PVLM. Second, the value of our designed PVLM-oriented prompt is pronounced on GPT-3.5 instead of the advanced GPT-4. Low-cost CLOVER model outperforms those models trained with LLM tuning on the instruction data generated by GPT-4 in multiple settings. We also recognize template-based instructions without relying on GPT can significantly supplement generation-based instructions. A combined use of these two type instructions can further enhance CLOVER’s understanding capabilities. Third, we confirm that CLOVER can be effectively developed with a limited scale of instructions towards a frugal and real-world application. This is evidenced by the appealing performance in fine-tuning and zero-shot experiments on two visual question answering datasets, as well as in the few-shot experiment on a real-world clinical dataset.

We systematically measure CLOVER’s visual question answering (VQA) performance using two public benchmark datasets, namely PathVQA [35] and QUILT-VQA [19], and one independent clinical data set (see Sec. Datasets). We compare CLOVER’s performance with state-of-the-art (SOTA) PVLMs trained with notable costs to observe the cost-effectiveness of CLOVER.

Comparison with SOTA PVLMs on PathVQA. From Table 1, CLOVER outperforms major competing methods, showing a remarkable improvement of 26.13% in accuracy at maximum for closed-ended questions. Our model performance is even approaching to LLaVA-Med (37 times more model parameters) with a minor accuracy difference of 1.83%. Note that LLaVA and LLaVA-Med achieve their results through extensive parameter tuning of LLM and training on a vast instruction dataset generated by GPT-4. To illustrate, our method involves training with a frozen LLM, adjusting only a small portion of parameters (about 1/37 of LLaVA-Med). Meanwhile, CLOVER only involves the use an entry-level GPT-3.5 with a smaller scale of instruction dataset (2/3 of LLaVA-Med). As seen in Table 1, our model also shows superior performance improvement in open-ended question scenarios. CLOVER’s performance is twice that of BLIP-2 and approaches five times that of LLaVA in open-ended question scenarios. Compared with BLIP-2, our model improves in both closed-ended and open-ended question-answering. This key finding suggests that vision-language models could evidently benefit from the high-quality pathology-sensitive instruction data towards a low-cost model development.

Comparison with SOTA PVLMs on QUILT-VQA. We validate CLOVER’s zero-shot generalization capability on the QUILT-VQA dataset. From Table 2, CLOVER achieves the leading performance in precision and F1-score. Our results are even close to LLaVA and LLaVA-Med in terms of recall. Due to the longer answer length of LLaVA and LLaVA-Med, these models are advantageous in the recall evaluation metric. The average length (word counts) of true answers is 18 and that number for LLaVA and LLaVA-Med are 38 and 36 respectively, both of which are higher than BLIP-2 and our model (4 and 22 respectively). In addition, our precision exceeds LLaVA’s by 9.77% and LLaVA-Med’s by 10.25%, while our F1 score also surpasses LLaVA’s by 6.73% and LLaVA-Med’s by 6.53%. In terms of a performance-cost ratio, CLOVER achieves the leading values for all metrics. Overall, these findings indicate the strength of CLOVER to retain the high-level performance while minimizing costs.

To gain insight into model outputs, we present the representative cases from VQA experiments involving comparisons with LLaVA, LLaVA-Med, BLIP-2, and our CLOVER on QUILT-VQA and LLaVA-Med-17K datasets. In Table 3 for results from QUILT-VQA, we observe that the output of LLaVA is unrelated to the actual content, reflecting general image contents such as “people”, “water”, and “lake”. While LLaVA-Med identifies a tissue sample and provides definitions for pathology and immunohistochemistry, it does not reveal the specific type contexts of tissues. BLIP-2 could only provide a simple answer without specific information. In contrast, CLOVER identifies that the image shows a compositional section of bone tissue and describes the composition of bone tissue and the structural shape, which serves as the basis for model reasoning. In addition, we perform evaluation on the LLaVA-Med-17K instruction. It is noted that results of LLaVA-Med tend to be overly optimistic due to the known overlap with its own training set [18]. A case is shown in Table 4. LLaVA’s answers still remain unrelated to pathology, while BLIP-2 could only answer generally without image details. Our model describes these inflammatory cells and provides a more detailed description of specific types of inflammatory cells. This is facilitated by our proposed PVLM-oriented prompts and generation-based instructions enriching the pathology knowledge distilled from GPT-3.5. Overall, CLOVER demonstrates its descriptive ability in reasonably responding to professional queries related to tissue characteristics and pathological outcomes.

To assess the efficiency of CLOVER in real-world applications with minimal human labour, we validate CLOVER’s few-shot learning capability under a challenging task of cancer detection (i.e., cancer/non-cancer tissue classification). We implement a K-shot learning testing scheme, meaning that our model can only use K WSIs (K = 1, 2) from each class for a model fine-tuning. We evaluate the model performance on the external validation dataset on two cancer tissues. Fig. 2 details the performance of CLOVER in the intestinal cancer detection. We report comparative results from BLIP-2 model as it is not feasible for a RTX 3090 to fine-tune resource-demanding LLaVA and LLaVA-Med. In addition, Fig. 3 presents results in the gastric cancer detection. Overall, both Fig. 2 and 3 show that CLOVER demonstrates a performance advantage over BLIP-2. Even under the extreme condition of 1-shot learning, CLOVER achieves a commendable accuracy of 75.49% in the intestinal classification task in Fig. 2. In the gastric cancer detection, CLOVER’s performance exhibits a rapid improvement with the increase of training samples. Starting from an accuracy of 75.04% in the 1-shot scenario, the accuracy swiftly increased to 87.91% with 2-shot learning in Fig. 3. This marked improvement underscores CLOVER’s potential on few-shot learning capabilities in cancer detection on the both stomach and intestine cancer tissues, especially under extreme sample conditions.

Comparison with SOTA instruction data. We conduct ablation studies on assessing the value of instruction data. We compare our instruction data with SOTA instruction datasets using the same model setting using BLIP-2 and use PathVQA as the final testing set. In detail, we use the generated 15k generation-based and 30k template-based instructions as our instruction data. In Table 5, the gains from using our instructions generated by GPT-3.5 based on noisy internet data are higher compared to using instructions generated by GPT-4 based on high-quality data. This finding can be attributed to the developed form of the instructions (generation-based and template-based instructions) and the quality of the instructions (generated through PVLM-oriented prompts). In particular, our costs are only $8, whereas the estimated API costs of LLaVA-Med are hundreds of times higher than our low-cost approach. When using the FlanT5XL model, our performance on closed-ended question-answering closely approaches that of the LLaVA-Med-Pathology-IM (LM-IM) [18], evidently outperforming LLaVA-Med-Pathology (LM) [18] and LM-IM on open-ended question-answering by 2.22% and 3.13% respectively. With the Vicuna 7B model, our results surpass the LM on both closed-ended and open-ended question-answering settings.

The effectiveness of template-based instruction data. We investigate the effectiveness of template-based instructions, aiming to provide the LLM with more direct and comprehensive language guidance. As shown in Table 6, results indicate that the addition of template-based instructions to the original instruction data is effective. Especially in FlanT5XL, there is a remarkable improvement in the open-question answering. We reason that template-based instructions can provide a complete answer about the image caption. Therefore it is possible to compensate for the information loss during the creation of generation-based instructions. Meanwhile, template-based instruction design enables the language model to capture visual contents and related contextual descriptions comprehensively.

The effectiveness of PVLM-oriented prompts. To explore whether using simplified question-answering prompts can achieve results comparable to our designed PVLM-oriented prompts, we again utilize GPT-3.5 to generate instruction data but modify the prompt to be a single sentence. The defined prompt describes that “Now you are a pathologist, and your task is to generate question-answer pairs based on the captions of the images.” (examples are offered in Table 7). The observable difference is that the answers of GPT-3.5 become shorter and lack additional knowledge. Results in Table 8 show that for the relatively weaker-performing FlanT5XL, low-quality prompts have a serious impact, yielding a recall of only 12.64% in open-question scenarios, even lower than BLIP-2’s 18.69%. This finding indicates that our PVLM-oriented prompts play a crucial role in enriching the instruction data with medical knowledge, thereby contributing to the performance improvement of the pathological vision-language model.

Impact of scales of Generation-based instructions. Due to the expensive cost in the use of GPT API and high training expense, our goal is to generate less instruction data while retaining a high performance of PVLM. We are particularly focused on measuring the model performance in terms of a low demand of instruction data. We conduct experiments using 5K, 10K, 15K, and 30K generation-based instruction data for the FlanT5XL and Vicuna models. As seen in Fig. 4, our method achieves high-quality results on small-scale instruction data. When comparing the cases of 15K and 30K instructions, the result difference is not statistically significant, indicating that large-scale instruction datasets may not always be instrumental. More critically, we recognize that the form of the instructions (whether they possess template-based instructions) and the quality of the instructions (whether they are generated with high quality through PVLM-oriented prompts) are more differential factors compared to the quantity of instruction data.

Homogeneity of instruction data. The quality of the generated instruction data could vary across different subsets. To validate the homogeneity and robustness of the instruction dataset, we randomly divide the 15K generation-based instruction data into three subsets of 5K each and use them for training separately. From results in Table 9, we observe a stable performance across subsets, which indicates a potential generalization ability and adaptability to variations within the dataset. This homogeneous ability is critical for measuring the robustness of instruction data since variations and noises can be introduced during data collection.

Impact of epochs at each training stages. We conduct ablation experiments using Vicuna as LLM in Stage 1 (alignment), Stage 2 (instruction tuning), and the fine-tuning stage. We use the same model parameter initialization of BLIP-2. The experimental results are shown in Fig. 5, indicating that sufficient training on pathological images in Stage 1 (Fig. 5(a) and (b)) and Stage 2 (Fig. 5(c) and (d)) is necessary for BLIP-2’s adaptation to the pathological domain. In the fine-tuning stage (Fig. 5(e) and (f)), increasing the number of epochs from 10 to 20 leads to a noticeable performance improvement in open-question tasks, as the model needs to adapt to the evaluation dataset according training dataset.

Generating large-scale instructions via GPT-4 incurs significant financial costs. In this study, we specialize in developing effective domain-specific instruction data generation at a low cost. The proposed framework includes (i) generation-based instructions with a specialized prompt for employing GPT-3.5, and (ii) template-based instructions without any additional financial cost. The construction of generation-based instructions involves a generation of question-answer (QA) from the captions using GPT and a PVLM-oriented prompt, while that of template-based instructions involves matching the captions as answer to a set of pre-designed template questions. Notably, the template-based QA dataset permits a comprehensive understanding of image contextual information, while the generation-based QAs emphasize the pathological knowledge distilled from GPT-3.5.

Generation-based Instruction. We meticulously design a prompt tailored for pathological question answering. We use GPT-3.5 [7] for instruction dataset generation enabling a low cost operation with prompt examples seen in Table 10. Given the high variance of prompt design, we have designed four desired principles of the prompt construction. First, we use GPT-3.5 to simulate a scenario where users (patient or doctor) and AI assistants (CLOVER model) conduct question and answering (QA). Since GPT-3.5 does not have access to images, QAs generated by GPT-3.5 are based on the textual description of the image. To reduce the over-reliance on textual descriptions, we emphasize avoiding minor information that can not be obtained from the image (e.g., reference dates or magnification ratios). Second, we focus on adding visual detailed information such as tissue structure, cell morphology, potential lesions, and locations. Third, the noise in the original textual description is avoided such as vocabularies related to context or narrator. Finally, we seek to generate answers of GPT-3.5 to exhibit the cautiousness, aligning with the medical field’s expectation for producing prudent answers. To enhance the quality of generated data, we additionally introduce few-shot examples in the prompts to inject more relevant information for in-context learning [48].

Template-based Instruction. The above process of generation-based instruction with GPT-3.5 is often based on partial captions derived from images. Generation-based instructions only capture a portion of the original information, preventing LLMs from fully capturing the visual and descriptive content. To address this challenge, we construct useful template-based instructions, providing the LLM with more structured and comprehensive language guidance, overcoming the shortcoming where visual information are often overlooked or incomplete in pathology. We use the same 17 descriptive statements as LLaVA [9], which instructs the model to intricately describe the content of images using various expressions. These statements are the questions and the corresponding answers from the original captions of the images. In detail, we merge multiple captions for the same image and filter out those with a word count less than 25. Subsequently, we randomly choose 30K image-text pairs for generating the template-based instruction. For each image-text pair, we randomly select only one description from 17 statements to form our template-based instructions. Note that this generation process requires no additional fee as the use of GPT is not involved.

Model training involves two main stages: (i) alignment of vision and language and (ii) supervised fine-tuning with instructions. In the first training stage, to align pathological images and text, we train BLIP-2 on the original image-text pairs directly obtained from Quilt-1M dataset [23]. BLIP-2 takes inputs in the form of a pair of image and text (caption or question). The visual encoder is utilized for extracting features from image and generating visual tokens $\textbf{V}=\{v_{1},v_{2},...,v_{n_{v}}\}$, where $n_{v}$ is the number of visual tokens. Next, the lightweight Q-former handles text and visual learnable queries, incorporating a self-attention [49] mechanism to share the transformer for both visual and textual components. A cross-attention mechanism is used to facilitate the interaction between visual tokens and visual queries. We simultaneously optimize the Q-former using image-text contrastive loss, image-grounded text generation loss, and image-text matching loss [32]. The image-text contrastive loss aims to maximize the similarity between the same image-text pair by comparing visual queries and text representations. The image-grounded text generation loss trains a text transformer to generate corresponding text given an image. Meanwhile, the image-text matching loss is a binary classification loss used to determine whether an image and text belong to the same pair.

In the second stage training, we introduce the customized instruction dataset for activating LLM and completing visual language question answering. In order to stimulate the domain speciality of LLM, we add a task-driven prompt before the input question as “Now that you are a pathologist, please answer the following questions based on the images”. We utilize a standard tokenization to obtain a sequence of text tokens, including prompt tokens $\textbf{P}=\{p_{1},p_{2},...,p_{n_{p}}\}$, question tokens $\textbf{Q}=\{q_{1},q_{2},...,q_{n_{q}}\}$, and answer tokens $\textbf{A}=\{a_{1},a_{2},...,a_{n_{a}}\}$, where $n_{q}$, $n_{p}$ and $n_{a}$ represent the token lengths of prompt, questions and answer. We feed the Q-former both the visual tokens $\textbf{V}=\{v_{1},v_{2},...,v_{n_{v}}\}$ and the text tokens, and then adjust a linear layer to map the dimension of image token into the the dimension of text. Our optimization objective is to fine-tune the learnable parameters $\theta$ of Q-former by maximizing the following likelihood:

where $p_{\theta}(a_{i}|P,Q,V,a_{1},a_{2},...,a_{i-1})$ is the probability to generate $a_{i}$ given $P$, $Q$, $V$, $a_{1}$, $a_{2}$, …, and $a_{i-1}$ under the model parameters $\theta$.

We choose EVA-ViT-G/14 [50] as the frozen visual encoder, utilizing the output from its second-to-last layer as the visual feature representation. This choice has been validated as a superior solution in BLIP-2 [33]. Regarding the LLM, we select the decoder-only Vicuna 7B [31] and encoder-decoder based FlanT5XL [30], which could best utilize the pre-trained BLIP-2 parameters. To enhance the training efficiency, we convert the parameters of visual encoder and LLM to FP16. In the first stage of model training, we conduct training for 20 epochs with a batch size of 36. In the second stage, training continues for 30 epochs, with a batch size of 2 for Vicuna and 8 for FlanT5XL. The remaining hyperparameter settings remain consistent with BLIP-2. We complete the two-stage training using Vicuna in 4 days with 4 RTX-3090 GPUs.

We conduct a comprehensive comparison with the standard BLIP-2 and other strong baseline approaches, including VL Encoder-Decoder [51], Q2ATransformer [52], M2I2 [3], LLaVA [9], and LLaVA-Med [18]. Since VL Encoder-Decoder, Q2ATransformer and M2I2 do not have zero-shot generalization ability, these models are not considered when evaluating the performance on QUILT-VQA.

LLaVA-Med-Pathology and LLaVA-Med-Pathology-IM [18] are the public instruction datasets that we use to compare with our proposed instruction dataset. LLaVA-Med-Pathology is a high-quality conversational instruction dataset focused on the pathological domain and is a subset of LLaVA-Med [18]. LLaVA-Med-Pathology-IM is another version of LLaVA-Med-Pathology, which adds inline content to the original image descriptions as supplementary textual information. These datasets are created by transforming parts of the PMC-15M dataset [22] into instruction datasets using GPT-4 based on specific prompts.

For PathVQA, we use the recall of the true answer that appears in the predicted answer in open-ended questions, and we report the accuracy in closed-ended questions. For QUILT-VQA, we find that under zero-shot learning, the answer lengths of different models vary greatly, and thus we report the recall, precision and F1-score. F1-score as an integration of recall and precision is used as a more comprehensive metric among the three. All above metrics are reported in percentage. To evaluate the cost-effectiveness of different models, we further compute the ratio of performance to the log of the number of parameters as a key metric.

This study is supported in part by Shanghai Artificial Intelligence Laboratory.

The authors declare no competing interest.

The Ethics Committee of Xinhua Hospital approved this study after data anonymization.

The QUILT-1M and QUILT-VQA can be accessed in https://quilt1m.github.io/. And PathVQA can be downloaded from https://paperswithcode.com/dataset/pathvqa. The CLOVER instruction data will be released upon acceptance. The clinical dataset from Xinhua Hospital is available upon request from the corresponding author due to the restriction of hospital.

The code and model will be released upon acceptance at https://github.com/JLINEkai/CLOVER.

KC, ML, MZ and SZ are major contributors to drafting and revising the manuscript for content and analyzing the data. FY, LM, XS, LW, XW, LZ, and ZW played major roles in the acquisition of data. KC, ML, MZ, FY, XS, LW, LM and XW substantially revised the manuscript. KC, ML, MZ and SZ conceptualize and design the study. ML, LZ and ZW interpret the data. All authors read and approved the final manuscript.