Resource-Efficient Medical Report Generation using Large Language Models

Contrastive language image pretraining (CLIP) [11] introduces a method for training a text and image-based encoder model on multimodal data using contrastive learning. This approach aims to bring similar data pairs closer together in the model’s projection space while pushing dissimilar pairs farther apart, thereby bridging the gap between different modalities such as text and image. As a result, closely related text and image samples exhibit high cosine similarity scores, while dissimilar pairs show lower scores.

We utilize a medical CLIP (i.e., MedCLIP [12]), to extract visual embeddings, which we term prefix embeddings for use in the medical report generation. MedCLIP is trained to capture detailed information present in chest X-ray images. When these visual embeddings are translated through a lightweight mapping network into the language model’s space, they provide valuable visual context. This enables the model to generate patient-specific radiology reports.

Large language models are a class of powerful neural networks that are trained on vast amounts of text data, enabling them to generate human-like text and perform a wide range of natural language processing tasks. To evaluate the performance of our method, we utilized a variety of LLM models. Specifically, we employed the pioneering GPT-2 [13] models as well as the more recent Qwen1.5 [14] model. Both GPT-2 and Qwen1.5 are LLMs, but they differ in their training datasets. The GPT-2 models were trained on the WebText dataset, which comprises 8 million web pages. The training objective for these models was next-word prediction, which is a common objective for autoregressive models. In contrast, Qwen1.5 is an LLM model that has demonstrated better performance on natural language generation tasks, even with a relatively smaller model size. By using both GPT-2 and Qwen1.5 LLMs in our experiments, we aimed to explore the performance of our method on different LLMs and understand how the choice of pre-trained LLM can impact the overall results for medical report generation. The LLMs in our framework are frozen and thus reduce the requirement of resources making them resource-efficient.

Prefix Tuning, as proposed by [15], involves adding task-specific vectors at the beginning of the input sequence of language models while keeping the model parameters fixed. This only optimizes the prefix while the LLM is frozen. The language model is conditioned on this prefix, along with other information, to generate the medical report. Since we are integrating the language model with vision, we also employ a small mapping network. This network’s role is to translate vision features from the medical domain-adapted vision model to the language space of the GPT-based model.

The mapping network is the only trainable part of our framework. Drawing inspiration from the [16] approach, we employ a compact transformer-based architecture to convert visual information from CLIP’s embedding space to the LLM embedding space. This Mapping Network has a small number of parameters and is trained to effectively translate from CLIP’s 512 dimensions to LLM’s 768 dimensions for GPT-2 based models and 1024 dimensions for the Qwen1.5 model. By integrating this mapping network, the LLM gains visual capabilities, enhancing its performance in generating reports from chest X-ray images. The visual embeddings from MedCLIP are converted into sequences of tokens based on the hyperparameter of clip length. A trainable prefix is also added before these tokens and these are then processed by the transformer. The prefix retrieves meaningful information from CLIP embedding through the multi-head attention and it learns to adjust the fixed LLM to the new data.

For our work, we utilized the publicly available dataset: MIMIC-CXR [17]. The MIMIC-CXR dataset contains 377,110 chest X-ray images and their corresponding free-text reports. We adopted the standard dataset split of MIMIC-CXR for our experiments. As a preprocessing step, we removed any special characters from the reports and converted all tokens to lowercase. The datasets include a variety of image views, such as frontal and lateral views. However, given the dominance of the anteroposterior (AP) and posteroanterior (PA) views, we focused our experiments solely on the unique AP and PA view image and report pairs which amounted to 243,334. The number of samples for train, val, and test sets is 237,972, 1,959, and 3,403 respectively.

Our method showed improved performance on the task of medical report generation. The GPT-2 based LLMs performed relatively lower while the Qwen1.5 LLM performed better showing improved performance on the NLG metrics [18] such as Bleu [19] scores as shown in TABLE I.

Despite the effectiveness of GPT-2 LLMs in various language tasks, including text generation, their performance on specific NLG metrics such as Bleu scores was not as strong as expected. This could be attributed to their architecture, which might not be optimized for certain types of NLG tasks, such as report generation from medical images.

In contrast, the Qwen1.5 LLM exhibited significantly improved performance on the NLG metric compared to the GPT-2 models. The Qwen1.5 model appears to be more effective for tasks requiring generation from multimodal inputs. This improved performance suggests that the Qwen1.5 model can generate more accurate and fluent text, especially in the context of medical report generation from chest X-ray images.

TABLE II highlights the effectiveness of Prefix Tuning as compared to Fine Tuning of the LLM. Prefix Tuning showed improved performance and is more resource-efficient as compared to the finetuning of the framework in which the LLM is also trained along with the mapping network. This shows that by leveraging the pretrained LLMs we can get better results on the task of medical report generation. This will enable more clinical automation as the LLMs get efficient in their sizes and become better in their performance in the future.

The Qwen1.5 LLM even outperformed past full transformer-based frameworks and other larger LLM-based solutions on Bleu metric. This is a noteworthy result, as past transformer-based and LLM-based frameworks were resource-intensive and were specifically designed for handling complex NLG tasks. The Qwen1.5 model’s ability to surpass these frameworks indicates its effectiveness in integrating visual information and generating high-quality text outputs while being resource-efficient, showing the strength of our framework for the medical report generation task.

The results presented in TABLE III demonstrate the potential of our proposed method for medical report generation as compared to full transformer-based and LLM-based frameworks. R2gen(base) [5] denotes the vanilla Transformer, with three layers, 8 heads, and 512 hidden units without other extensions and modifications while RadDialog-INS [6] denotes the Vicuna-7b [20] based framework trained on the Instruction dataset for medical report generation. Our approach consists of Qwen1.5-0.5B as the LLM. Notably, our approach, which consisted of 601 million total parameters including 101M of trainable parameters of the mapping network as compared to 72 million and an astounding 7 billion parameters of previous studies, was able to achieve competitive performance compared to the more resource-intensive and complex frameworks.

At inference time it would require approximately 1GB of VRAM to load the transformer-based models, 2GB to load our Qwen1.5-0.5B LLM-based framework, and a huge number of 25GB VRAM to load the Vicuna-7b LLM-based solutions to generate the medical reports [21]. This hinders the adoption of these solutions in the medical domain pertaining to their resource requirements. Our framework provides a resource-efficient alternative to these frameworks while achieving better performance.

A key strength of our method is its performance on the Bleu-1, Bleu-2, and Bleu-3 metrics, which measure the n-gram overlap between the generated text and reference reports. Our model achieved Bleu-1, Bleu-2, and Bleu-3 scores of 34.2, 20.0, and 13.0 respectively, outperforming the more sophisticated transformer-based and LLM-based models included in the comparison.

Our proposed framework represents an efficient approach to medical report generation by leveraging the generative abilities of LLMs efficiently through the use of prefix tuning demonstrating its superior performance on Natural Language Generation (NLG) metrics specifically the Bleu score compared to full transformer-based and larger LLM-based frameworks.