MoVL:Exploring Fusion Strategies for the Domain-Adaptive Application of Pretrained Models in Medical Imaging Tasks

In Natural Language Processing (NLP), the technique of prompting has been extensively utilized to stimulate and guide Pretrained Language Models (PLMs) in downstream tasks, which can be categorized into two types: hard prompts and soft prompts.

Hard Prompts, also known as templates, typically refer to predefined and fixed input patterns that are directly inserted into the model’s inputs during model training or inference. Petroni et al. (2019) effectively demonstrated the potential of hard prompts in extracting implicit knowledge from pre-trained models. Brown et al. (2020) revealed the important role of hard prompts in quickly adapting models to new tasks. Wei et al. (2022) utilised ”chain of thought” to enhance the model’s capability in logical reasoning and problem solving.

Soft Prompts, on the other hand, are those prompts that are dynamically, which do not change the parameters of the model, but rather influence the model’s behaviour by adjusting the distribution of inputs. Hambardzumyan et al. (2021) discussed that soft prompts can improve the flexiblity and effectiveness of pre-trained language models. Chen et al. (2024) demonstrated the effectiveness of continuous soft prompts in enhancing the model’s ability to perform knowledge-intensive tasks, whilst also highlighting the potential of automatically generated prompts in reducing manual intervention and increasing efficiency.

From the above research, it is evident that prompting techniques show immense potential in enhancing language model performance, knowledge extraction, and understanding model behavior.

As computer vision (CV) advancing, prompting techniques have expanded into visual tasks. While not yet widely applied, recent studies Zhou et al. (2022a); Gu et al. (2023) have demonstrated the unique potential of visual prompting techniques in enhancing model performance, interpreting model behavior, and facilitating cross-modal understanding.

A study Jia et al. (2022) demonstrated the effectiveness of visual prompt tuning (VPT) on a variety of image classification tasks. The approach achieves prompting of input visual features by adding a small number of learnable parameters before the transformer block. This approach adds trainable prompts only at the input of the model, avoiding the need to completely retrain the model and effectively reducing the training cost while maintaining the performance of the model.

Visual prompting Bahng et al. (2022), by adding a ring of learnable perturbation pixels around the input images, guides the model to better perform in specific visual downstream tasks. Activation Prompt Anonymous (2023) involves adding perturbations to the feature maps generated during the feature extraction process. This method enhances the training efficiency of visual prompts and offers a new perspective for understanding and improving the internal workings of visual models. Another research Chen et al. (2023), from the perspective of label mapping, explored and optimized the correspondence between visual prompts and labels to improve model classification accuracy, providing a deeper understanding of how models comprehend and process visual prompts.

A different approach does not superimpose visual prompts directly onto images but treats visual prompts as an additional and independent learnable component Wu et al. (2022). It enhances model accuracy and adaptability in complex visual tasks by appropriately scaling images to coordinate their pixel ratio with visual prompts. The paper also indicates that proper normalization can improve the performance of visual prompting.

Efficiently finetuning pretrained models for specific tasks is an attractive area of research in CV. Currently, despite full finetune, mainstream efficient finetuning techniques can be categorized into two main types: methods that change the model structure and those that do not. Methods that alter the model structure include Adapter Chen et al. (2022) and LoRA Hu et al. (2021). Full finetune is a traditional finetuning method that involves updating all parameters of the model. Adapter and LoRA are lightweight finetuning techniques that introduce a minimal number of additional parameters into the model. Without changing the original pretrained parameters of the model, only the newly introduced parameters are trained for finetuning. The distinction is that Adapter works in series with the backbone model, while LoRA operates in parallel.

As for techniques that do not change the model structure, the popular Linear Probe approach does not alter the model’s architecture but adds a linear classification head after the pretrained model. It adapts to new tasks by swiftly learning the weights of the linear layer. Prompt engineering refers to changing the distribution of input data to enhance model performance in downstream tasks.

The choice of an appropriate finetuning strategy depends on specific task requirements, resource availability, and the similarity between the model and the task. From the comprehensive learning of full finetune to the parameter economy of LoRA and Adapter, and the structural preservation of LP and VPT Jia et al. (2022), each method offers different trade-offs in practical applications.

In this paper, the primary concern on determining the most effective approach for leveraging natural pretrained models to achieve accurate classification of medical images. Input images $x_{T}$ are randomly selected from the target medical domain dataset $D_{T}$. Subsequently, we add VP to the input image. This process can be denoted as:

where $\hat{x_{T}}$ denotes the prompted images, $\delta$ denotes VP and $\oplus$ denotes different ways of adding VP, which can create a pixel patch at random or fix location, paddling around the images Bahng et al. (2022) or resize the input images first and then paddling around the images Wu et al. (2022). Next, we input $\hat{x_{T}}$ into the pretrained backbone model $f$, and used the classifier $c$ to get final result. In this process, we formulate the optimization object:

where $l$ denotes loss function, $D_{T_{Tr}}$ denotes training dataset of target medical domain, and $\theta_{S}$ denotes pretrained backbone model parameters.

For transfer learning with only LP, the input can be rewrited as

There is no visual prompt added to the input image and vanilla medical images will be sent to natural image encoder. We notice from $x_{T}$ to $\theta_{S}$ existing a gap from target to source domain. For example, ResNet He et al. (2016) is pretrained on imagenet Deng et al. (2009), which is a dataset of natural images. $\theta_{S}$ of ResNet is a set of particular parameters to extract the natural features. Despite medical images may contains some of the common features, but still maintains unique medical features.

For transfer learning with only VP, the formula can be rewrited as:

$c_{LM}$ denotes the classifier for label matching (LM). Random label matching (RLM) and frequency label matching (FLM) are usually used in VP label matching process. RLM appoint the output label randomly, without considering the relationship between the labels. FLM select labels according to the probability of the labels, but without considering the change of VP. Although there is already work Chen et al. (2023) researching on LM, the accuracy of VP is still hard to reach LP’s.

MoVL (Mixture of Visual Prompting and Linear Probe) is an designed algorithm containing training strategy and joint learning loss. The algorithm process is following:

We conducted the experiments with four datasets, inclulding HAM10000Tschandl et al. (2018), PathMNIST, BloodMNISTYang et al. (2023, 2021), and Camelyon17 Bandi et al. (2018). HAM10000 Tschandl et al. (2018) is a dataset of skin lesion images, which contains a total of 10,015 images of seven types of skin lesions from clinical settings: actinic keratoses, basal cell carcinoma, bening keratosis-like lesions, dermatofibroma, melanocytic nevi, vascular lesions, and melanoma.

PathMNIST and BloodMNIST are two tasks in MedMNIST for the classification of biomedical images.Yang et al. (2023, 2021). PathMNIST is a dataset collected from colorectal cancer histology slides, which contains 107,180. There are nine types of tissues in PathMNIST: adipose, background, debris, lymphocytes, mucus, smooth muscle, normal colon mucosa, cancer-associated stroma, colorectal adenocarscinoma epithelium. BloodMNIST contains images of normal cells without infection, hematologic or oncologic disease, and free of any pharmacologic treatment at the time of blood collection. It consists of a total of 17,092 images categorized into 8 classes, which are basophil, eosinophil, erythroblast, immature granulocytes, lymphocyte, monocyte, neutrophil, and platelet.

Camelyon17 Bandi et al. (2018) is a dataset designed to probe model’s capability on the condition of out-of-distribution (OOD). We use the version from WILDS benchmark Koh et al. (2021); Sagawa et al. (2022). The tissue patches are collected from different hospitals, then was divided into training, validation and test set, which is a classic circumstance of distribution shift. It contains 45K images of data with two categories, benign or tumor.

The official split is adopted for all datasets.

We conducted the experiments with ResNet He et al. (2016) and CLIP Radford et al. (2021) as backbone models. For each of the architecture, we select two different sizes to show the robustness of our method. For ResNet, we choose ResNet18 and ResNet50, which are pretrained on Imagenet-1K Deng et al. (2009). For CLIP, we choose ViT-B/32 and ViT-L/14 visual encoder.

We train the model with AdamW optimizer Loshchilov and Hutter (2017). Except for mix strategy of ViT-L/14, we use the learning rate of 0.01. Because we find that it is hard for mix strategy convergence with such big backbone model as the same epochs with other model settings, we modify the learning rate to 0.1 for all mix strategy experiments with ViT-L/14. We use cosine scheduler Loshchilov and Hutter (2016) to decrease the learning rate as epochs increasing and weight decay is set to 0. Other parameters are default for AdamW optimizer. Warmup steps is set to 10. For HAM10000, PathMNIST and BloodMNIST datasets, we use the batch size of 128. For Camelyon17 dataset, we use the batchsize of 256. For all experiments, we use the prompt method of paddle and the prompt size is set to 30 pixels. The input images are first resize to (224-2*30)*(224-2*30) and then adding VP, so the prompted input images are 224*224 size Wu et al. (2022). VP is initialized as all zeroes pixels with minimum $\epsilon$ = 0.001.

For only LP or mix training period, namely one stage training, we use 20 epochs totally. For two stage training, we train 10 epochs for each stage.

For model validation, we first choose full finetune as the upper benchmark on ID datasets, given its capability to update a larger number of parameters. We also choose LP and VP as two baseline finetuning methods. These methods do not change the original parameters or architecture of the backbone image encoder. For LP and VP, we adhered to the same settings mentioned in section 4.1. For full finetune, we still use learning rate 0.01 for ResNet but 0.0001 for CLIP. The text prompt for CLIP follows the work byRadford et al. (2021),specifically ’This is a photo of a { }’. For dataset Camelyon17, we use text labels ’benign tissue region’ and ’tissue tumor region’. For other datasets, we use the labels as described in section 4.1.

From Table 1, VP performs better using CLIP architecture. This may be caused by output label matching gap of frozen linear probe in ResNet. For text prompt of CLIP, calculating the similarity between text and image can remit the gap to some extent, but professional medical terms can still be awkward. LP performs relatively stable than VP. However, there is a great distance from full finetune, which may caused by medical input and natural image encoder gap.

Our proposed MoVL (average of 90.917%) can outperform than single LP (average of 88.304%) or VP (average of 81.758%) for most experiments, indicating that only one module involved can be restricted by its proverties. However, a mixture strategy can be complementary and improve the accuracy. When contrast with full finetune (average of 91.132%), there is potential for MoVL to reach accuracy of full finetune.

We notice that on OOD dataset Camelyon17, full finetune can get worse performance than other methods. This is consistent with prior work Kumar et al. (2022). There is a gap between ID and OOD data, and full finetune only fit with ID, but distort OOD features. However, MoVL can be less influenced by this distribution shift and take 5.178% lead.

We observe that for ViT-L/14, some resluts of MoVL are lower than ViT-B/32, which may be caused by more training epochs to get convergence for larger backbone model, and we use the same consistent epochs for different models.

We observe MoVL (average of 0.029M) use far less parameters compared with full finetune (average of 107.3M), with the sum parameters of LP and VP. Parameters of VP is the same because of the same prompt size and input image size, but LP will change with feature map channel and number of categories.

In this section, we do experiments with different training strategies: LP only, LP$\rightarrow$VP, LP$\rightarrow$mix and mix all time. We choose LP only as a baseline to analyze, for LP is the base of other strategies. We also use various backbone models to show the efficiency of our proposed method. Both ID and OOD datasets are used to show the validation experiment results.

From Table 2, we observe adding VP may not be outperform than train LP for the full time, which indicates that adding modules in arbitrary combination doesn’t guarantee enhanced accuracy.

Under most circumstances, use mix strategy can improve the accuracy of categorizing tasks, no matter after LP training or mix training from the beginning, which indicates validity of mix strategy. There is also a trend that with more epochs of LP and VP training together, the better performance of the result, which indicates that both input and output space trainable is helpful with the improvement of the accuracy.

We also do the strategy analysis on OOD dataset, Camelyon17. The conclusion of ID dataset can still work on OOD dataset, but with a discount on improvement. This can be explained by the distribution shift from training and testing dataset. Nonetheless, mix strategy can help with combining the benefits of LP and VP.

To prove the training strategy is efficient for different situations, we change the visual prompt method. Following the workflow of Bahng et al. (2022), we directly add VP onto the original input images, which may cover the border of the images. The results still shows that mix strategy can be more competitive than other training strategies.

From the experiments above, we have shown that mix strategy can be quite a competitive strategy combining LP and VP. In this section, we will interpret our designed joint learning loss function can make use of information from original output without prompt to improve the accuracy of the model.

We counted data from different models on different datasets and found that using the joint learning loss function can improve the accuracy of the models. This may because cross entropy loss can only get the information after adding VP, with only one forward propagation. However, as original images getting through image encoder and LP, then gaining the original output, which can still contains information. For example, an original dog image can be wrongly categorized as cat, but this information can be referred by prompted one, which is not cat. As the model classify the image correctly, it can still be a reference for prompted one to exceed. On considering such information, VP can reach and beyond reference output.

We found initialization of VP can greatly influence the accuracy. With a simple ’blank’ VP, which is all zeroes plus a minimum $\epsilon$ = 0.001 initialization, can be better than random initialization.