¹¹institutetext: ^†Graduate School of Information Science and Technology, Hokkaido University, Japan
^††Faculty of Information Science and Technology, Hokkaido University, Japan
E-mail: {zhu, togo, ogawa, mhaseyama}@lmd.ist.hokudai.ac.jp

Which Client is Reliable?: A Reliable and Personalized Prompt-based Federated Learning for Medical Image Question Answering

He Zhu^† Ren Togo^†† Takahiro Ogawa^†† and Miki Haseyama^††

Abstract

Conventional medical artificial intelligence (AI) models face barriers in clinical application and ethical issues owing to their inability to handle the privacy-sensitive characteristics of medical data. We present a novel personalized federated learning (pFL) method for medical visual question answering (VQA) models, addressing privacy reliability challenges in the medical domain. Our method introduces learnable prompts into a Transformer architecture to efficiently train it on diverse medical datasets without massive computational costs. Then we introduce a reliable client VQA model that incorporates Dempster-Shafer evidence theory to quantify uncertainty in predictions, enhancing the model’s reliability. Furthermore, we propose a novel inter-client communication mechanism that uses maximum likelihood estimation to balance accuracy and uncertainty, fostering efficient integration of insights across clients. The code will be available soon.

Keywords:

Medical VQA personalized federated learning large-scale model.

1 Introduction

Data heterogeneity and privacy protection [1] are key challenges in the application of deep learning in the healthcare field. Federated Learning (FL) [12], which allows for learning without centralizing data, is considered a promising solution to this issue. FL is a learning approach that enables the training of distributed models by sharing parameters instead of data. In the typical problem setting of FL in the medical field [24], a model is locally trained in each hospital using patient data and sends only the learned model parameters to a central server. The central server aggregates these parameters to create an improved model. This approach allows for improving the model using data from multiple hospitals while protecting patient privacy. In recent years, research [14, 17] has been conducted to introduce FL to medical tasks, addressing privacy and data heterogeneity issues. Many studies in medical FL focus on simple tasks such as segmentation [20] and classification [5, 28, 21], using predefined input and output formats. Also, most of these studies aim to construct highly versatile models by integrating information from multiple client models. However, the medical field often requires models that are personalized for specific targets rather than those with broad generalization abilities. It can be clinically more beneficial to fine-tune a model using insights from other models trained on vastly different data rather than relying solely on a single generalized model. This is especially important in clinical practice because the same medical images or symptoms often can be interpreted differently for each patient, highlighting the need for personalized models. Hence, FL remains room in the current medical field to tackle more challenging tasks and problem settings.

Medical visual question answering (VQA) [9] is a multimodal task capable of handling diverse questions and answers and is considered more challenging than tasks such as segmentation or classification. Implementing a VQA system enables the provision of specialized advice to physicians across different departments and facilitates the delivery of medical advice tailored to specific patients. In natural images, the proposal of Transformer-based baseline models has significantly improved the accuracy of the VQA task [19, 2]. However, sufficient consideration has not been given to the application in the medical field, as various clinical constraints need to be taken into account for its implementation.

Personalized Federated Learning (pFL) [4] is an approach that focuses on the performance of individual clients rather than the global model of a central server, using protected data to learn personalized models. This approach demonstrates significant compatibility with the medical sector, yet it simultaneously highlights the potential for enhancement in client information aggregation. For example, recent studies [29] based on prompt learning [23, 22] do not consider the relevance between clients, leading to a degradation of model performance. Assessing the reliability of each client is a crucial issue when considering clinical applications in pFL [11].

In this paper, we explore the combination of pFL and VQA, tackling a more clinically challenging problem setting. The VQA task has the potential for significant roles in clinical applications with advancements in large language models (LLMs). We propose a novel Transformer model that quantifies the uncertainty of each client and efficiently aggregates beneficial information in this new problem setting. Specifically, we simulate different departments in a hospital and set up clients for medical images from various organs, each consisting of a VQA model. To reduce the communication burden, we introduce learnable prompts to the Transformer’s multi-head attention (MHA) layer, allowing efficient learning of personalized client data distributions. Furthermore, we propose a Dynamic Likelihood-weighted Uncertainty Calibration (DLUC) process to effectively aggregate information in inter-client communication. This process evaluates the uncertainty of client VQA models using the Dempster-Shafer evidence theory (DST) [16] and dynamically adjusts weights based on this evaluation. This ensures the reliability of the model and supports effective decision-making. Extensive qualitative and quantitative experiments on two closed and open-ended medical VQA datasets demonstrate that our method can efficiently aggregate information for personalized clients.

2 Methodology

Refer to caption — Figure 1: Overview of the proposed pFL-VQA framework.

2.1 Problem Formulation

An overview of our method is shown in Fig. 1. We define $T$ separate clients $\{C_{t}\}_{t=1}^{T}$ , each of which consists of a VQA model on distributed datasets $\{D_{t}\}_{t=1}^{T}$ through the pFL setting. Given a client model $C_{t}$ with parameter $\theta_{t}$ and the evaluation metric $Q(\cdot)$ , the final optimization objective can be expressed as $\mathop{\arg\max}\limits_{\theta_{t}}Q(A,U|\theta_{t},D_{t})$ , where $A$ and $U$ denote the generated answer and the uncertainty, respectively. By optimizing this objective, the local training tries to generate $A$ and $U$ , which will be used as a criterion of the model’s performance evaluation during the next step of the inter-client communication process. Since the reduction in data utilized for training client models inevitably leads to a degradation in performance, the sharing of learned data patterns among clients facilitates a collaborative training process that is mutually advantageous.

Specifically, for client $C_{t}$ , we define the communication method $E$ , and the objective is $\mathop{\arg\max}\limits_{E}\sum_{t=1}^{T}Q(A|E,D_{t},\theta_{1},...,\theta_{T}).$ The objective is to optimize a method that maximizes the information obtained from the other clients for the local training of the client $C_{t}$ . By optimizing the two mentioned objectives, we can train personalized client models for different medical data distributions without data sharing. This method also maximizes the aggregation of valid information from other clients for each client $C_{t}$ . The local training and inter-client communication processes are described in Sections 2.2 and 2.3, respectively.

2.2 Local Training for Medical VQA Client Model

For client $C_{t}$ , the dataset $D_{t}$ comprises $n$ instances that consist of an input question $Q_{t}$ , an image $I_{t}$ , and the corresponding ground truth answer $A_{t}$ . We utilize Transformer-based encoders $E_{I}$ and $E_{Q}$ to extract features of the input image and question separately. Subsequently, an answering decoder $D$ generates the predicted answer.

In the proposed method, we add learnable prompts $\mathbf{p}\in\mathbbm{R}^{d\times m}$ ( $d$ is the length of the prompt, $m$ is the hidden dimension of the Transformer) to the multi-head attention (MHA) layers of the encoders. Specifically, we use Prefix-Tuning [8] splitting $\mathbf{p}$ into $\{\mathbf{p}^{K},\mathbf{p}^{V}\}\in\mathbbm{R}^{d/2\times m}$ , which are introduced into the MHA layer as follows:

Attention(\mathbf{Q},\hat{\mathbf{K}},\hat{\mathbf{V}})=softmax(\frac{\mathbf{Q}\hat{\mathbf{K}}^{T}}{\sqrt{m}})\hat{\mathbf{V}};\ \hat{\mathbf{K}}=\left[\mathbf{p}^{k};\mathbf{K}\right],\hat{\mathbf{V}}=\left[\mathbf{p}^{v};\mathbf{V}\right],

(1)

where $\mathbf{Q}$ , $\mathbf{K}$ , and $\mathbf{V}$ are the original inputs of the MHA layer.

Conventional VQA models output the prediction for the answer probability directly from the hidden state of the decoder output via a softmax activation function. In order to capture the uncertainty for evaluating the model performance, based on the DST theory, we consider the probability of answers to obey a Dirichlet distribution with parameter $\mathbf{\alpha}_{j}=\{\alpha_{j}\}_{j=1}^{J}$ ( $J$ is the length of the answer list). Specifically, we define the output hidden states of the decoder $D$ as the evidence $\mathbf{e}=\{e_{j}\}_{j=1}^{J}$ of each candidate’s answer. Since the evidence is non-negative, in order to get the evidence, we replace the final softmax layer of the conventional VQA models with the activation layer $exp(\cdot)$ , and $\mathbf{e}$ can be calculated as $e_{j}=exp(f(D_{t}|\theta_{t}))$ . The Dirichlet parameter $\mathbf{\alpha}=\{\alpha_{j}\}_{t=1}^{T}$ can be calculated by evidence as $\alpha_{j}=e_{j}+1$ . Based on $\mathbf{\alpha}$ , the belief mass $\mathbf{b}=\{b_{j}\}_{j=1}^{J}$ of the model for each candidate’s answer can be defined as $b_{j}=\frac{e_{j}}{S}$ , where $S=\sum_{j=1}^{J}\alpha_{j}$ . Unlike traditional probability distributions, the sum of $\mathbf{b}$ does not equal 1, and the difference from 1 is the uncertainty $u$ of the model, which can be calculated as $u=1-\sum_{j=1}^{J}b_{j};u>0,b_{j}>0$ .

To assign more evidence to the correct answer, we employ the following Dirichlet term:

Dirichlet(\theta_{t})=\int\frac{1}{B(\mathbf{\alpha}_{j})}\prod_{j=1}^{J}p_{j}^{\alpha_{j}-1}d\mathbf{p}_{j},

(2)

where $B(\cdot)$ is the beta function. The Dirichlet term is introduced into the cross-entropy loss function according to obtain $\mathcal{L}^{unc}(\cdot)$ as follows:

\mathcal{L}^{unc}(\theta_{t})=\sum_{j=1}^{J}y_{j}(\psi(S_{j})-\psi(\alpha_{j})),

(3)

where $y_{j}$ is the one-hot vector of the ground truth answer, and $\psi(\cdot)$ is the digamma function. Note that Eq. (3) aims to allocate the sum of all evidence generated by predictions as much as possible to the correct answers, providing positive feedback. However, since the above loss does not ensure that incorrect labels yield less evidence, the KL term is introduced to minimize the evidence for incorrect labels as much as possible to 0:

	$\displaystyle KL[D(\mathbf{p}_{j}\|\hat{\mathbf{\alpha}}_{j})\|\|D(\mathbf{p}_{j}\|\mathbf{1})]$	$\displaystyle=log(\frac{\Gamma(\sum_{j=1}^{J}\hat{\alpha}_{j})}{\Gamma(J)\prod_{j=1}^{J}\Gamma(\hat{\alpha}_{j})})$		(4)
		$\displaystyle+\sum_{j=1}^{J}(\hat{\alpha}_{j}-1)[log(\hat{\alpha}_{j})-log(\sum_{i=1}^{J}\hat{\alpha}_{j})]$		(5)

Here, $\mathbf{1}$ is a parameter vector of $K$ ones, and $\hat{\mathbf{\alpha}}_{j}=\mathbf{y}_{j}+(1-\mathbf{y}_{j})\odot\alpha_{i}$ is the Dirichlet parameter to prevent penalties for correct evidence (when $y_{j}=1$ , $\hat{\mathbf{\alpha}}_{j}=1$ and the loss becomes 0). Therefore, the final loss function can be calculated as follows:

\mathcal{L}(\theta)=\mathcal{L}^{unc}(\theta)+\lambda KL[D(\mathbf{p}_{j}|\hat{\alpha}_{j})||D(\mathbf{p}_{j}|\mathbf{1})],

(6)

where $\lambda$ is a balance parameter. The client model can output the answer text and the uncertainty $U_{n}$ by training the model using the proposed loss function. In the inter-client communication process, uncertainty evaluates the model’s performance.

2.3 DLUC-based Inter-client Communication Process

We define a dynamic weight $w_{t}$ for client $C_{t}$ , and the prompt $\mathbf{p}_{t}$ is added with the weighted sum of the prompts of the other clients $\mathbf{p}_{t}^{S}$ . Given the data $D_{t}$ and the clients’ parameters $\theta_{t}$ , the conditional expectation, as a lower bound on the true likelihood, is computed as

\mathbbm{E}\{logPr[D_{t}|\theta_{t}]\}=\sum_{t=1}^{T}logPr(A_{t}|D_{t},\theta_{t},\mathbf{p}_{t}+\mu\mathbf{p}_{t}^{S}),

(7)

where $\mathbf{p}_{t}^{S}=\sum_{k=1}^{T,k\neq t}w_{k}\mathbf{p}_{k}$ , $\mu$ is an aggregation rate, and the $w_{t}$ is the weight optimized by the EM method $\mathbbm{E}$ . Since the prompts represent the client data schema, we use a neural network to optimize the objective of Eq. (7). We take the uncertainty $u$ as the network input and the weights $w$ as the output. During training, we set up local training step $step_{l}$ and inter-client communication $step_{c}$ separately, representing the number of iterations each is executed, and cross these two processes. This DLUC aims to find the best configuration of weights to maximize the total likelihood of the model for all client data. This optimization allows the client to aggregate the maximum amount of information from the server while avoiding the interference of invalid information.

3 Experiments

We are motivated by the excellent performance of transformers on multimodal tasks in the medical domain. Considering the wide range of VQA task applications, we selected medical images and QA pairs from VQA-RAD [6] and Slake [10] datasets. These datasets were further partitioned into sub-datasets based on the anatomical focus of the images. With reference to the setup of different departments in the actual clinic, we split Slake into “Lung” (Client1), “Abdomen” (Client2), “Brain” (Client3), and “Other” (Client4), VQA-RAD into “Abdomen” (Client1), “Chest” (Client2), “Head” (Client3), respectively. Details of the dataset can be found in Table 1 of Supplementary. To mitigate the impact of accuracy fluctuations observed among some clients, attributed to the limited size of their test datasets, we determined the final accuracy based on the average results and the variance from the last round of local training. Furthermore, we also tested the method’s performance when the number of clients spikes using the large-scale dataset PMC-VQA [27].

Table 1: Quantitative results of accuracy comparison with previous medical VQA models. The Enc. and Upd. denote the number of parameters in the baseline and parameters that need to be updated during training, respectively.

Method	VQA accuracy (%)							Param. (M)
Method	Slake				VQA-RAD			Enc.	Upd.
PMC-VQA [27]	82.5				86.8			7000	7000
BioMedGPT [25]	82.5				81.3			1500	1500
LoRA [18]	82.1				-			1500	15
M2I2 [7]	83.2				70.8			600	600
BioMedCLIP [26]	86.7				79.8			150	340
MEVF-BAN [13]	75.4				75.1			50	50
VGG+SAN [10]	75.4				74.0			50	50
	Client1	Client2	Client3	Client4	Client1	Client2	Client3
PMVQA [29]	84.4	72.3	76.9	81.4	68.9	64.7	74.2	63	0.3
PM	$81.7_{\pm 0.5}$	$\mathbf{74.7}_{\pm 0.8}$	$\mathbf{81.3}_{\pm 0.3}$	$78.2_{\pm 1.0}$	$\mathbf{72.1}_{\pm 1.2}$	$\mathbf{65.1}_{\pm 0.5}$	$\mathbf{81.2}_{\pm 0.8}$	63	0.01

Table 2: Comparison of VQA accuracy (%) across several common baseline networks before and after applying the proposed method.

Baseline	Param.	Slake				VQA-RAD
Baseline	Param.	Lung	Abdomen	Brain	Others	Chest	Abdomen	Head
VIT-B/32 [15]	63M	$-1.2_{\pm 1.0}$	$\mathbf{+1.8}_{\pm 0.8}$	$\mathbf{+0.9}_{\pm 0.2}$	$\mathbf{+4.7}_{\pm 1.7}$	$\mathbf{+2.8}_{\pm 1.6}$	$\mathbf{+2.1}_{\pm 0.1}$	$\mathbf{+0.6}_{\pm 0.2}$
PubMedCLIP [3]	63M	$-0.3_{\pm 0.5}$	$\mathbf{+0.3}_{\pm 0.9}$	$\mathbf{+6.8}_{\pm 0.9}$	$\mathbf{+4.7}_{\pm 0.4}$	$\mathbf{+0.2}_{\pm 1.2}$	$-2.3_{\pm 1.4}$	$\mathbf{+2.7}_{\pm 1}$
VIT-L/14 [15]	300M	$-0.7_{\pm 1.0}$	$\mathbf{+0.3}_{\pm 1.2}$	$\mathbf{+0.2}_{\pm 0.8}$	$\mathbf{+3.0}_{\pm 0.7}$	$\mathbf{+2.8}_{\pm 2.1}$	$\mathbf{+2.4}_{\pm 1.4}$	$-1.52_{\pm 2.5}$
ViT-L/14@336px [15]	300M	$-1.2_{\pm 0.2}$	$-2.7_{\pm 0.6}$	$\mathbf{+2.3}_{\pm 0.4}$	$\mathbf{+9.5}_{\pm 2.9}$	$\mathbf{+1.1}_{\pm 0.8}$	$\mathbf{+2.0}_{\pm 1.4}$	$\mathbf{-0.4}_{\pm 2.5}$

Table 3: Ablation study on Slake.

	CLOSED				OPEN
	Client1	Client2	Client3	Client4	Client1	Client2	Client3	Client4
PM w/o client	$77.8_{\pm 0.2}$				$58.2_{\pm 0.4}$
PM w/o server	$\mathbf{83.7}_{\pm 0.4}$	$71.3_{\pm 0.7}$	$80.8_{\pm 0.6}$	$74.9_{\pm 1.6}$	$59.5_{\pm 0.2}$	$\mathbf{56.3}_{\pm 0.8}$	$51.1_{\pm 0.9}$	$43.8_{\pm 0.8}$
PM w/o DLUC	$79.9_{\pm 0.4}$	$74.8_{\pm 0.6}$	$80.9_{\pm 0.4}$	$78.4_{\pm 0.4}$	$65.9_{\pm 0.1}$	$55.9_{\pm 0.2}$	$63.7_{\pm 0.5}$	$61.1_{\pm 0.8}$
PM	$81.7_{\pm 0.5}$	$74.7_{\pm 0.8}$	$\mathbf{81.3}_{\pm 0.3}$	$\mathbf{78.9}_{\pm 1.0}$	$\mathbf{66.0}_{\pm 0.3}$	$56.1_{\pm 0.3}$	$\mathbf{63.8}_{\pm 0.2}$	$\mathbf{61.7}_{\pm 0.8}$

Table 4: Ablation study on VQA-RAD.

	CLOSED			OPEN
	Client1	Client2	Client3	Client1	Client2	Client3
PM w/o client	$67.5_{\pm 0.9}$			$42.6_{\pm 0.4}$
PM w/o server	$68.8_{\pm 0.7}$	$62.9_{\pm 1.6}$	$80.8_{\pm 1.4}$	$44.2_{\pm 0.2}$	$38.2_{\pm 0.3}$	$34.5_{\pm 0.1}$
PM w/o DLUC	$71.0_{\pm 0.5}$	$61.8_{\pm 0.6}$	$76.5_{\pm 0.8}$	$47.7_{\pm 0.2}$	$39.9_{\pm 0.7}$	$\mathbf{35.3}_{\pm 0.2}$
PM	$\mathbf{72.1}_{\pm 1.2}$	$65.1_{\pm 0.9}$	$\mathbf{81.2}_{\pm 0.8}$	$\mathbf{47.9}_{\pm 0.2}$	$40.3_{\pm 0.3}$	$34.3_{\pm 0.3}$

For specific modeling details, we employed the pre-trained Contrastive Language–Image Pre-training (CLIP) [15] model as the image and text encoders $E_{I}$ and $E_{Q}$ . All clients share a CLIP with a fixed weight, and only the client’s prompts are updated during training. In our experiments, we got the best results when we added prompts in the first four blocks of the Transformer and set the length of the prompts to 24 and 30 for the Slake and VQA-RAD, respectively. The reason is that the feature in different organs focus more on patterns in smaller regions, and the Transformer’s beginning few blocks outputs tend to capture more localized information. The learning rate of the prompts is initially 0.001, multiplied by 0.5 after each $Step_{l}$ , i.e., one complete local update. The answer layer is implemented as a 2-layer MLP with a hidden size 512 and a dropout rate of 0.2. We used a fully connected layer for the DLUC, whose learning rate is initially 0.01 and multiplied by 0.1 after each complete inter-client communication.

Comparison Methods: Given the complexity and the substantial size of the baseline model employed, along with the intricacies of the multimodal task setup, direct comparison with previous FL methods is not feasible. Considering that we aim to enhance the personalized performance of medical VQA, we compare the proposed with several state-of-the-art medical VQA models regarding accuracy and model parameters that need to be updated. Specifically, we employed PMC-VQA [27], BioMedGPT [25], M2I2 [7], BioMedCLIP [26], MEVF-BAN [13], VGG+SAN [10], LoRA [18] and PMVQA [29].

As the results shown in Table 1, compared to the PMVQA [29] with the same task setup, we averaged a 3.5% accuracy improvement on most clients. Compared to other locally trained models, our method achieves higher precision when the number of baseline parameters is close. However, there is still a gap when comparing methods using more parameter baselines. It is worth noting parameters that need to be updated by our method are 0.01% of the Vision Transformer-based model and 0.00001% of the current SOTA model.

To demonstrate that our methodology can be effectively extended to all transformer structure networks, we tested the performance of our method on multiple baselines. As shown in Table 2, our method can effectively improve the personalization performance of multimodal transformers, and we achieved an average of 3% improvement on most clients. Further analysis of the differences in weights and performance across clients is in the supplementary.

Ablation studies. We designed: trained on the whole data without setting up the client, setting up the client but without parameter sharing in sever, and without DLUC for information aggregation (equal weights). As shown in Table 3 and Table 4, the results demonstrate that DLUC can effectively aggregate clients’ information to improve the performance of each client.

Analysis of inter-client weights. Fig. 2 illustrates the weights generated by DLUC during inter-client communication for each client, and the results show significant differences in the dependency between clients. For example, the image prompt of “Lung” is highly dependent on “Abdomen” information because the images of these client in the Slake are tomographic CT scans from immediate area that have similar features. Furthermore, the images from the three clients of VQA-RAD are obtained from CT, X-ray, and MRI, so there is no significant dependence on the results due to the large modality gap. The results show that our generation of weights is clinically meaningful.

Cross evaluation between prompts. To further demonstrate that prompts can effectively personalize the model rather than learn generalizability information, we performed accuracy tests for each client using prompts from the other clients. As shown in Fig. 4, there is a significant drop in accuracy when the prompts are exchanged, proving that the prompts learn information about different data distribution.

Impact of hyperparameters. The model’s performance is affected by three primary parameters: the local training step $step_{l}$ , the DLUC step in inter-client communication $step_{c}$ , and the aggregation rate $\mu$ . The comparison results on step are shown in Fig. 1 in Supplementary. The results shows that higher $step_{l}$ tends to perform better, and performances on higher $step_{c}$ are better when the $step_{l}$ is low. This is because when there is less local training, the model cannot fully adapt to the local data, and optimizing the aggregation weights by DLUC can integrate information from different clients more effectively. As for $\mu$ , both too high and too low result in deterioration of the results due to excessive or insufficient influence of external information as shown in Fig. 4.

Extreme Tests: Since each of our clients essentially retains only learnable prompts with only 0.01M parameters, our method can set up a tremendous number of clients, which was absolutely difficult to achieve with the previous pFL method. We experimented with setting up 500 clients on the PMCVQA dataset, consumed only 13G of VRAM, and experimental results show an average 1.8% accuracy improvement on 56% of the clients.

4 Conclusion

We present a prompt-based, reliable pFL method for medical VQA. Our method creates clients for heterogeneous medical data and achieves aggregation of information through the novel DLUC. In the DLUC process, we calculate the uncertainty of each client through DST theory and generate aggregation weights based on maximum likelihood estimation through uncertainty. The experimental results demonstrate that our method can effectively improve the performance of the transformer-based client model with minimal overhead. Then, the computation of our method is independent of the size of the baseline network and can be applied in personalized learning with any transformer of any size. In future work, we will introduce the method to personalize larger-scale models.

References

[1] Abouelmehdi, K., Beni-Hessane, A., Khaloufi, H.: Big healthcare data: preserving security and privacy. Journal of big data 5(1), 1–18 (2018)
[2] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
[3] Eslami, S., de Melo, G., Meinel, C.: Does clip benefit visual question answering in the medical domain as much as it does in the general domain? arXiv preprint arXiv:2112.13906 (2021)
[4] Fallah, A., Mokhtari, A., Ozdaglar, A.: Personalized federated learning: A meta-learning approach. arXiv preprint arXiv:2002.07948 (2020)
[5] Jiang, M., Zhong, Y., Le, A., Li, X., Dou, Q.: Client-level differential privacy via adaptive intermediary in federated medical imaging. In: MICCAI. pp. 500–510 (2023)
[6] Lau, J.J., Gayen, S., Abacha, A.B., Demner-Fushman, D.: A dataset of clinically generated visual questions and answers about radiology images. Scientific Data 5, 1–10 (2018)
[7] Li, P., Liu, G., Tan, L., Liao, J., Zhong, S.: Self-supervised vision-language pretraining for medial visual question answering. In: ISBI. pp. 1–5 (2023)
[8] Li, X.L., Liang, P.: Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 (2021)
[9] Lin, Z., Zhang, D., Tao, Q., Shi, D., Haffari, G., Wu, Q., He, M., Ge, Z.: Medical visual question answering: A survey. Artificial Intelligence in Medicine 143, 102611 (2023)
[10] Liu, B., Zhan, L.M., Xu, L., Ma, L., Yang, Y., Wu, X.M.: Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering. In: ISBI. pp. 1650–1654 (2021)
[11] Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., Neubig, G.: Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys 55(9), 1–35 (2023)
[12] McMahan, B., Moore, E., Ramage, D., Hampson, S., y Arcas, B.A.: Communication-efficient learning of deep networks from decentralized data. In: AISTATS. pp. 1273–1282. PMLR (2017)
[13] Nguyen, B.D., Do, T.T., Nguyen, B.X., Do, T., Tjiputra, E., Tran, Q.D.: Overcoming data limitation in medical visual question answering. In: MICCAI. pp. 522–530 (2019)
[14] Nguyen, D.C., Pham, Q.V., Pathirana, P.N., Ding, M., Seneviratne, A., Lin, Z., Dobre, O., Hwang, W.J.: Federated learning for smart healthcare: A survey. ACM Comput. Surv. 55(3) (feb 2022)
[15] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: ICML. pp. 8748–8763 (2021)
[16] Sensoy, M., Kaplan, L., Kandemir, M.: Evidential deep learning to quantify classification uncertainty. In: NIPS. p. 3183–3193 (2018)
[17] Sheller, M.J., Edwards, B., Reina, G.A., Martin, J., Pati, S., Kotrotsou, A., Milchenko, M., Xu, W., Marcus, D., Colen, R.R., et al.: Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data. Scientific reports 10(1), 12598 (2020)
[18] van Sonsbeek, T., Derakhshani, M.M., Najdenkoska, I., Snoek, C.G.M., Worring, M.: Open-ended medical visual question answering through prefix tuning of language models. In: MICCAI. pp. 726–736 (2023)
[19] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)
[20] Wang, J., Jin, Y., Stoyanov, D., Wang, L.: Feddp: Dual personalization in federated medical image segmentation. IEEE Transactions on Medical Imaging (2023)
[21] Wang, M., Wang, L., Xu, X., Zou, K., Qian, Y., Goh, R.S.M., Liu, Y., Fu, H.: Federated uncertainty-aware aggregation for fundus diabetic retinopathy staging. In: MICCAI (2023)
[22] Wang, Z., Zhang, Z., Ebrahimi, S., Sun, R., Zhang, H., Lee, C.Y., Ren, X., Su, G., Perot, V., Dy, J., et al.: Dualprompt: Complementary prompting for rehearsal-free continual learning. In: ECCV. pp. 631–648 (2022)
[23] Wang, Z., Zhang, Z., Lee, C.Y., Zhang, H., Sun, R., Ren, X., Su, G., Perot, V., Dy, J., Pfister, T.: Learning to prompt for continual learning. In: CVPR. pp. 139–149 (2022)
[24] Xu, J., Glicksberg, B.S., Su, C., Walker, P., Bian, J., Wang, F.: Federated learning for healthcare informatics. Journal of Healthcare Informatics Research 5, 1–19 (2021)
[25] Zhang, K., Yu, J., Yan, Z., Liu, Y., Adhikarla, E., Fu, S., Chen, X., Chen, C., Zhou, Y., Li, X., et al.: Biomedgpt: A unified and generalist biomedical generative pre-trained transformer for vision, language, and multimodal tasks. arXiv preprint arXiv:2305.17100 (2023)
[26] Zhang, S., Xu, Y., Usuyama, N., Bagga, J., Tinn, R., Preston, S., Rao, R., Wei, M., Valluri, N., Wong, C., et al.: Large-scale domain-specific pretraining for biomedical vision-language processing. arXiv preprint arXiv:2303.00915 2(3), 6 (2023)
[27] Zhang, X., Wu, C., Zhao, Z., Lin, W., Zhang, Y., Wang, Y., Xie, W.: Pmc-vqa: Visual instruction tuning for medical visual question answering. arXiv preprint arXiv:2305.10415 (2023)
[28] Zhou, Q., Zheng, G.: Fedcontrast-gpa: Heterogeneous federated optimization via local contrastive learning and global process-aware aggregation. In: MICCAI. pp. 660–670. Springer (2023)
[29] Zhu, H., Togo, R., Ogawa, T., Haseyama, M.: Prompt-based personalized federated learning for medical visual question answering. arXiv preprint arXiv:2402.09677 (2024)