Federated Adaptive Prompt Tuning for Multi-Domain Collaborative Learning

The idea of aggregating client data distribution information to the cloud can be traced back to SCM (Li et al. 2007). Recently, FedAvg (McMahan et al. 2017) extends the aggregation idea to neural networks and proposes the federated averaging algorithm. Despite FedAvg’s strong performance under the assumption of IID, as the non-IID degree increases (due to the diverse distributions of client datasets), the aggregation model’s performance continues to deteriorate, making the non-IID problem a key area of research. Some efforts attempt to enhance the performance of the global aggregation model by improving the optimization algorithm (Li et al. 2020; Karimireddy et al. 2020; Reddi et al. 2020; Wang et al. 2020b; Su, Li, and Xue 2023) or designing better aggregation methods (Wang et al. 2020a; Singh and Jaggi 2020), while others (Dinh, Tran, and Nguyen 2020; Fallah, Mokhtari, and Ozdaglar 2020; Hanzely et al. 2020; Li et al. 2021; Luo et al. 2022) turn to assigning different models to each client, i.e., personalized FL, such as using local batch normalization (Li et al. 2021), decoupling the model into personalized and global parts (Luo et al. 2022). The proposed FedAPT belongs to the former, i.e., to enhance the global model.

Some works (Chen et al. 2022; Nguyen et al. 2023) study the impact of pre-training on FL. Unlike the fine-tuning methods, they need to train the complete model. FedPCL (Tan et al. 2022) studies a setting where each client has $K$ fixed backbones, they fine-tune the projection head by sharing the prototype features of user data. Unlike FedPCL, we do not need to maintain multiple backbones locally, and we do not upload any prototype features. All in all, different from the existing works, we pay more attention to how to effectively utilize the knowledge in the powerful pre-trained models.

Recently, FedIns (Feng et al. 2023) shares a similar spirit with our work, both enabling Instance-adaptive Inference. The difference lies in the implementation: FedIns achieves adaptive inference for ViT through learnable Keys and SSF pool, while our approach achieves adaptive inference for CLIP through randomly initialized keys and a learnable query network coupled with prompts.

The contrastive vision-language pre-trained model CLIP (Radford et al. 2021) transforms image recognition into an image-text matching problem, freeing the object recognition task from human-annotated data, such that a huge amount (400M) of noisy image-text pairs can be used for training. The pre-trained model can be transferred to multiple datasets. Google releases ALIGN (Jia et al. 2021) to expand CLIP’s training data to 1.8B, which achieves better results. Thanks to its powerful representation capability, CLIP has also been successfully applied to a variety of downstream visual tasks, such as RegionCLIP (Zhong et al. 2022) and DetCLIP (Yao et al. 2022) for object detection, MedCLIP (Wang et al. 2022) for medical image analysis, CCR-CLIP (Yu et al. 2023) for text recognition.

Prompt (Liu et al. 2021a) adds additional words or sentences to the input text of the pre-trained language model, to make the pre-trained model better handle downstream tasks. The succeeding works (Li and Liang 2021; Liu et al. 2021b) treat the prompts as continuous vectors in order to learn new knowledge in the process of fine-tuning, which is called Prompt Tuning. In CLIP’s follow-up work, other researchers (Zhou et al. 2022b, a) tried to improve prompt tuning in CLIP. The work most relevant to ours is CoCoOp (Zhou et al. 2022a), which trains prompts conditioned on visual features. However, it is designed for centralized training and heavily relies on high GPU computational power, making it hard to apply to clients in federated learning. In contrast, in this paper, the training of conditions and prompts are decoupled, significantly accelerating the training process.

In the field of federated image classification, work related to CLIP is rare. PromptFL (Guo et al. 2022) extends CoOp (Zhou et al. 2022b) to the federated learning scenarios, averaging multiple prompts on the server. PromptFL shows the potential application of prompt tuning for CLIP in FL, however, it lacks optimizations tailored for multi-domain scenarios. In comparison, FedAPT proposes an adaptive prompt tuning method for multi-domain image classification. FedCLIP (Lu et al. 2023) adds an adapter consisting of two fully connected layers to the end of the image encoder. FedTPG (Qiu et al. 2023) proposes a prompt generator with better generalization performance on unknown categories. However, they do not generate personalized prompts for the images based on the domain information.

Consider a federated learning scenario, such as autonomous driving, where multiple cars are deployed in $K$ regions (domains). Data from the same region exhibits similar styles, while data from different regions possesses distinct styles. Each region has $N_{dom}$ clients. We assume that each client has sufficient computing power to train the prompts. In supervised setting, $\mathcal{D}_{n}=\{\mathbf{x}_{i},y_{i}\}_{i=1}^{m_{n}},n=1,\cdots,N$ denote the local datasets, $m_{n}$ is the number of images in the $n$-th client. In unsupervised setting, $\mathcal{D}_{n}=\{\mathbf{x}_{i}\}_{i=1}^{m_{n}},n=1,\cdots,N$. Images in diverse domains have different styles, see Figure 1. Note that there may exist domain and category divergences across different clients at the same time, the divergence of category refers to the different label distribution of different clients. Let $\mathcal{I}(\cdot)$ and $\mathcal{T}(\cdot)$ denote the image encoder and the text encoder. To reduce the calculation and communication costs, we freeze the parameters of the two encoders and perform the training task by fine-tuning the prompts.

Let $\mathbf{t}^{c}$ be the word embedding of the $c$th-class, the class text could have the form ‘a picture of a dog’. Then the prediction of CLIP for image classification is:

$\displaystyle p(c\mid\mathbf{x})=\frac{\exp\left(\cos\left(\mathcal{T}(\mathbf{t}^{c}),\mathcal{I}(\mathbf{x})\right)/\tau\right)}{\sum_{i=1}^{C}\exp\left(\cos\left(\mathcal{T}(\mathbf{t}^{i}),\mathcal{I}(\mathbf{x})\right)/\tau\right)}$

(1)

where $C$ is the number of classes, $\tau$ is a temperature parameter learned by CLIP.

To fine-tune the model, one method is to manually set prompts, such as replacing ‘a picture of a dog’ with ‘a picture of a cute dog’. However, in this way, because the change of words will directly affect the performance of the model, we need to find the most appropriate words. Another method is prompt tuning (Li and Liang 2021; Liu et al. 2021b; Zhou et al. 2022b). Let $\mathbf{p}^{c}=[\boldsymbol{v}^{c}_{1},\boldsymbol{v}^{c}_{2},\ldots,\boldsymbol{v}^{c}_{M}]$ be the prompt. The input of the $c$-th class for the text encoder is $(\mathbf{p}^{c};\mathbf{t}^{c})$, where $\{\boldsymbol{v}^{c}_{i}\}^{M}_{i=1}$ are learnable prompts for the $c$-th class. $\boldsymbol{v}^{c}_{i}$ has the same dimension as word embedding $\mathbf{t}^{c}$, $M$ is the number of learnable prompts.

The concept of FL is introduced in FedAvg (McMahan et al. 2017) to collaboratively train a global model that works for each participant. A straightforward way to apply CLIP’s prompt tuning to the FL field is that each client trains its own prompt and uploads the prompt to the server. The server averages the prompts of all clients and distributes them to the clients. Let $\mathbf{p}^{c}_{n}$ denotes the prompt for the $c$-th class in the $n$-th client. For the sake of brevity, we omit $c$ and use $\mathbf{p}_{n}$ to denote the prompt in the $n$-th client. Let $\mathbf{p}_{g}$ denotes the prompt of the global model. Then the global objective is:

$\displaystyle\mathbf{p}_{g}=\arg\min_{\mathbf{p}\in\mathbb{R}^{d}}\frac{1}{N}\sum_{n=1}^{N}\mathbb{E}_{\mathbf{x},y\sim\mathcal{D}_{n}}[\ell(\mathcal{T}(\mathbf{p});\mathcal{I}(\mathbf{x}))]$

(2)

where $\ell$ is the loss of the $n$-th client. Although this approach can handle scenarios where multiple clients are IID, it is still difficult for a single global prompt to process clients from diverse data domains.

The input of the text encoder in PromptFL always uses a fixed prompt for all samples: $\mathbf{p}_{g}\in\mathbb{R}^{C\times s\times d}$, which is the learned prompt by federated averaging, $C$ is the number of classes, $s$ is the length of prompt, $d$ is the embedding dimension. However, under the setting in this paper, each client may come from a specific domain, and there are obvious domain divergences among the images. A fixed prompt is no longer suitable for images in all domains. It is necessary to design a new form. An intuitive way is to introduce personalized prompts into the input of the text encoder for each domain:

$\displaystyle\mathbf{p}_{g}+\mathbf{p}_{p_{k}}$

(3)

where $\mathbf{p}_{p_{k}}$ encodes the personalized domain knowledge of the $k$-th domain. Unfortunately, under the goal of one-model-fits-all, we need a unified global model, rather than adding personalized parameters $\mathbf{p}_{p_{k}}$ for each client. Then a compromise improvement for Eq. 3 is to keep all $\mathbf{p}_{p_{k}}$ in the global model, then filter prompts based on the image features:

$\displaystyle\mathbf{p}_{g}+f([\mathbf{p}_{p_{1}},\cdots,\mathbf{p}_{p_{K}}],\mathcal{Q}(\mathcal{I}(\mathbf{x})))$

(4)

where $\mathbf{p}_{p_{k}}$ is the personalized prompt for the $k$-th domain, $f(\cdot)$ is a filter function, $\mathcal{Q}$ is an adaptive network that is used to provide auxiliary information for filtering. However, this operation needs to retain a number of $\mathbf{p}_{p_{k}}$, thus introducing additional parameters in the global model.

Finally, we adopt an adaptive prompt tuning approach. Let $\mathcal{P}$ denotes the output of the APT module, the input for this module is the image features $\mathcal{I}(\mathbf{x})$. We use the key vector $\mathbf{e}_{k}\in\mathbb{R}^{s\times d}$ to encode personalized domain knowledge into $\mathbf{p}_{g}$, and decode it when necessary. $\mathcal{P}$ in the global model is:

$\displaystyle\mathcal{P}(\mathcal{I}(\mathbf{x}))=\mathbf{p}_{g}+\mathbf{p}_{g}\odot\sum_{k=1}^{K}q_{k}\mathbf{e}^{\prime}_{k}$

(5)

where $q_{k}=\mathcal{Q}_{g}(\mathcal{I}(\mathbf{x}))_{k}$, $\sum_{k}q_{k}=1$, $\odot$ represents the element-wise multiplication, $\mathbf{e}^{\prime}_{k}\in\mathbb{R}^{C\times s\times d}$ is copied from $\mathbf{e}_{k}$ for $C$ times to match the dimension of $\mathbf{p}_{g}$, and $\mathcal{Q}_{g}$ is the adaptive network with parameters $\phi$, which can give a one-hot or soft-membership vector of $K$ dimensions. In the $n$-th client from the $k$-th domain, the prompt for the text encoder in the training process is:

$\displaystyle\mathbf{p}_{n}+\mathbf{p}_{n}\odot\mathbf{e}^{\prime}_{k}$

(6)

where $\mathbf{p}_{n}$ is initialized by $\mathbf{p}_{g}$, $\mathbf{e}^{\prime}_{k}$ is frozen. Through the design of Eqs.5 and 6, we do not need to introduce additional learnable parameters $\mathbf{p}_{p_{k}}$, but use the unified $\mathbf{p}_{g}$, which is named “meta prompt”. The objective for the meta prompt is:

$\displaystyle\mathbf{p}_{g}=\arg\min_{\mathbf{p}}\frac{1}{N}\sum_{n=1}^{N}\mathbb{E}_{\mathbf{x}\sim\mathcal{D}_{n}}[\ell(\mathcal{T}(\mathbf{p}+\mathbf{p}\odot\mathbf{e}^{\prime}_{n});\mathcal{I}(\mathbf{x}))]$

To enable multiple clients to train APT cooperatively, there are three steps. At the beginning of federated training, the server randomly initializes the keys and sends them to different clients (Step 1).

Step2: Local Training. To evaluate the capabilities of CLIP and FedAPT in various FL settings, we consider two different client settings:

In the supervised setting, each client updates the global $\mathbf{p}_{g}$ and $\mathcal{Q}_{g}$ with the local data respectively to obtain $\mathbf{p}_{n}$ and $\mathcal{Q}_{n}$. The prediction of the $n$-th local model is:

$\displaystyle p(c\mid\mathbf{x})=\frac{e^{\cos\left(\mathcal{T}(\mathbf{p}^{c}_{n}+\mathbf{p}^{c}_{n}\odot\mathbf{e}_{k}),\mathcal{I}(\mathbf{x})\right)/\tau}}{\sum_{i=1}^{C}e^{\cos\left(\mathcal{T}(\mathbf{p}^{i}_{n}+\mathbf{p}^{i}_{n}\odot\mathbf{e}_{k}),\mathcal{I}(\mathbf{x})\right)/\tau}}$

where $k$ is the domain index of the client. Then the cross entropy loss for image classification is:

$\displaystyle\mathcal{L}_{c}=\frac{1}{\left|\mathcal{D}_{n}\right|}\sum_{\mathbf{x}_{i},y_{i}\sim\mathcal{D}_{n}}-\log p\left(y_{i}\mid\mathbf{x}_{i},\mathbf{p}_{n}\right)$

(7)

In an unsupervised setting, we design a two-stage training approach. In the first stage, we augment the unlabeled data (same augment strategy as SimCLR (Chen et al. 2020)) and utilize pseudo-labels with high confidence from the original data as supervised information to perform self-training on the augmented data. The training loss is:

$\displaystyle\mathcal{L}_{pseudo}=\frac{1}{\left|\mathcal{D}_{n}\right|}\sum_{\mathbf{x}_{i}\sim\mathcal{D}_{n}}-\log p\left(\hat{y}_{i}\mid\hat{\mathbf{x}}_{i},\mathbf{p}_{n}\right)$

(8)

where $\hat{y}_{i}$ is the pseudo one-hot label of $\mathbf{x}_{i}$ predicted by the original CLIP, $\hat{\mathbf{x}}_{i}$ is the augmented version of $\mathbf{x}_{i}$. After the first stage, we can get a coarse prompt.

In the second stage, use $\mathbf{z}_{i}$ to represent the prediction score vector of $\mathbf{x}_{i}$, we hope to compare the outputs among different samples. For inter-sample comparison, motivated by AaDLoss (Yang et al. 2022) in domain adaptation, for a certain sample, the logits of $K$-nearest neighbor samples with similar image features should be as similar as possible, otherwise, there should be differences. Then the loss is:

$\displaystyle\mathcal{L}_{inter}=\frac{1}{\left|\mathcal{D}_{n}\right|}\sum_{\mathbf{x}_{i}\sim\mathcal{D}_{n}}(-\sum_{j\in\mathcal{C}_{i}}\mathbf{z}_{i}^{\top}\mathbf{z}_{j}+\lambda\sum_{m\in\mathcal{B}_{i}}\mathbf{z}_{i}^{\top}\mathbf{z}_{m})$

(9)

where $\mathcal{C}_{i}$ contains $K$-nearest neighbors, $\mathcal{B}_{i}$ contains the rest samples. A feature bank and a score bank are used to efficiently implement this loss.

As for intra-sample comparison, we need that the prediction of a sample is close to that of its augmented counterpart:

$\displaystyle\mathcal{L}_{intra}=\frac{1}{\left|\mathcal{D}_{n}\right|}\sum_{\mathbf{x}_{i}\sim\mathcal{D}_{n}}KL(\hat{\mathbf{z}}_{i}\|\mathbf{z}_{i})$

(10)

where $\hat{\mathbf{z}}_{i}$ is the predict score of the augment data, $\mathbf{z}_{i}$ is extracted from the score bank. Then the overall loss in the second stage is $\mathcal{L}_{c}=\mathcal{L}_{inter}+\mathcal{L}_{intra}$.

The adaptive network $\mathcal{Q}_{n}(\cdot)$ is a fully-connection layer with parameters $\phi_{n}$, which tries to classify domain information from the features of images. Each client in the $k$-th domain trains its own adaptive network with the following cross-entropy loss:

$\displaystyle\mathcal{L}_{q}=\frac{1}{\left|\mathcal{D}_{n}\right|}\sum_{\boldsymbol{x}\in\mathcal{D}_{n}}-\log p\left(k\mid\mathbf{x}_{i},\phi_{n}\right)$

(11)

The whole loss function for local training is $\mathcal{L}_{c}+\mathcal{L}_{q}$.

Note that training $\mathcal{Q}_{g}(\cdot)$ is a special FL problem, because each client can only access the data of its own domain. In this paper, we find that in the multi-domain scenario, we can obtain good enough $\mathcal{Q}_{g}(\cdot)$ by directly using cross-entropy for local training and federated averaging for aggregation. To further improve the effect of $\mathcal{Q}_{g}(\cdot)$, it is necessary to use special training methods, such as FedAws (Yu et al. 2020). In addition, directly uploading the data prototype (Tan et al. 2022; Peng et al. 2019b), can also train the $\mathcal{Q}_{g}(\cdot)$ in the server, but in order to prevent privacy disclosure, we do not upload any data features or prototypes.

Step3: Aggregation. After the local training step, the $n$-th client uploads $\mathbf{p}_{n}$ and $\phi_{n}$ to the server. Due to $\phi_{n}\in\mathbb{R}^{d_{img}\times K}$, $d_{img}$ is the dimension of each image feature, which is 512 in CLIP, the additional traffic brought by transmitting $\phi_{n}$ is extremely small compared with prompts. The server averages the parameters to obtain the aggregation model parameters:

$\displaystyle\mathbf{p}_{g}=\frac{1}{N}\sum_{n=1}^{N}\mathbf{p}_{n},\quad\phi_{g}=\frac{1}{N}\sum_{n=1}^{N}\phi_{n}$

(12)

Repeat Step 2 and Step 3 for $T$ rounds, then $\mathbf{p}_{g}$ and $\mathcal{Q}_{g}(\cdot)$ with parameters $\phi_{g}$ can be used to establish the global model.

Improve inference efficiency. Notably, during the inference process, assuming a batch size of $B$, since each sample is associated with distinct prompts, the text encoder needs to be executed $B$ times. This issue is also encountered in other visually conditioned prompt methods (Zhou et al. 2022a). To address this concern, we propose an enhanced strategy in Figure 3. By precomputing the text encodings for each key, we eliminate the need for redundant execution of the text encoder, leading to a significant reduction in inference costs.

Datasets. We adopt two datasets, Office-Caltech10 (Gong et al. 2012) and DomainNet (Peng et al. 2019a). Office-Caltech10 (OC) is a small dataset that has four domains (amazon, caltech, dslr, and webcam), and each domain contains 10 overlapping categories between the Office-31 (Saenko et al. 2010) dataset and the Caltech-256 (Griffin, Holub, and Perona 2007) dataset. There are at least 100 and at most 800 images in different domains. Since the number of images is small, we use each domain as a client. DomainNet (DN) is a large-scale multi-domain dataset containing six domains (clipart, infograph, painting, quickdraw, real, and sketch), and each domain contains about 33k-120k images with 345 categories. We set the number of clients $N_{dom}$ in each domain as 1 or 5. When $N_{dom}=5$, we use the Dirichlet distribution to construct the non-IID property across clients. Both datasets contain 224$\times$224 pixel images. Figure 4 depicts the divergences of different domains.

Compared Methods. To comprehensively evaluate the performance, we compare the following methods: 1) ResNet-full: Federated training with ResNet50 (He et al. 2016), all parameters participate in the training procedure. 2) ResNet-tuning, where the backbone of ResNet50 is frozen, only the last layer is trained. 3) ViT-full and 4) ViT-tuning: Replace ResNet in the above two baselines with ViT. 5) CLIP-zeroshot do not make any changes to the CLIP, and directly perform zero-shot image classification in all domains. This helps us understand how much the prompt tuning improves the CLIP. 6) CLIP-FC: Freeze the parameters of CLIP, and insert a learnable fully-connection layer at the end of the image encoder. Only the learnable layer is shared to the server. 7) FedCLIP (Lu et al. 2023): Add an adapter at the end of the visual backbone and perform federated tuning. 8) PromptFL (Guo et al. 2022): A prompt tuning method for CLIP, details can be seen in Section 3.3. 9) FedAPT: our proposed method. In the unsupervised experiment part, we utilize the local training method proposed in Section 4.2 uniformly as FedCLIP and PromptFL have not yet explored federated tuning in an unsupervised setting.

Implementation Details. We use PyTorch to implement all methods. The SGD optimizer is used for both datasets. We set Office-Caltech10 with a learning rate of 0.001 and batch size of 32, and DomainNet with a learning rate of 0.01 and batch size of 256. The global communication round $T_{g}$ is set to 50, and the local training epoch $T_{l}$ is set to $1$. The length of prompts $s$ is 16. In supervised setting, we set class-specific prompts $\mathbf{p}_{g}\in\mathbb{R}^{C\times s\times d}$. In unsupervised setting, due to the lack of supervision information, we use the class-shared prompts $\mathbf{p}_{g}\in\mathbb{R}^{s\times d}$. The CLIP used in this paper takes ViT-B/32 (Dosovitskiy et al. 2020) as the image encoder. Each experiment is repeated three times with different seeds, and the mean result is reported. All experiments are completed with one GeForce RTX 3090 GPU.

Supervised setting with domain differences. We first treat each domain as a separate client with the same class distribution but different domain characteristics.

The results on the two datasets are reported in Table 1. We highlight the following points: 1) From ResNet-full vs. ResNet-tuning and ViT-full vs. ViT-tuning, we can see that pre-trained ViT has more powerful representation ability than pre-trained ResNet. As a result, ResNet must be fully trained in order to achieve the accuracy rate obtained through ViT-tuning. 2) CLIP’s zero-shot classification capability is insufficient to meet the requirements of various domains. The performance gap with federated prompt tuning is large, which indicates that the original CLIP lacks knowledge of the client domains. 3) Inserting a fully-connection layer after the image encoder of CLIP can remarkably enhance the performance on DomainNet, as indicated by CLIP-FC, which approaches the basic version of PromptFL. 4) Considering the performance of different methods and the amount of communication parameters in each round, FedAPT achieves the highest performance at a very low cost, which shows its great potential in the application of pre-trained CLIP to FL scenarios.

Supervised setting with domain and category differences. We split each domain in DomainNet into five clients, i.e., sample $\boldsymbol{\pi}_{c}\sim\operatorname{Dir}\left(\beta\mathbf{1}_{5}\right)$ and allocate a $\boldsymbol{\pi}_{c,i}$ proportion of the instances with label $c$ to the training set of the $i$-th client in each domain. Smaller $\beta$ leads to larger data distribution differences among clients and more imbalanced classes. Finally, we obtain 30 clients. In addition to the divergences in data domains, there are also divergences in category distribution among clients. In each communication round, we randomly select one client from each domain, meaning six clients are selected for training. Note that because the random seeds are the same, the clients selected in each round by different methods are consistent.

The results are shown in Table 2. As we can see, FedAPT still outperforms the competitors overall. Furthermore, it is worth noting that when dealing with extremely unbalanced categories, all CLIP tuning methods exhibit similar performance. This suggests that the pre-trained CLIP model needs to be augmented with additional designs to achieve better performance, as its existing capabilities may be insufficient.

Unsupervised setting. We replicate the above experiments in an unlabeled setting, and the results are shown in Table 3. Specifically, we observe that 1) On DomainNet dataset, the performance of unsupervised federated tuning is comparable to that of supervised tuning. This indicates that the unsupervised fine-tuning method proposed in Section 3.3 is effective. 2) Under the unsupervised setting, FedAPT still outperforms the baseline methods, demonstrating its significant effectiveness. In addition, as we use the class-shared prompts for FedAPT and PromptFL in this setting, the communication parameter sizes per round are 0.011M and 0.008M, respectively. This demonstrates the advantage of federated prompt tuning in terms of communication costs.

Impact of keys. We do a comprehensive study of the role of ‘key’. We disassemble the global model trained with FedAPT in Table 1 and compare its performance under different situations: 1) Use the learned meta prompt $\mathbf{p}_{g}$ to test without any key. 2) Use keys from different domains for testing, such as $\mathbf{p}_{g}+\mathbf{p}_{g}\odot\mathbf{e}^{\prime}_{k}$. 3) Use the complete APT module, i.e., Eq. 5. The results are reported in Tabe 4. The classification ability of the meta prompt $\mathbf{p}_{g}$ is shown in the first row. Because there is no suitable key, its performance is mediocre. Then the middle six rows show the performance of $\mathbf{p}_{g}$ with different keys. We can see that the $k$-th key $\mathbf{e}^{\prime}_{k}$ can significantly improve the accuracy in the $k$-th domain.

The affect of the adaptive network. We evaluate the performance of the global model in both client domains and unseen new domains. We introduce softmax temperature $\tau$ to the output of the adaptive network, and the performance at $\tau=0.01$ can be interpreted as directly determining “which key should be used”. The results in Table 6 show that directly selecting a single key ($\tau=0.01$) improves the performance in client domains but sacrifices generalization in unseen domains. By adjusting $\tau$, we can significantly enhance client performance while maintaining nearly unchanged generalization performance in unknown domains.

Other results. 1) One worrying point about using the key on the client side is whether the key will limit the learning ability of the local model. We verify this point with an experiment. We repeat the experiment in Table 1 and report the performance of local models from different domains in Table 5. The results show that the use of the key does not limit the learning of the local model. 2) We also compare different initialized keys. The results suggest that the keys work well for different types of random initialization, including the uniform and the standard normal distribution initialization and random orthogonal initialization. We provide the numerical results in the supplementary materials. 3) After precomputing and saving the text branch outputs of different keys, only the image branch and domain classification weights need to be computed during inference. The final inference cost is as follows: CLIP’s GFLOPs is 4.42 while that of ResNet50 is 4.14. 4) We demonstrate the impact in inference efficiency of the strategy in Figure 3. While maintaining nearly unchanged accuracy, the inference time has been reduced from thirty minutes to 30 seconds in one GPU card. Detailed numerical results are provided in the supplementary materials.