HyperPELT: Unified Parameter-Efficient Language Model Tuning for Both Language and Vision-and-Language Tasks

As recent models grow rapidly in size, how to finetune pretrained models with a small number of trainable parameters becomes more crucial. Existing research (He et al., 2021; Lester et al., 2021; Liu et al., 2021a; Mao et al., 2021) have explored a large amount of methods on parameter-efficient tuning. These methods generally include two categories according to whether new trainable parameters are introduced. One category is that only a subset of model parameters can be updated while freezing the remain (Liu et al., 2021b; Lee et al., 2019). The other is introducing a few task-specific new parameters to different parts of pretrained models, such as before multi-head attention (Li & Liang, 2021), after feedforward layers (Houlsby et al., 2019) or Mixed-and-Match methods (MAM adapter) proposed by He et al. (2021).

In addition, fine-tuning language models pretrained on pure large text corpora have led to noticeable improvements to V&L tasks. This line of research such as VL-T5 (Cho et al., 2021) and Frozen (Tsimpoukelli et al., 2021) attempts to tune large language models (e.g. T5; GPT-3) to achieve transfer learning for V&L tasks. For example, Frozen aligns the image representation into the word representation space of frozen GPT-3 model which thus is able to generate captions for those images. PICa (Yang et al., 2021) utilizes a pretrained image captioner to convert the image into captions that GPT-3 can understand, and then adapt GPT-3 to solve the VQA tasks in a few-shot manner. Sung et al. (2021) introduces a limited set of new trainable parameters to VL-T5 via a adapter-based method that can match the performance of fine-tuning the entire model.

Learning a unified model to perform well on multiple different tasks (i.e., multi-task learning) is a challenging problem in both NLP and V&L domains. It has to address many challenges such as catastrophic forgetting, and model overfitting in low-resource tasks while underfitting in high-resource tasks (Aharoni et al., 2019). Radford et al. (2019) highlights the ability of language models to perform a wide range of multitasks in a zero-shot setting. As mentioned above, involving task-specific new parameters such as adapter (Houlsby et al., 2019), can be trained for each task separately while keeping the model fixed. von Oswald et al. (2020) propose a task-conditioned hypernetwork to generate all the weights for the targeted model, while Mahabadi et al. (2021) use a shared hypernetwork to only generate weights for a small number of parameters in adapter modules, to allow the model to adapt to each individual task efficiently.

All of our models are built on top of the state-of-the-art language model, T5 (Raffel et al., 2020), consisting of an encoder-decoder Transformer (Vaswani et al., 2017) with minor modifications. It frames language tasks as sequence-to-sequence generation, and is trained simultaneously on multiple task datasets. This large-scale T5 model achieves state-of-the-art performances across a diverse set of tasks. We use the T5 backbone as it enables training a universal model that interfaces with many downstream language tasks.

Our paper targets at a general multi-task learning problem, where we are given the data from a set of tasks $\{\mathcal{D}_{\tau}\}^{T}_{\tau=1}$. $T$ is the total number of tasks and $\mathcal{D}_{\tau}=\{(x^{i}_{\tau},y^{i}_{\tau})\}^{N_{\tau}}_{i=1}$ is the training data of the $\tau$-th task with $N_{\tau}$ samples. We are also given a large-scale pretrained language model, i.e., T5, parameterized by $\theta$, which generates the output $y^{i}_{\tau}$ for input $x^{i}_{\tau}$. The standard multi-task finetuning minimizes the following loss on the training set:

where $\mathcal{L}_{\text{task}}$ is the loss function of the tasks that is usually defined as the cross-entropy loss. Our goal is to efficiently finetune the given model in this multi-task learning setting, allowing knowledge sharing across tasks, and at the same time, enabling the model to adapt to each individual task.

The key idea of hypernetwork (Ha et al., 2017; von Oswald et al., 2020) is to learn a parametric task-specific hyper-embedding $\{I_{\tau}\}^{T}_{\tau=1}$ for each task. The hyper-embedding is fed to a hypernetwork $h(.)$ parameterized by $\theta_{h}$, which generates task-specific parameters $\Delta\theta=h(\theta_{h},I_{\tau})$ for other networks. In the multitask training, the hypernetwork is trained to capture the shared knowledge across tasks. It enables positive transfer among related domains and tasks, and mitigates the catastrophic forgetting problems to some extent. In this paper, we use a simple linear projection layer as the hypernetwork that takes hyper-embeddings $I_{\tau}$ as input, and outputs weights for subset networks of PLMs.

Considering a flexible parameterization of task-specific parameters for $L$ layers of transformer, we introduce a set of layer id embeddings $\mathcal{I}=\{l_{i}\}^{L}_{i=1}$, and block type embeddings $\mathcal{B}=\{b_{j}\}^{5}_{j=1}$, which specify the position where the parameters $\Delta\theta$ are inserted to. The five block positions to insert $\Delta\theta$ include the self-attention, feed-forward block of an encoder, and the self-attention, cross-attention, and feed-forward block in a decoder. The layer id and block type embeddings together determine the exact targeted transformer block, and are also used as input to the hypernetwork.

Then, we compute a hyper-embedding $I^{t}_{\tau}\in\mathbb{R}^{d_{I}}$ for each individual task via a task projector network $h^{t}_{I}(.)$. It is a multi-layer perceptron consisting of two feed-forward layers and a ReLU non-linearity:

where the task projector network $h^{t}_{I}(.)$ learns a suitable compressed hyper-embedding from a concatenation of task embeddings $z_{\tau}\in\mathbb{R}^{d_{t}}$, layer id embeddings $l_{i}\in\mathbb{R}^{d_{t}}$, and block type embeddings $b_{j}\in\mathbb{R}^{d_{t}}$. In this way, this hypernetwork is able to produce distinct weights for tuning each task, and each transformer block at each layer.

Prefix-tuning (Li & Liang, 2021) prepends a number of task-specific trainable prefix vectors to the parameters of multi-head attention (i.e., keys and values) at each transformer layer. In the original implementation, the prefix vectors of each attention block are reparameterized by a two-layer feed-forward network:

where $P\in\mathbb{R}^{d\times N}$, $W_{\text{down}}\in\mathbb{R}^{d_{\text{mid}}\times d}$, $W_{\text{up}}\in\mathbb{R}^{d\times L\times 2\times d_{\text{mid}}}$, $N$ denotes the prefix length, $d$ denotes the dimension size of PLMs and $E\in\mathbb{R}^{d\times N}$ is a randomly initialized embedding matrix. For prefix-tuning, the block type $\mathcal{B}$ has three: the self-attention block of encoder, the self-attention block and cross-attention block of decoder. Therefore, this original method indeed introduces quite a large number of parameters in the finetuning phase, due to the separate embedding matrix $E$ and feed-forward networks at each block.

In our method, we extend the dimension for different embeddings to match the prefix length $N$, i.e., $z_{\tau}\in\mathbb{R}^{N\times d_{t}}$, $l_{i}\in\mathbb{R}^{N\times d_{t}}$, $b_{j}\in\mathbb{R}^{N\times d_{t}}$, and then compute the hyper-embedding $I^{t}_{\tau}\in\mathbb{R}^{N\times d_{I}}$. We finally employ a hypernetwork $h^{t}_{P}(.)$ with trainable parameters $\theta_{h^{t}_{P}}$, to project $I^{t}_{\tau}$ to prefix vectors $P^{t}\in\mathbb{R}^{N\times d}$, named HyperPrefix:

In this way, since the hyper-embedding has already perceived information of which block at which layer, the number of hypernetwork parameters, i.e., $\theta_{h^{t}_{P}}$, can be largely reduced to $\mathbb{R}^{d_{I}\times d}$. We further explain the relationship between prefix-tuning and hypernetwork in the Appendix A.

To further capture knowledge across tasks and transfer to others, we follow the adapter-tuning (He et al., 2021), and input the hyper-embedding to a hypernetwork for generating the weights in adapters. As depicted in Figure 1, we introduce a hypernetwork-based adapter layer with a trainable scaled parameter $\lambda$, which is inserted parallelly with feed-forward blocks, named HyperPELT. This task-conditioned adapter layer $A_{\tau}$ consists of a down-projection, $W^{\tau}_{\text{down}}\in\mathbb{R}^{d_{\text{mid}}\times d}$, GeLU non-linearity, and up-projection $W^{\tau}_{\text{up}}\in\mathbb{R}^{d\times d_{\text{mid}}}$, where $h$ is the input dimension, $d_{\text{mid}}$ is the bottleneck dimension for the adapter layer, and $x$ is the input hidden state, mathematically defined as:

We generate adapter weights $(W^{\tau}_{\text{up}},W^{\tau}_{\text{down}})$ through a hypernetwork $h^{t}_{A}(.)$:

Note that in Section 4.2, we use the prefix length $N$ as the dimension for hyper-embeddings. We utilize an adaptive pooling operation on hyper-embeddings to adjust the dimension for adapter hypernetwork. Note that due to we extend the dimension of the components of hyper-embeddings in the last section, we utilize an adaptive pooling operation for hyper-embeddings to adjust the dimension for adapter hypernetwork.

Transferring pretrained language models to V&L tasks (Cho et al., 2021), with updating the whole parameters is inefficient. Thus, it motivates us to employ parameter-efficient tuning approaches to extend language models for V&L tasks. We follow Cho et al. (2021) to unify V&L tasks as a text generation problem. As illustrated in Fig. 1, we use CLIP (Radford et al., 2021) (parameterized by $\theta^{v}$) with a trainable visual projection layer (parameterized by $\theta^{v\rightarrow I_{v}}$), which projects the visual representation to the identical dimension of task embedding, i.e., $z_{\tau}\in\mathbb{R}^{N\times d_{t}}$. Then we feed this visual representation $v$ to a visual projector network $h^{v}_{I}(.)$, whose architecture is same to the mentioned task projector network $h^{t}_{I}(.)$. In this way, we learn the visual hyper-embeddings $I^{v}_{\tau}\in\mathbb{R}^{d_{I}}$. Finally, taking this visual-specific hyper-embeddings as input, we use a visual hypernetwork $h^{v}(.)$ to generate new visual-specific parameters to different modules in PLMs.

Similar to the Section 4.2 & 4.3, the incorporation of visual-specific parameters to PLMs are the same as task-specific ones, e.g., used as prefix vectors via a prefix hypernetwork $h^{v}_{P}(.)$ and adapter weights via an adapter hypernetwork $h^{v}_{A}(.)$. We name it VL-HyperPELT. Note that $I^{v}_{\tau}$ is also calculated as the Equation 3 via replacing task embedding $z_{\tau}$ by visual representation $v$. As illustrated in Fig. 1, the hyper-embeddings $I^{t}$ and $I^{v}$ separately produce weights for parallel adapters. In addition, their produced prefix vectors are appended together for multi-head attention modules.

Our framework is evaluated on the GLUE benchmark (Wang et al., 2019b) in terms of natural language understanding. This benchmark covers multiple tasks of paraphrase detection (MRPC, QQP), sentiment classification (SST-2), natural language inference (MNLI, RTE, QNLI), and linguistic acceptability (CoLA). The original test sets are not publicly available, and following Zhang et al. (2021), for datasets fewer than 10K samples (RTE, MRPC, STS-B, CoLA), we split the original validation set into two halves, one for validation and the other for testing. For other larger datasets, we randomly split 1K samples from the training set as our validation data and test on the original validation set.

In addition, we evaluate the few-shot domain transfer performance on four tasks and datasets: 1) the natural language inference (NLI) datasets CB and 2) the question answering (QA) dataset BoolQ from SuperGLUE (Wang et al., 2019a); 3) the sentiment analysis datasets IMDB (Maas et al., 2011); and 4) the paraphrase detection dataset PAWS (Zhang et al., 2019). For CB and BoolQ, since the test set is not available, we split the validation set into two halves, one for validation and the other for testing. For IMDB, since the validation set is not available, we similarly split the test set to form validation. For PAWS, we report on the original test set.

To evaluate our framework on V&L tasks, we experiment on four datasets COCO (Lin et al., 2014), VQA (Goyal et al., 2017), VG-QA (Krishna et al., 2017) and GQA (Hudson & Manning, 2019). We further evaluate our framework on three datasets for multi-modal few-shot transfer learning: OKVQA (Marino et al., 2019); SNLI-VE (Xie et al., 2018).

Our models are built on T5${}_{\text{BASE}}$ (Raffel et al., 2020) ²²2https://huggingface.co/t5-base and use the same tokenizer of T5 to tokenize text inputs. We set $N=49$, $d=d_{t}=768$, $d^{\text{mid}}_{I}=128$, $d_{I}=64$ for all the experiments. Following the training strategies from Raffel et al. (2020), we fine-tune all models with a constant learning rate of 0.001, use $2^{18}=262144$ steps in all experiments with batch size of 128 and sample tasks via the conventional temperature-based sampler, with temperature $T=2$, i.e., sample corresponding task proportional to $p^{1/T}_{\tau}$, where $p_{\tau}=\frac{N_{\tau}}{\sum^{T}_{i=1}N_{\tau}}$ and $N_{\tau}$ is the number of training samples for the $\tau$-th task. We did not experiment with other complex sampling strategies or tuning of $T$. For the experiments under multi-task training settings, we save a checkpoint every 1000 steps and report results on a single checkpoint with the highest average validation performance across all tasks.

For evaluating our framework on vision-language scenarios, we follow Cho et al. (2021) to convert V&L tasks to a text generation format. We use ResNet101 as our vision encoder, and initialize it with CLIP (Radford et al., 2021) ³³3https://github.com/openai/CLIP pretrained weights. Input images are resized to 224 $\times$ 224 for the memory efficiency. We extract the 7 $\times$ 7 grid features produced by the last convolutional layer. The percentage of updated parameters is also reported as one metric for approach efficiency, and we do not take visual encoder into computation since it is frozen in our experiments. We count the number of tunable parameters and list the input-output formats of each task in the Appendix B and C.

We conduct experiments on GLUE for both single- and multi-task settings. As shown in Table 1, we also report the total number of parameters and trainable parameters for all models. Our methods are built on T5${}_{\text{BASE}}$ which contains 12 layers and 222M parameters. For adapter-related methods, we experiment with reduction factors of $r=32$ (Mahabadi et al., 2021), where $r=\frac{d}{d_{\text{mid}}}$. We follow the original T5 implementation (Raffel et al., 2020), which is slightly different from HYPERFORMER++ (Mahabadi et al., 2021). In our re-implementation of T5${}_{\text{BASE}}$ and HYPERFORMER++, we prepend task-prefix tokens to the textual input sequence as the original T5 does. We also experiment with generating weights for layer normalization via a hypernetwork. However, we find that it makes no difference on performances.

For Prefix-tuning (Li & Liang, 2021) and MAMAdapter (He et al., 2021), their original implementation is single-task training on BART (Lewis et al., 2020). To make a fair comparison to other baselines, we apply their methods to T5 in a multi-task training setting. ⁴⁴4For adapting Prefix-tuning from BART to T5, a noteworthy point is that since they use different position embedding, i.e., absolute position embedding for BART and relative position embedding for T5, it is necessary to manually concatenate all-zero vectors on the relative position bias of each layer in T5. For each model, we share the parameters of both prefix vectors and adapter weights across multitasks.

Overall, our HyperPELT method obtains the best performance with less trainable parameters. Compared to the single-task Adapters that finetunes all the introduced parameters in adapters, our method yields a significant improvement by 2.21% with much fewer trainable parameters, which illustrates the effectiveness of our proposed multi-task training framework.

In multi-task training, the proposed hypernetwork-based prefix-tuning strategy, e.g., HyperPrefix, decreases the number of trainable parameters (e.g., 1.01$\times$ of HyperPrefix vs. 1.14$\times$ of Prefix-tuning), while achieves a better performance at the same time (e.g., 86.65% of HyperPrefix vs. 86.09% of Prefix-tuning). It is noticeable that the number of trainable parameters per task is 11$\times$ fewer than Prefix-tuning.

HyperPELT obtains a superior performance over HyperPrefix, and the main reason lies in that we further combine the hypernetwork-based adapters and add them to the feedforward layers in a parallel manner. In this way, the average performance is further enhanced (+0.44%) by involving a small number of parameters (0.09% parameters per task). The comparison to MAMAdapter shows that using hypernetwork to tune each transformer block and learn the shared knowledge across multitasks leads to an improvement.

We use the above models trained on GLUE as reported in Table 1, and evaluate them on the test set of five different tasks after being few-shot finetuned on each target training data. Following Mahabadi et al. (2021), we use the task embedding respectively trained on the most similar GLUE task for initialization, i.e., MNLI for CB, QNLI for QA, SST-2 for sentiment analysis, and QQP for paraphrase detection. As suggested by Perez et al. (2021) and Zhao & Schütze (2021), we randomly select the same number of samples from training and validation set, making it a reasonable few-shot scenario. Checkpoints are selected via early stopping on the selected validation set, and the stopping metric is the default metric for each task.

In the first three columns of Table 2, we show the results of full fine-tuning of T5${}_{\text{BASE}}$, HYPERFORMER++ (finetuning both hypernetworks and task embeddings) and our proposed HyperPELT. Overall, our method achieves the best performance in the few-shot tasks.

For the tasks of CB and BoolQ from SuperGLUE, even though the backbone T5 was previously trained on the train sets of these two, the performance of all methods differs a lot. The two baselines still do not work with very few samples, like 4 and 16 samples, while our method is significantly better than them. Therefore, we assume that the two baselines suffer from catastrophic forgetting problems to some degree during multi-task training. In contrast, our proposed HyperPELT works effectively on these two tasks. We speculate that the reason might be the use of hypernetworks on both prefix-tuning and adapter-tuning modules of transformer. We leave this exploration to our future work.

Besides, in the last two columns of Table 2, we show the results of Prompt-tuning (Lester et al., 2021) and fine-tuning only the task embedding in our HyperPELT. Note that in this comparison, we keep the same trainable parameters between these two methods, i.e., $\mathbb{R}^{N\times d_{t}}$, where $N$ denotes the prompt length in Prompt-tuning method. Our HyperPELT TaskEmbed mostly achieves a comparable or even better performance than Prompt-tuning except TREC task, where the task format and verbalizer are quite different from previous tasks. In this case, only tuning the very limit parameters of our model with very few samples is unable to quickly transfer. In comparison, HYPERFORMER++ (Mahabadi et al., 2021) claims only fine-tuning the task embedding in their model achieves low performance in the few-shot learning and they have not reported any result. Our framework with only fine-tuning the task embedding, though not perfect on all tasks, may still present a new promising parameter-efficient way towards few-shot learning with an extreme low number of tunable parameters (Sanh et al., 2021).

Next, we move to the experiments of applying the proposed hypernetwork-based parameter-efficient training framework to V&L tasks. We compare to the pre-trained and full fine-tuning VL-T5 (Cho et al., 2021), and other adapter-based methods built on top of T5, i.e., CLIP-T5 and VL-Adapter (Sung et al., 2021) in the multi-task training setting.

The results and the number of trainable parameters are reported in Table 3. To match the same amount of trainable parameters as VL-Adapter, we experiment with the reduction factors $r=16$, where $r=\frac{d}{d_{mid}}$. Since the test dataset is slightly different from Sung et al. (2021) and their checkpoint is not avaliable at this time, we re-implement CLIP-T5 and VL-Adpater. Compared to which, our method achieves a comparable performance with fewer number of trainable parameters (e.g., 7.16% of VL-Adapter vs. 6.62% of VL-HyperPELT).

To our best knowledge, we are the first to employ the visual modality to tune the very few parameters of different transformer blocks, instead of normally inserting image patch tokens to the input sequence. Experimental results evidence the effectiveness of our novel approach, thus providing a new perspective on how to extend the multi-modality capability on top of PLMs. It is to use the features from different modalities as the input of hypernetwork to generate parameters for modules in PLMs, instead of as a part of input sequence to accomplish the multimodal tasks. One advantage in our approach is still keeping the original maximum text input length, since no other modalities such as visual and audio features occupy it. It is promisingly useful in document-level and text-heavy tasks such as multimodal summarization (Zhang et al., 2022).

We believe the resulting performance might be even better with a more complex design combination of methods across tuning task-specific and visual-specific parameters in PLMs, but we leave this exploration in future work.

We further use the models trained on V&L tasks as reported in Table 4 and evaluate them on the test set after few-shot fine-tuning on OKVQA (Marino et al., 2019) and SNLI-VE (Xie et al., 2018). For OKVQA, since there is no test set, we split its original validation set into two halves, one for validating and the other for testing. For SNLI-VE, we use its validation set for validating, and test-P set for testing and reporting results. We follow the methods in Section 6.2 to select samples, and report results in Table 4.

Compared with the full parameter fine-tuning, i.e., CLIP-T5, and the other baseline VL-Adapter, our method achieves the best performance with smaller variances in this multimodal few-shot learning setting. We find that VL-Adapter is inferior to CLIP-T5 when with fewer samples (e.g., fewer than 500) on the OKVQA dataset. The reason may be that there exists a lot of out-domain knowledge and complex image content in OKVQA, which makes it more challenging for parameter-efficient VL-Adapter to achieve accurate prediction. In other words, the small number of samples are not enough to train the introduced randomly initialized parameters in VL-Adapter.

However, our approach can still tackle with fewer samples. We use the hypernetwork to generate trainable parameters in adapters and multi-head attention, as well as directly integrating image features into attention modules in the form of prefix tuning vectors. We believe such method, though training less parameters, can still capture knowledge across tasks and transfer them in a few-shot setting. It is also worth noting that for the used five random seeds, the variance of our method is generally smaller than VL-Adapter, which indicates that our method is more robust in this few-shot learning scenario.