Residual Prompt Tuning: Improving Prompt Tuning
with Residual Reparameterization

We propose to use a more flexible parameterization of soft prompts using a shallow network with a skip connection (Figure 1). Specifically, we project the sequence of prompt embeddings $P$ consisting of $n$ virtual tokens $[P_{1},...,P_{n}]$ into a reparameterized sequence $P^{\prime}$ as follows:

$$P^{\prime}=[P^{\prime}_{1},...,P^{\prime}_{n}]=[\Phi(P_{1}),...,\Phi(P_{n})],$$

(3)

where $\Phi(\cdot)$ is a reparameterization function composed of a shallow network $\phi(\cdot)$ with a residual connection. $\Phi(\cdot)$ is applied independently to each prompt token:

$$\Phi(P_{i})=\phi(P_{i})+P_{i},\;i\in\{1...n\}$$

(4)

Our $\phi(\cdot)$ network is a multi-layer perceptron (MLP) that follows a "bottleneck" design, as in commonly used ResNet building blocks (He et al., 2016) and adapter modules (Houlsby et al., 2019). It consists of down-projection $\mathbf{W}_{\text{down}}\in\mathbb{R}^{d\times m}$ and up-projection $\mathbf{W}_{\text{up}}\in\mathbb{R}^{m\times d}$ layers (as shown in Figure 2), a combination of which has been thoroughly explored in literature (He et al., 2016; Houlsby et al., 2019). Here, $d$ is the dimensionality of model embeddings and $m$ is the bottleneck size of the MLP (hyperparameter of our approach). We train only the prompt embeddings $\theta_{P}$ and the repremeterization parameters $\theta_{\phi}$ on the downstream task, while keeping all other parameters frozen. The training objective is to maximize the log-likelihood of correct output $y$ given the input text $x$ concatenated with the reparameterized soft prompt $P^{\prime}$:

$$\max_{\theta_{P},\theta_{\phi}}\sum_{x,y\in T}\log p_{\Theta}(y|[P^{\prime};x]).$$

(5)

We discuss here several important design choices for the reparameterization network $\Phi$.

Residual connection. We find that residual connection plays a key role in boosting performance and speeding up the convergence in Residual Prompt Tuning (Section 5.1, Appendix B.2). Similar to ResNets (He et al., 2016), we hypothesize that residual learning gives the model more flexibility to decide between using a separate embedding for each prompt token versus the representation obtained from the shared network. We discuss further benefits of residual connection in Appendix B.2.

Depth and width of MLP. We use two-layer MLP, whose up- and down-projection matrices $\mathbf{W}_{\text{up}}$ and $\mathbf{W}_{\text{down}}$ constitute the additional trainable parameters. Increasing the dimensionality $m$ of the hidden layer results in higher performance (see Section 5.6), suggesting that the overparameterization (Allen-Zhu et al., 2019) of prompt tokens is important for the performance improvement. More details on parameter-efficiency are in Appendix A.6.

Non-linearity and normalization. We select LayerNorm (Ba et al., 2016) as our normalization layer and ReLU as our non-linearity. We find that LayerNorm helps to stabilize the performance, while the effect of the specific choice of the non-linear layer is of lesser importance.

Parameter sharing. In our setup, we apply a shared reparameterization network $\Phi$ to each virtual token embedding. Another design choice is to apply a separate network to each prompt embedding. We compare both variants in Section 5.6. Overall, a shared MLP is significantly more parameter-efficient and offers the benefit of knowledge sharing in limited data settings.

During training, we jointly optimize prompt embeddings $P$ and parameters of the reparameterization network $\Phi(\cdot)$, while keeping the backbone model frozen. The reparameterized prompt is inserted before the input text embeddings and fed into the language model (see details in Section 4.2). Importantly, we use task-specific prompts, meaning that reparameterized prompt embeddings are not dependent on the input.

After training is complete, we project prompt embeddings through the learned reparameterization network $\Phi(\cdot)$, and replace the original prompt embeddings with their corresponding projections $P^{\prime}=\Phi(P)$. During inference, we discard the reparameterization network and solely use the projected prompt embeddings $P^{\prime}$. Specifically, we insert $P^{\prime}$ in front of the input text embeddings, and feed them together to the frozen pre-trained model.

Following previous works on prompt tuning (Lester et al., 2021; Vu et al., 2021), we use NLU tasks from the SuperGLUE benchmark to assess the performance of the language model (Wang et al., 2019). Specifically, we use the following 8 datasets: BoolQ (Clark et al., 2019), CB (De Marneffe et al., 2019), COPA (Roemmele et al., 2011), MultiRC (Khashabi et al., 2018), ReCoRD (Zhang et al., 2018), RTE (Giampiccolo et al., 2007), WiC (Pilehvar and Camacho-Collados, 2018) and WSC (Levesque et al., 2012). More details on are discussed in Appendix A.1,  A.2.

Residual Prompt Tuning is a model-agnostic approach that can be used with any transformer architecture – similarly to the original prompt tuning (Lester et al., 2021). In our experiments, we explore the performance of our method with encoder-decoder T5 model²²2While Lester et al. (2021) reports better performance with T5 v1.1 compared to T5, several works find version v1.1 less stable for prompt tuning compared to the original T5 and report worse performance (Karimi Mahabadi et al., 2021; Asai et al., 2022). Thus, in this work we use the original T5 model. (Raffel et al., 2020) and encoder-only BERT model (Devlin et al., 2018). Specifically, we focus on BERT-Base (110M parameters), T5-Base (220M parameters) and T5-Large (770M parameters) model variants.

BERT. For BERT experiments, we insert the trainable prompt in front of the input sequence, but before the $\mathtt{[CLS]}$ token, resulting in the following input $\hat{x}$ to the language model: $\hat{x}=\mathtt{concat[}\mathbf{E}(\mathtt{[CLS]}),P^{\prime},\mathbf{E}(S\mathtt{[EOS]})\mathtt{]},$ where $P^{\prime}$ is the embeddings matrix of the reparameterized soft prompt, $S$ is the input sentence, $\mathtt{[CLS]}$ and $\mathtt{[EOS]}$ denote special tokens (for sentence classification and marking end-of-sentence), and $\mathbf{E}$ denotes tokenization and extraction of embeddings.

To predict the class of input text $\hat{x}$, we follow the original (Devlin et al., 2018) setup and use encoder representation of the $\mathtt{[CLS]}$ token, $h_{\mathtt{[CLS]}}$, and add a linear transformation parameterized by $\mathbf{w}$ and a softmax layer to predict the class of $\hat{x}$:

$$p(y=c|h)=\frac{e^{\mathbf{w_{c}}h_{\mathtt{[CLS]}}}}{\sum_{k\in\mathcal{C}}{e^{\mathbf{w_{k}}h_{\mathtt{[CLS]}}}}}$$

After that, we apply cross-entropy loss to perform gradient updates on the prompt embeddings, linear head, and reparameterization network.

T5. For T5 experiments we cast all tasks as language modeling tasks, following Raffel et al. (2020); Lester et al. (2021). In this setup, we model the classification task as conditional generation, where output is a sequence of tokens that represent a class label. We prepend reparameterized prompt embeddings $P^{\prime}$ in front of the input text embeddings, hence total input $\hat{x}=\mathtt{concat}[P^{\prime},\mathbf{E}(S)]$ is passed into the pre-trained language model. T5 model applies a multi-headed self-attention over the input tokens followed by position-wise feed-forward layers to output a distribution over target tokens. We train prompt embeddings and parameters of the reparameterization network with cross-entropy loss. More details on input preprocessing and prompt initialization are in Appendix A.3,  A.4.

We compare Residual Prompt Tuning (Res PT) with approaches from two different categories: methods for prompt reparameterization and parameter-efficient tuning (PEFT) methods.

In our first set of experiments, we study how much residual reparameterization can improve prompt tuning performance and evaluate it against other reparameterization techniques. In sum, we compare our approach with the original prompt tuning (PT; no reparameterization Lester et al. 2021), prompt tuning with MLP reparameterization (PT w/ MLP; Li and Liang 2021), prompt tuning with LSTM reparameterization (PT w/ LSTM; Liu et al. 2021b) and fine-tuning.

In our second set of experiments, we assess the benefits of Residual Prompt Tuning method versus existing PEFT approaches. In addition to prompt tuning, we include a set of PEFT baselines: Adapter (Houlsby et al., 2019), AdapterDrop (Rücklé et al., 2020), SPoT (Vu et al., 2021), ATTEMPT (Asai et al., 2022). Adapter and AdapterDrop approaches are based on adapters by Houlsby et al. (2019), whereas SPoT and ATTEMPT are tranfer learning-based methods for prompt tuning, which find optimal prompt initializations by pre-training prompts on informative source tasks.

For all experiments with prompt tuning-based methods, we follow standard protocol by Lester et al. (2021) and report results on the validation set. Unless otherwise specified, we use standard metrics associated with each task to report final performance (see Table 7). For experiments where we compare Residual Prompt Tuning with PEFT methods (Section 5.1.2), we follow PEFT training protocol (Asai et al., 2022; Karimi Mahabadi et al., 2021). More experimental details are in Appendix A.5.

We study the performance of Residual Prompt Tuning across a wide range of learning rates (Figure 4). Previous works report that prompt tuning is very sensitive to the learning rate and requires extensive hyperparameter search to reach optimal performance (Lester et al., 2021; Vu et al., 2021).

We evaluate the performance of our proposed approach and prompt tuning (Lester et al., 2021) with learning rates from $\{0.001,0.01,0.03,0.3,10\}$ on SuperGLUE benchmark. For fair comparison, we use the most stable model variant: T5-Large model with 100-token prompt. Our results are shown in Figure 4. Notably, residual reparameterization allows stabilizing prompt tuning performance across a wide range of learning rates. Original prompt tuning often experiences fluctuations in its performance, with some tasks favoring lower learning rates (e.g. MultiRC), other tasks performing better with higher learning rates (e.g. CB), and yet other tasks achieving peak performance at a specific learning rate (e.g. WiC). In contrast to prompt tuning, Residual Prompt Tuning is robust to the choice of learning rate – it achieves strong performance with minimal fluctuations (less than $2$ points on average SuperGLUE score) with learning rates between 0.01 and 10 (over 100-fold variation).

Lester et al. (2021) finds that initialization of prompt parameters plays a major role in the final performance. Specifically, initializing prompt embeddings from sampled vocabulary embeddings can boost average SuperGLUE performance by up to $+10$ points compared to random uniform initialization (Lester et al., 2021). Here, we asked if Residual Prompt Tuning performance would depend on the choice of initialization.

Table 4 shows our results (initialization details are in Appendix A.4; we use T5B model with 10-token prompt). We can see that Residual Prompt Tuning is robust to the prompt initialization method, reaching comparable results with both initialization choices: $0.8$ points average performance difference between random uniform initialization and sampled vocabulary initialization. Of note, the initialization effect is more pronounced for smaller-scale dataset CB (250 samples) – random initialization attributes to $-0.3$ performance drop for Residual Prompt Tuning versus $-4.5$ score difference for the original prompt tuning.

We perform further experiments in few-shot settings (Figure 5). Specifically, we sample 5, 20, and 100 samples per class. To avoid variance due to selected samples, we fix the same training subset across all runs for each task; we use T5-Large model and 100-token prompt (as it reaches strongest performance for prompt tuning baseline). Residual Prompt Tuning is very effective in few-shot setup, boosting prompt tuning performance by $+7$ and $+2$ points on SuperGLUE benchmark with 5 and 20 samples per class.

We evaluate the Residual Prompt Tuning performance with smaller prompt sizes, and compare it to the original prompt tuning by Lester et al. (2021). Specifically, we explore the performance with prompts of lengths 2, 10, and 100 tokens with T5-Large model. Our results are shown in Table 5. In sum, Residual Prompt Tuning improves performance across all prompt lengths over prompt tuning, achieving average improvement of $+2.6$, $+1.1$ and $+0.8$ points with 2, 10, and 100-token prompts correspondingly.

Parameter sharing. We ablate the effect of a shared reparameterization network by assessing the performance when each prompt is reparameterized through a separate MLP with a skip connection (Table 6).

We select four SuperGLUE tasks of different sizes: small-scale CB and COPA (250 and 400 training examples), and larger-scale WiC and RTE (6,000 and 2,500 training examples). Interestingly, shared reparameterization network is beneficial in the low data regime, outperforming separate networks by $+2$ points on CB dataset. However, on larger datasets separate networks achieve slightly better performance at the expense of more trained parameters. We show more detailed results in Appendix C.1.

Overparameterization. To study the effect of overparameterization on the final performance, we ablate MLP width by varying the dimension of MLP hidden layer in the following range: $\{5,10,50,100,400,1500\}$ (Figure  6). Overall, we find that increase in dimensionality leads to performance gains, with performance saturating when the dimension reaches over 50 units.