Effectiveness of Data Augmentation for Parameter Efficient Tuning with Limited Data

While large pretrained language models have achieved superior performance and widespread adoption across many NLP tasks (Zaheer et al., 2020; Bengio et al., 2021), they often contain hundreds of millions or even hundreds of billions of parameters, which significantly limits their application to tasks in which computation and storage resources are constrained.

To address this issue, an entire family of techniques called parameter efficient tuning (PET) methods have been developed. Most notably, deep prompt tuning (i.e., prefix tuning or P-tuning)  (Li and Liang, 2021; Qin and Eisner, 2021; Liu et al., 2021) has attracted extensive attention, which, compared to fine-tuning, only tunes trainable continuous embeddings, resulting in a tiny percentage of tuned parameters for each task. Consequently, a single language model can be used for multiple tasks by swapping out the trained prompts on a task basis (Li and Liang, 2021). Such success has also been shown by the state-of-the-art P-tuning v2 model (Liu et al., 2021), which yields performance comparable to fine-tuning on various natural language understanding tasks. In addition, Low-Rank Adaptation (LoRA) (Hu et al., 2021) has also emerged as an alternative approach, whereby the weights of rank decomposition matrices that are injected into each layer are optimized in lieu of the full network weights, reducing trainable parameters while also achieving improvements in accuracy over fine-tuning and other PET methods.

When training data is scarce in a task, data augmentation (DA) is a widely used strategy that can boost the performance of deep learning models (see  (Feng et al., 2021) for a survey on data augmentation in NLP). Then a basic question that needs to be answered is how effective is it when the above PET frameworks are applied in conjunction with data augmentation. To this end, we study three common task-agnostic DA methods, EDA (Wei and Zou, 2019), Back Translation (Sennrich et al., 2016), and Mixup (Guo et al., 2019a) with two common language models BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), trained using P-tuning v2 (Liu et al., 2021) and LoRA (Hu et al., 2021). We set up our study across five tasks, including 4 from SuperGLUE (Wang et al., 2019), in which the sizes of training data are small.

We show that data augmentation can increase the accuracy of prefix tuning models. However, the performance of each technique varies depending on the dataset and underlying pretrained model, and the effective techniques differ from fine-tuning. Certain methods can even lead to a notable degradation in performance, particularly when using larger models. To better understand the above phenomena, we visualize sentence representations of data under prefix tuning, a technique where many augmentation methods fail to get good results, to observe whether any limitations are present with how data is represented that are leading to poor performance in certain cases. Through this, we find that these models struggle more with separating the augmented embeddings from different classes. This observation is further supported by our additional experiments showing a lower robustness to heavily altered augmented data compared to fine-tuning, and an analysis of the cosine similarities between the sentence embeddings of augmented sentences and their original counterparts, where we see that prefix tuning produces highly similar embeddings irrespective of alterations. We seek to improve the performance when training on augmented data by adding a contrastive loss term  (Oord et al., 2018) to minimize distances between intra-class embeddings in P-tuning v2, helping mitigate the above issues with sentence representations, and resulting in some improvements in accuracy. We hope this empirical study helps facilitate future work on leveraging data augmentation when training transformer-based models using parameter efficient tuning methods.

A wide range of data augmentation techniques are used in NLP. Some general types of methods include rule-based techniques that use predetermined, rule-based transformations like synonym replacement that are not dependant on any model architecture (Feng et al., 2021). Example methods include EDA. Model-based techniques utilize language modelling to generate new data through means like seq2seq translation models as in back translation (Sennrich et al., 2016), masked language modelling using transformers on randomly masked inputs (Garg and Ramakrishnan, 2020), or using generative models like GPT-2 as in Anaby-Tavor et al. (2020) where a label-conditioned generation model is obtained by fine-tuning GPT-2 (Radford et al., 2019).

Despite widespread use in computer vision, DA is not as commonly used in the training of NLP models, largely due to the discrete nature of language making it difficult to apply perturbations without compromising the meaning, and the inconsistency in performance of different techniques (Feng et al., 2021). The effectiveness of DA on fine-tuned models has been studied previously. (Longpre et al., 2020) detail how BT and EDA could not generate consistent benefits on classification tasks with transformer-based models. Similarly,  (Okimura et al., 2022) expand this previous study and test the impact of 12 different DA methods and find little benefit when training on datasets with thousands of examples, but that there is some improvement in performance with very limited data (only a few hundred training cases). Our study conducts a similar examination on the effectiveness of data augmentation with pretrained models, however we focus on prefix tuning and LoRA, and we inspect how the properties of prefix tuning under DA differ from fine-tuning and what practical effect this has, particularly in terms of sentence representations.

We focus primarily on small datasets with less than a few thousand annotated data points, since limited data presents challenges in learning generalizable models and hence is a common use case for DA. Previous works have shown DA with pretrained language models tends to be largely ineffective for larger datasets (Okimura et al., 2022).

We test on five text classification datasets. Four are from the SuperGLUE benchmark (Wang et al., 2019), consisting of RTE (Dagan et al., 2006; Bar Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009), CB (De Marneffe et al., 2019), COPA (Roemmele et al., 2011), and WSC (Levesque et al., 2011). These are the smallest datasets in the benchmark at only 2500, 250, 400, 554 training samples respectively. They capture a range of difficulty, and include diverse problems like textual entailment, commonsense-reasoning-based coreference resolution, determining cause or effect, and clause commitment. To capture performance on a simpler single sentence sentiment analysis task, we also use a downsampled subset of SST-2 (Socher et al., 2013), consisting of 400 training samples and 100 validation sentences. We experiment with BERT-base (Devlin et al., 2019) and RoBERTa-large (Liu et al., 2019), which have 110 million and 355 million parameters, respectively. Compared to larger models, their size makes them suitable when storage concerns are present. Each model is trained until convergence and we report results at the epoch where the highest validation accuracy is achieved. We provide the detailed training settings for our models in Appendix A. Below we describe the settings for each of the data augmentation methods:

EDA: We apply a combination of the synonym replacement, random swap, random insertion, and random deletion operations to a certain percentage of the words in a sentence to generate multiple augmented sentences per original sentence. For the amount of augmented sentences generated, and the percentage of words affected, we follow the recommended usage guidelines laid out by Wei and Zou (2019) which are dependant on the size of the training set. 16 augmented sentences per original are generated for the smallest datasets; CB, COPA, WSC, and SST-2 since they only contain a few hundred training samples samples. The percentage of words in the original text affected for each of the 4 operations is 5% ($\alpha=0.05$). 8 sentences are generated per original for the RTE training set, with the same percentage of words being affected as the aforementioned datasets.

BT: For our implementation, we back-translate the sentences using a series of English to a target language and target language to English translation models for 4 common languages: French, Spanish, German, and Chinese. The specific models we use are the Helsinki NLP Opus translation modules (Tiedemann, 2020). These languages are chosen due to the substantial amount of parallel corpora available for training the respective language models, which yields higher quality translations that are better able preserve the semantics of the original sentence while still introducing some diversity.

Mixup: Inspired by Mixup in image classification (Zhang et al., 2018; Guo et al., 2019b), where random pairs of input images and their labels are linearly interpolated to help generated synthetic images, Mixup can be similarly applied over embeddings for text. As in Guo et al. (2019a), we adapt Mixup for text classification by interpolating on sentence embeddings (senMixup). In a mini batch of size $N$, we take the same number of random pairs of inputs texts, and interpolate over the $d$-dimensional (768 in the case of BERT-base and 1024 for RoBERTa-large) classification ([CLS]) tokens of the final hidden layer of the transformer encoder produced after each of the inputs is fed into transformer. The interpolated token is then passed into a fully connected later to generate the final softmax prediction vector. The interpolation for a given pair [CLS] token pair $(x^{i},x^{j})$ with corresponding one hot label vectors $(y^{i},y^{j})$, for inputs $i$ and $j$ is conducted as follows,

which results in interpolated vectors $\tilde{x}^{ij}$ and $\tilde{y}^{ij}$. Mixup is parameterized by mixing-ratio $\lambda$, which is different for every pair, and is obtained by sampling from the $Beta(\alpha,\alpha)$ distribution with the hyperparameter $\alpha=1.0$, corresponding to a uniform distribution. This was the recommended setting in Guo et al. (2019a) and consequently we use this value as well. Mixup is done in a similar manner for prefix tuning, except we use the pooled version of the classification token (i.e. one that has gone through a linear layer and tanh activation).

Procedure for Augmentation: For simplicity, since most of the datasets use multiple input sentences/words per training sample (e.g. premise and hypothesis), we apply the static DA methods over the primary (first) input, which is the premise for RTE, CB, and COPA, though we use the full sentence for WSC and SST2.

We reveal how when limited training data is available, data augmentation can provide crucial improvements in accuracy when training classification models using PET methods, though the benefits are, however, not universal, and limitations exist when using certain methods. We also demonstrated how prefix tuning may have more difficulty learning sentence representations of augmented data. We further showed that contrastive learning can be a solution to these issues, and suggest that more work be done to find similar methods that can be more generalizable.

Although we present our results across multiple datasets and models, we have largely adhered to natural language understanding tasks. How effective the augmentation methods are on generation-related tasks warrants further study.

In addition, another limitation is that we do not study the level of perturbations needed to have a negative effect on prefix tuning. Knowing this can shed light on what particular types of transformations are a problem for prefix tuning so that they can be avoided. Our hypothesis here is that such perturbation levels are sensitive to many factors during training, such as the difficulty of the task and the characteristics of the data augmentation approaches applied, which could be difficult to be quantified.

Any sort of technique where synthetic text data is generated in an automated fashion, especially with augmentation techniques like BT and EDA that can add or remove random words or completely rephrase sentences, can alter the original meaning or interpretation of a sentence. This has some potential to introduce additional unexpected biases into the training data, particularly when biased models are used to generate synthetic data (e.g. a racially biased translation model). Even with improved performance these biases can cause models to make unfair judgments, which can become a major concern in safety critical domains like in the legal or medical fields. As a consequence, before any data augmentation technique should be applied to PET techniques, there has to be a consideration of the underlying biases that can be introduced through the methods of producing synthetic data.

We would like to thank the anonymous reviewers for providing valuable feedback for polishing the paper. This work was supported by the Indigenous and Black Engineering and Technology (IBET) Momentum Fellowship and the NSERC Discovery Grants [RGPIN-2018-06415].