Back-Translated Task-Adaptive Pretraining:
Improving Accuracy and Robustness on Text Classification

Masked language model (MLM) proposed in Devlin et al. (2019) is a pretraining method that predicts original tokens in a sentence where some tokens are masked with a special token [MASK]. Let $X=(x^{(1)},x^{(2)},...,x^{(N)})$ denote a set of unannotated sentences, where $x=(t_{1},t_{2},...t_{M})$ is a sequence of tokens in a sentence, and $t_{i}\in x$ is a token in a sequence. In the pretraining process of the masked language model, noise is added to the sequences $x$ by randomly replacing some tokens with [MASK] token. Let $\hat{x}\in\hat{X}$ be a noised sentence and $\bar{x}\in\bar{X}$ be a subset of tokens that is randomly replaced. The training objective can be formulated as follows:

$$\displaystyle\min_{\theta}L_{MLM}(\theta;\hat{X},\bar{X})=-\sum_{\hat{x},\bar{x}\in\hat{X},\bar{X}}logp_{\theta}(\bar{x}|\hat{x}).$$

Because MLMs do not require any task-specific labels, they can be trained on a large unannotated corpus, which makes researchers believe they can learn a general representation of single or even multiple languages. This belief has been partially supported because pretrained MLMs based on a gigantic text corpus followed by fine-tuning on a small task-specific data often outperform the state-of-the-art models for a wide range of NLP tasks Liu et al. (2019); Lan et al. (2019); Song et al. (2019); Raffel et al. (2020).

LMs are pretrained with a vast amount of corpora from different domains such as BookCorpus Zhu et al. (2015), Wikipedia, CC-News Liu et al. (2019), OpenWebText Gokaslan and Cohen (2019), and Stories Trinh and Le (2018). Contrary to the expectation that LMs can properly generalize the language representation despite the text corpora based on which the LMs are trained, LMs have been reported that they are highly dependent on the training data domain. Moreover, their performances on other domains are not as good as on the training data domain.

To resolve this issue, language model adaptation, which re-pretrain the LMs before fine-tuning for specific tasks Sun et al. (2019); Gururangan et al. (2020), was proposed. There are two main streams in language model adaptation: domain-adaptive pretraining (DAPT) and task-adaptive pretraining (TAPT). DAPT re-pretrains the LM based on a new dataset on the same domain of a target task, whereas TAPT directly re-pretrain the LM using the given target task. If a task domain is uncommon or the data acquisition cost is expensive, TAPT can be more suitable than DAPT.

Data augmentation for NLP is less actively used than image augmentation thus far. While many simple label-unchanged operations exist for images such as flipping, rotation, cropping, and translation, languages barely have such operations. Hence, synonym replacement using WordNet has been the most popular augmentation method in NLP Wang and Yang (2015); Zhang et al. (2015).

Recently, easy data augmentation (EDA) techniques consisting of synonym replacement, random insertion, random swap, and random deletion proved useful for text classification Wei and Zou (2019). Despite the performance enhancement, the EDA has a significant limitation that the semantics of the original sentences are usually damaged except the synonym replacement Kumar et al. (2020). As an alternative to preserving the semantics of original sentences, back-translation, translating a sentence $\mathbf{x}_{source}$ in the source language into the other language and then translates it back into $\hat{\mathbf{x}}_{source}$ in the source language, was proposed. The technique showed to improve the performances in various NLP tasks, such as machine translation Sennrich et al. (2016a), question answering Yu et al. (2018), and semi-supervised text classification Xie et al. (2019).

While using TAPT contributes to task-specific performance improvement, Gururangan et al. (2020) showed that continued pretraining using human-curated unlabeled data – corpus from the same source with task data – ensured additional performance gain, which is named human-curated TAPT Simultaneously, the automatic selection approach retrieving unlabeled data aligning with the task data distribution from the in-domain corpus is also proven beneficial. Despite the favorable results, using those methods is impossible when human-curated data or in-domain data are unavailable. Motivated by this limitation, we propose an advanced adaptive pretraining approach requiring only task data.

If the amount of task data is insufficient during TAPT, the LM may still be underfitted on the task domain. In this case, the LM would be more generalized to the task if more task-related sentences were available. The proposed BT-TAPT is an additional adaptation method using human-like task-related sentences. To make plausible sentences, we use a back-translation using a nucleus(top-$p$) sampling. We used the nucleus sampling instead of traditional beam search because the former can better implement the purpose of augmentation than the latter, creating label-unchanged and semantic-preserved data that are not exactly the same as the original data. Although back-translated sentence using beam search tends to be an almost identical sentence to the original sentence, sentences generated by back-translation using the proper sampling method are usually paraphrases of the original sentence, which have various expressions without significantly deviating from the domain of the original data.

When the LM is further pretrained only with task data, noises applied to the input at every epoch are changing the location of the [MASK] tokens and infinitesimal random word transition. Otherwise, noise used in BT-TAPT includes not only the change of [MASK] positions or word transition but also synonym replacement or rephrasing facilitated by the sampling-based back-translation.

We first apply TAPT on the pretrained LM to expose it to the task domain sentence with correct grammar. Then, we generate multiple sentences with back-translation. We adopt nucleus sampling with $p=0.95$ to ensure proper diversity and fluency for the generated sentences. We generate 20 sentences for each original sentence to expose the LM to semantically and syntactically sufficiently diverse expressions. We further re-pretrain the LM with those sentences where domain-related but less accurate expressions appear. Learning from those paraphrases, the LM can cover a broader range of task distribution, as depicted in figure 1. After the entire adaptive pretraining phase is completed, the LM is fine-tuned on the task data.

Four sentiment classification datasets, i.e., IMDB Maas et al. (2011), MR Pang and Lee (2005), SST2 Socher et al. (2013), and Amazon He and McAuley (2016); Gururangan et al. (2020) and two other classification datasets, i.e., TREC Li and Roth (2002) question classification and AGNsews Zhang et al. (2015) topic classification, were used in the experiments. As a classification performance metric, a simple accuracy was used except for Amazon, for which the macro-$F_{1}$ is used owing to the class imbalance.

We divided the datasets into two groups: datasets with more than 20,000 training examples were considered high-resource datasets, while the others were considered low-resource datasets. To formulate additional low-resource datasets, IMDB and AGNews were down-sampled 2,500 per class, and Amazon was down-sampled 10% of the training dataset. The description of each dataset is summarized in Table 1.

As the pretrained LMs, BERT-base and RoBERTa-base trained from huggingface Wolf et al. (2020) ¹¹1 https://github.com/huggingface/transformers were employed. For back-translation, we employed the transformer-big model of Facebook Ng et al. (2019) ²²2 https://github.com/pytorch/fairseq/tree/master/examples/translation, a WMT’19 winner. We translated sentences from English to German and then back-translated them from German to English. When generating the translated sentence in the decoding process, we used the nucleus sampling with the sampling probability of $p=0.95$. The back-translation generated a total of 20 sentences for each original sentence.

For both TAPT and BT-TAPT, we re-pretrained the pretrained LMs 100K steps in the high-resource setting (IMDB, Amazon, and AGNews), following Sun et al. (2019). In contrast, in the low-resource setting, we re-pretrained 50K steps because the model converged faster with a small dataset than a large dataset. We used a batch size of 64, with a max sequence length of 512, and AdamW optimizer using a learning rate of 5e-5 with a linear learning rate scheduler for which the 10% of the steps were used for warming-up, and the rest of steps were used for decay. For task-specific fine-tuning, the modes were trained 2 or 3 epochs with a batch size of 8, 16, or 32, a weight decay of 0 or 0.01, and a fixed learning rate of 2e-5 with the same linear learning rate scheduler as re-pretraining. We stopped the fine-tuning when the validation loss starts to increase and reported the best performance among all hyper-parameter combinations. To compensate for the effect of random seed initialization, we repeated each experiment five times and reported the average and the standard deviation of the performance metrics. All the experiments were conducted using the Nvidia DGX-station with 4 $\times$ 32GB Nvidia V100 GPUs.

Tables 2 and 3 show the classification performance of each dataset in the high- and low-resource settings, respectively. In both tables, we can observe that further re-pretraining with the augmented dataset based on the back-translation improves the model either in terms of classification accuracy or the model stability regardless of the base pretraind LM. The proposed BT-TAPT yields either higher classification accuracy without the increase in standard deviation than TAPT (e.g., IMDB with BERT, SST2 with BERT, and MR with RoBERTa) or reduce the performance variation while maintaining the classification accuracy (e.g., IMDB with RoBERTa, AGNews with RoBERTa, SST2 with RoBERTa). Note that there are some exceptions: re-pretraining might harm the performance as mentioned in Sun et al. (2019) for those datasets with insufficient amount of sentences to generally represent the domain distribution, such as TREC dataset with RoBERTa and MR dataset with BERT. However, BT-TAPT seems to succeed to compensate the gap by supplementing additional augmented data.

Figure 3 shows the effect of the re-pretraining of LMs with task data and back-translated task data in the full IMDB dataset with the BERT model. The solid line denotes the classification performance after the fine-tuning is completed using the re-pretrained LMs with the corresponding training steps in the $x$-axis, and the shaded area is the standard deviation of accuracy for the five repetitions. With TAPT, the accuracy is increased along with the step size. However, the effect of TAPT vanishes beyond a certain step size. For example, the accuracy degenerates after 80K in Figure 3. Note also that the performance variation is also unstable with TAPT. When the proposed BT-TAPT is applied after 100K steps, the accuracy is rebounding along with the training steps. We can also observe that the accuracy not only further improves but also retains a low variability with BT-TAPT because a narrow bandwidth of the shaded area is retained.

We compared back-translation with other widely used augmentation methods – EDA, Embedding, and TF-IDF on adaptive pretraining. Embedding Mrkšić et al. (2016) replaces an arbitrary token with a close token in the embedding space. In contrast, TF-IDF Xie et al. (2019) replaces uninformative words with low TF-IDF scores in sentences while preserving those with high TF-IDF scores. As a baseline, we also include the model without any augmentation method, which means the TAPT is applied again with the same training steps for the other augmentation methods.

We first applied TAPT on BERT-base for 50K steps using the IMDB low-resource dataset and then re-pretrained using the augmented dataset generated from each method using a transformation probability of 0.1. Table 4 shows the classification accuracy for different augmentation methods. As mentioned regarding Figure 3, excessive TAPT often degenerated the final performance: another 100K steps of TAPT resulted in a $0.2\%p$ lower accuracy than that without additional re-pretraining. In addition, none of the benchmark text augmentation methods can neither succeed in improving the classification performance nor reduce the performance variation. Only the proposed BT-TAPT enhanced the classification accuracy without the loss of variation. The reason behind this observation is that the back-translation can generate diverse and realistic paraphrases, while the others tend to modify the sentence partially with a simple technique. These simply modified sentences sometimes harm the semantic meaning of the original sentence, which disturbs an appropriate LM re-pretraining.

To investigate the number of back-translation for the downstream task performance, we generated between 1 to 50 sentences in the BT-TAPT process from the IMDB low-resource dataset. Figure 4 shows the classification accuracy regarding the number of augmentations. In general, classification accuracy increases with TAPT as the number of back-translated sentences increases and becomes mature beyond a certain number of augmentations. Because the performance differences beyond 20 augmentations are marginal, we chose 20 as the final number of back-translated augmentations per sentence.

We investigated all possible strategies of back-translation deployment in BT-TAPT. TAPT & BT implies using both task and back-translated data for re-pretraining simultaneously. BT → TAPT and TAPT → BT refer to the re-pretraining with back-translated data and further re-pretraining with task data sequentially and vice versa. As seen in Figure 5, applying TAPT followed by re-pretraining with back-translated sentences yields the highest accuracy with the smallest variation. However, simultaneously using the back-translated sentences on re-pretraining enlarges the deviation even though it improves the accuracy, which implies that the language model should adjust to the target domain with a well-formed corpus first and then be exposed to various domain-related augmented sentences.

In a real-world situation, the task data is commonly insufficient. To validate that the BT-TAPT can effectively handle the data shortage, we conducted additional experiments with only 100, 500, and 1,000 sentences of the IMDB dataset. As shown in Figure 6, BT-TAPT is found to be much beneficial under an extremely low-resource circumstance because the performance is noticeably improved with only 100 sentences. Thus, this result supports that expression-diversified but semantic-preserved augmented texts based on back-translation can help LMs adapt to the domain distribution.

Five different realistic noise types were generated for the test datasets AGnews, SST2, MR, and TREC6. Note that these noises are added only to the test dataset, not the training dataset. Synonym   Replace the random word in the sentence with its synonyms using WordNet thesaurus (replacement probability $=0.1$). BT beam   Generate a paraphrase using back-translation with beam search. This method does not shift the original sentence substantially because the beam search does not employ sampling. Nevertheless, it produces more changes than the synonym replacement. BT top-p   Generate multiple paraphrases using back-translation with nucleus sampling. It creates diverse but partially incorrect sentences. Char swap   Generate random character-level noises with transformation probabilities of 0.1. The noise consists of deleting, adding, or changing the order of characters in a sentence. InvTest   Apply the invariance test proposed in Ribeiro et al. (2020). The invariance test applies label preserving-perturbations such as changing numbers or location names. The fine-tuned classifier should generate the same output whether the input changes by the invariance test.

Among the five noises scenarios, BT beam and BT top-p used English $\rightarrow$ German and German $\rightarrow$ English machine translation model adopted from Ng et al. (2019), which is the same model used in section 4. CharSwap, Synonym, and InvTest used the TextAttack Morris et al. (2020) python package. As randomness exists in all methods except for the BT beam, where the maximum probability is used for decoding, we created five different noised datasets for each dataset.

To evaluate the robustness of TAPT and BT-TAPT, we measured the accuracy gain, the amount of classification accuracy change on the noised test set after TAPT or BT-TAPT was applied. Figure 7 shows the accuracy gain regarding each dataset under different noise-added settings. Note that because the first two noise types, i.e., BT beam and BT top-p can generate similar sentences to BT-TAPT, BT-TAPT always improves the base re-pretrained model with a significant margin for most datasets, whereas TAPT sometimes fails to improve the base model performances. Beside these back-translation-based noises, BT-TAPT still reports favorable accuracy gains. However, TAPT not only achieves less significant accuracy gain than BT-TAPT, but also it degenerates the pretrained model (negative accuracy gain) even though it re-pretrained the base LMs. Consequently, we can conclude that the proposed BT-TAPT is more robust to unexpected data variations. Thus, the proposed method will be practically helpful in real-world tasks where languages are dynamically changing, evolving, and sometimes intentionally or unintentionally breaking.