Enhancing Pre-trained Language Model with Lexical Simplification

A well-adapted LS process is the essential to our approach. Previous works Li et al. (2020); Jin et al. (2019) show that the prediction of PrLMs would be easily misled by replacing only a few words with their synonyms in the given sentences. By carefully observing the adversarial examples, we find that changing the tense of verbs, changing the singular and plural form of nouns, and replacing words by its less frequent synonyms compose the majority of the adversarial examples. The observation is also confirmed by Mozes et al. (2020).

Inspired by such observation, our LRLS method is developed with two major steps: (1) lemmataization by transforming verbs and nouns into corresponding lemmas, and (2) replacing rare words with theirs more common synonyms. Firstly, we employ Natural Language Toolkit (NLTK) to detect the verbs and nouns in the given sentences, and transform every verb to its infinitive form and every noun to its singular form. Secondly, according to a word frequency list¹¹1https://github.com/hermitdave/FrequencyWords, we label every word whose frequency is less than a frequency threshold $n_{f}$ as a rare word in the given sentence. We then use a word embedding from Mrkšić et al. (2016), which is specially curated for locating synonyms, to find the top $n_{s}$ synonyms of identified rare words with the highest cosine similarity. Each rare word is replaced by its synonym with the highest frequency. A part-of-speech (POS) check is also applied to ensure that all the synonymous candidates hold the same POS as the original words.

Following Devlin et al. (2018), the original sentence and its simplified version are combined together in to a single sentence. In our approach, the original and simplified sentences are differentiated in two ways. First, a special separation token ([SEP]) is inserted between the two sentences. Second, a learned segmentation embedding is added to every token which indicates whether it belongs to the original sentence or the simplified sentence. In both training and inference phases, we feed PrLMs the original-simplified sequence as inputs. The rest of implementations remain the same as the original PrLMs.

We conduct our experiments on five benchmark text classification tasks: (1) SST-2: Stanford Sentiment Treebank Socher et al. (2013), (2) CR: customer reviews Hu and Liu (2004); Liu et al. (2015), (3) SUBJ: subjectivity/objectivity dataset Pang and Lee (2004), (4) MR:Movie reviews Pang and Lee (2005), and (5) AG: AG’s News, classification task with regard to four news topics: World, Sports, Business, and Science.

We use (1) BERT-base Devlin et al. (2018) with 12 layers, 768 hidden units, 12 heads and 110M parameters, and (2) ELECTRA-large Clark et al. (2020) with 24 layers, 1024 hidden units, 16 heads and 340M parameters as our baseline PrLMs.

As shown in Table 1, we run both BERT-base Devlin et al. (2018) and ELECTRA-large Clark et al. (2020), with and without LS, across all five datasets. The average gain is 1.1 for BERT-base and 0.7 for ELECTRA-large. As ELECTRA-large is a very strong baseline, the result prove the effectiveness of our approach. As show in Figure 1, we select several examples from MR and SST-2 to further illustrate how PrLMs can benefit from the auxiliary input of simplified sentences.

Since our LRLS method is composed of two steps: transformation of verbs and nouns into their lemmas, and replacement of rare words. To investigate the impact of different LS methods, we firstly apply the two steps separately and compare with our LRLS method. We also include BERT-LS Qiang et al. (2020), which leverages masking language model of BERT to generate synonym candidates of rare words, for further comparison.

As shown in Table 2, the lemma transformation and rare words replacement are both effective, but we can further improve the performance by combining these two methods together. The performance of our method also exceeds that of BERT-LS. Moreover, our method is more than a hundred faster than BERT-LS, since our method is entirely rule-based, while BERT-LS uses a large pre-trained neural network to detect and replace the complex words recursively.

The process of the rare word replacement is controlled by two hyperparameters: $n_{f}$ and $n_{s}$. $n_{f}$ is the frequency threshold under which the word will be labelled as rare word and replaced. The larger the $n_{f}$, the more words will be replaced. $n_{s}$ is the number of synonym candidates. The larger the $n_{s}$, the larger possibility that the rare words will be replaced by more common but less similar candidates. In order to investigate the effect of these two hyper-parameters, we change these two hyperparameters separately and conduct experiments on MR and SST-2 to see the impact on the performance.

As shown in Figure 2, the best performance gain is obtained with middle-sized $n_{f}$ and $n_{s}$, which is consistent with our expectation. Because if $n_{f}$ and $n_{s}$ is too small the simplified sentence will be almost the same as the original version, on the contrary if $n_{f}$ and $n_{s}$ is too large, it may change the underlying meaning of the sentence.

We use simplified sentences as auxiliary inputs to improve the prediction accuracy of PrLMs. However, there are other frameworks to incorporate lexical simplification with PrLMs.

One alternative framework is to feed PrLM only the simplified sentences in both training and inference phases. In this case, prediction is made solely based on simplified versions.

Another framework is to leverage LS as a data augmentation technique. To illustrate, let $D=\{x_{i},y_{i}\}_{i=1\dots N}$ denote the training dataset. For a given sample $\{x_{i},y_{i}\}$ in the training dataset, we generate an augmented sample by simplifying the sentence $x_{i}$ to $x^{\prime}_{i}$ and preserving the label $y_{i}$. In this way, we generate an augmented dataset $D^{\prime}=\{x^{\prime}_{i},y_{i}\}_{i=1\dots N}$. PrLMs can thus learn from both the training set $D$ and the augmented set $D^{\prime}$.

Experiments are conducted to compared our framework with the two alternative frameworks mentioned above on BERT-base.

As show in Table 3, framework using simplified sentences as the only input (LRLS only) would slightly harm the performance of PrLM. This is because a part of semantic meanings carried by original sentences may be lost during the simplification process. Experiments also show that leveraging lexical simplification for data augmentation (LRLS Aug) is also beneficial for the overall performance. However, this framework would double the training time and the performance is still worse than our framework (LRLS Aux) .

While we leverage LRLS method to paraphrase the original sentence and generate auxiliary inputs for PrLMs, we wonder if other commonly used paraphrasing techniques are effective.

These paraphrasing methods include (1) random replacement of several words by their synonyms Wu et al. (2019); Wei and Zou (2019), (2) translating an existing example $x$ in language A into another language B, and then translating it back into A to obtain a paraphrased example $x^{\prime}$ (back-translation) Xie et al. (2019); Edunov et al. (2018) , and (3) randomly delete several words in the sentence (cutoff) Shen et al. (2020).

The upper mentioned paraphrasing methods are applied on original sentences respectively to generate auxiliary inputs, and then incorporated into PrLMs. Performance on MR and SST-2 from different paraphrasing methods are compared.

As show in Table 4, cutoff would slightly harm the overall performance. This is because it simply randomly deletes several words in the original sentence to generate a paraphrased version, which tends to twist the original semantic meaning and adds noise for predictions. Although back-translation and random replacement can slightly boost the performance of PrLMs, our LRLS method remains the most effective.