Chinese Spelling Correction as Rephrasing Language Model

Chinese Spelling Correction (CSC) aims to correct all misspelled characters in the source sentence. Given a source sentence $X=\{x_{1},x_{2},\cdots,x_{n}\}$ of $n$ characters with potential spelling errors, the model seeks to generate the target sentence $Y=\{y_{1},y_{2},\cdots,y_{n}\}$ of the same length with all potential errors corrected. The above process can be formulated as a conditional probability $P(Y|X)$. Specifically for $x_{i}$, suppose that it is an error character and its ground truth is $y_{i}$, then the probability to correct $x_{i}$ to $y_{i}$ can be written as $P(y_{i}|X)$.

Sequence tagging is a common model in many natural language processing tasks, where the model is trained to map one character to another correspondingly, e.g. named entity recognition, part-of-speech tagging. All these tasks share an pivotal property in that they strongly rely on the alignment information between input and output characters. However, deep neural models like Transformer (Vaswani et al. 2017) are always good at exploiting spurious clues if it is possible to achieve lower training loss, especially when the training data is not big enough (Wu et al. 2023a, b). For example, Norway is always a geopolitical entity (GPE) in entity recognition. Consequently, the model can memorize such a character-to-character mapping and make correct predictions in most situations. Similar in CSC, the model can greatly memorize the trivial edit pair of correcting $x_{i}$ to $y_{i}$ and continue to apply it in a different context of $x_{i}$, without referring to the semantics. Hence, the original training objective degenerates to:

$$P(y_{i}|X)\approx P(y_{i}|x_{i})$$

(1)

where we suppose $x_{i}$ is an error character.

However, the errors in CSC are much more diverse. As previously shown in Figure 1, age to can be both corrected to remember to or not to, which significantly relies on the immediate context. The resultant tagging model can hardly be generalized to unseen errors.

In this paper, we propose to substitute sequence tagging with rephrasing as the primary training objective for CSC.

Our intuition is to eliminate the trend that the model fits the training data by naively memorizing the errors. To do this, we train the pre-trained language models (PLMs) to rephrase the source sentence following it. Concretely, the Transformer layers first transfer the source sentence to the semantic space. Then, the model generates a new sentence while correcting all errors in it, based on the semantics. This process is consistent with the human process of doing spelling correction. When a person sees a sentence, he will first commit the sentence to his mind (akin to encoding it to the semantic space), and then transform the semantics to a new sentence based on his linguistic instinct (the pre-trained weights in the PLM). We see that the pre-training knowledge offers a great foundation for such learning rephrasing, which is aside from learning sequence tagging. Our following experiments show that sequence tagging does not make good use of the benefits from pre-training.

The process of rephrasing can be modeled based on the auto-regressive architecture with a decoder to generate the output characters one by one, e.g. GPT (Brown et al. 2020). Specifically, we concatenate the source characters $X$ and the target characters $Y$ as the input sentence, i.e. $\{x_{1},x_{2},\cdots,x_{n},\left<\rm s\right>,y_{1},y_{2},\cdots,y_{n},\left<\rm eos\right>\}$, where $\left<\rm s\right>$ and $\left<\rm eos\right>$ refers to the separate token and wrap token, and train the model to predict all the target characters $y_{i}$ auto-regressively. Hence, rephrasing-based spelling correction seeks to solve the following probability for $y_{i},i>=1$:

$$P(y_{i}|X)\approx P(y_{i}|X,y_{1},y_{2},\cdots,y_{i-1}).$$

(2)

Based on the BERT-based architecture, we propose Rephrasing Language Model (ReLM), a non-auto-regressive rephrasing model.

Eq 2 generates the sentences free of lengths. Given that the length of the target sentence is fixed in CSC, equal to that of the source sentence, such freedom may bring a negative impact. BERT is an encoder-only architecture (Devlin et al. 2019), pre-trained by randomly setting a portion of characters to a mask symbol $\left<\rm m\right>$. In contrast to the auto-regressive model, which keeps generation until $\left<\rm eos\right>$, BERT is programmed to only infill the pre-set slots of mask.

As shown in Figure 2, we concatenate the source characters $X$ and a sequence of mask characters $M=\{m_{1},m_{2},\cdots,m_{n}\}$ of the same length as the input sentence, i.e. $\{x_{1},x_{2},\cdots,x_{n},\left<\rm s\right>,m_{1},m_{2},\cdots,m_{n}\}$, where $m_{i}$ refers to the mask character for $y_{i}$, and train the model to infill all the mask characters $m_{i}$ following $X$. Since BERT can see both the left-side and right-side context, ReLM seeks to solve the following probability for $y_{i},i=1\sim n$:

$$P(y_{i}|X)\approx P(y_{i}|X,m_{1},m_{2},\cdots,m_{n}).$$

(3)

ReLM is superior to the auto-regressive model as it can always generate the output sentence of the same length as the input, which makes it more accurate. In our following experiments, we find that both auto-regressive rephrasing and ReLM outweigh previous tagging models, and the latter achieves more powerful results.

The emerging large language models (OpenAI 2023; Touvron et al. 2023) tend to handle diverse tasks at the same time, and it is time to study the incorporation of CSC into other tasks. In typical multi-task learning, we add a specific classification head for each task and train a shared encoder for all tasks. For instance, CSC will share the same language representation within the encoder with sentence classification. However, our empirical analysis shows that the performance of conventional tagging-based CSC may largely deteriorate when it is jointly trained with other tasks. The corresponding probing is in the following analysis section.

In contrast, ReLM, still a language model as its core, naturally suits the multi-task learning on top of language modeling, while tagging-based CSC does not. Concretely, each individual task is templated to the format of masked language modeling, as shown in Figure 2. In general, all tasks are unified to a rephrasing-like format, which enhances the transferablity of CSC to various tasks. In addition, ReLM supports prompt tuning (Lester, Al-Rfou, and Constant 2021; Liu et al. 2021b). We prefix a sequence of trainable characters to the input sentence as the prompt steering the model for different tasks, and optimize the corresponding prompt for each task. We find that introducing prompts can further improve the outcome, but to a slight extent.

We fine-tune each model separately on the three domains for 5000 steps, with the batch size selected from {32, 128} and learning rate from {2e-5, 5e-5}.

Table 1 summarizes the results on ECSpell. We find that naive tagging models of BERT and GPT2 perform poorly on all of three domains, while BERT performs slightly better than GPT2. However, ReLM yields amazing performance improvement. Concretely, it achieves the new SotA result on every domain, significantly outperforming the previous SotA masked-fine-tuned MDCSpell by 10.6, 12.8, and 16.7 absolute points respectively. We also apply the rephrasing objective to GPT2. We find that even GPT2-rephrasing outperforms BERT-tagging by a large margin (e.g. F1 39.8 $\rightarrow$ 71.2 on LAW), demonstrating the great superiority of the rephrasing objective, which prevents the model from simply memorizing errors.

However, we see ReLM is more powerful. It indicates that fixed-length rephrasing is naturally matched with CSC, while the auto-regressive one is also promising for future study. On the other hand, it is worth noting that ReLM surpasses all other enhanced architectures by simply training based on the original Transformer architecture. It highlights the pivotal role of the rephrasing objective, while the common tagging objective does not exploit the full power of PLMs, incurring the performance bottleneck.

We report the results of Baichuan2 and ChatGPT as representatives for LLMs. We find that ChatGPT does not work well on CSC even in a 10-shot setting. We speculate that this is due to the lack of high-quality annotated data for CSC on the web. However, fine-tuned Baichuan2 achieves promising results, outperforming GPT2-rephrasing by a large margin.

On LEMON, we evaluate models as a zero-shot learner. Following Wu et al. (2023c), we collect 34 million monolingual sentences and synthesize training sentence pairs using the confusion set. We train the model with the batch size 4096 and learning rate 5e-5 on 8 A800 sheets for 60,000 steps.

Table 2 compares the zero-shot performance of ReLM to previous SotA tagging models. We find that, though each LEMON domain varies greatly, ReLM brings a significant performance boost in almost every domain, and reaches the new SotA results, raising the previous SotA from 49.7 to 51.2. It indicates that ReLM is more generalizable to out-of-distribution errors over all other BERT-tagging variants.

We train the multi-task model on three distinct tasks, ECSpell for CSC, AFQMC for semantic similarity, and TNEWS for news classification. The three datasets are mixed together and we uniformly sample one batch from them during training. We fine-tune each model on all tasks for 15 epochs, with the batch size selected from {32, 128} and learning rate from {2e-5, 5e-5}. For the tagging models, we train three task-specific linear classifiers and one shared encoder. For ReLM, we share the entirety of model parameters for all three tasks. For ReLM with prompt embeddings, we train an additional prompt embeddings for each task. The MFT technique is exclusively applied on CSC.

Table 3 compares the results on multiple tasks. We find that the performances of two text classification tasks vary only marginally between multi-task and single-task settings. We speculate that these two tasks are less challenging, and the model can fit them well more easily. In contrast, the performance of CSC is strongly affected by other tasks, where both tagging models meet a great performance drop. However, ReLM can largely maintain the CSC performance and achieve competitive results on all three domains, almost without compromising other tasks. Adding additional prompt characters can further improve the performance. It suggests that ReLM contributes to better collaboration between different tasks, on top of templating all tasks to the MLM format, while tagging-based CSC is incompatible to such a training paradigm. The logic behind is that ReLM retains the useful features within the pre-trained language representation of PLMs. In our following analysis, we show that tagging-based CSC learns non-transferable features.

False positive rate (FPR) is a measurement to evaluate a CSC system in real-world applications, which refers to the ratio that the model mistakenly modifies an otherwise correct sentence, which is also known as over-correction. Table 4 shows that ReLM greatly reduces the FPR compared to tagging models. It suggests the tagging models are overly conditioned on the seen errors and thus tend to modify some new expressions to familiar ones, while ReLM does not. Additionally, we find that ReLM produces even lower FPR in multi-task learning. It indicates that by ReLM, the language representation learned from CSC and other tasks can complement each other, while sequence tagging cannot.

We further demonstrate that a high FPR may result in a gap between the development performance and real-world practice. Mathematically, we have $1/{\rm p}\propto\frac{N}{P}\cdot{\rm FPR}$, where $N$ and $P$ refer to the number of negative samples and positive samples, and ${\rm p}$ is the precision score. We can find that $p$ is negatively correlated with the ratio $\frac{N}{P}$, which means more negative samples lead to lower precision under the same FPR. However, negative samples are dominant in real-world situations ($\frac{N}{P}$ is large), since humans do not misspell very frequently. We can derive similarly results for the F1 score. Consequently, a higher FPR may exacerbate the decrease of the overall performance of the CSC system.

Figure 3 depicts the variation of the precision and F1 with the proportion of positive and negative samples, comparing MDCSpell-MFT to ReLM on ECSpell-ODW. We find that both F1 and precision curves of ReLM are more gentle, which decrease more slowly with the increase of $\frac{N}{P}$. It highlights the practical value of ReLM for real applications.

To investigate the transferability from CSC to other tasks, we perform a linear probing experiment (Aghajanyan et al. 2021). First, we fine-tune the model on CSC data (ECSpell). Second, we freeze the parameters of the encoder and initialize a new linear classifier following it. We fine-tune this classifer only on another task (TNEWS). The results of the linear probing reflect whether the learned features within the encoder are generalized to transfer to new tasks.

From Table 5, we find that both tagging models suffer from a severe drop when transferring their learned language representation from CSC to TNEWS. It suggests that sequence tagging does not learn any generalized features from CSC, even degrades the language representation of the original PLM. In contrast, we find that ReLM can transfer much better, suggesting that it retains generalized features within the language representation during fine-tuning.

We investigate two mask strategies of the auxiliary MLM when training ReLM. The first is to uniformly mask any characters in the sentence and the second is to mask non-error characters only. From Table 6, we can see that both mask strategies are effective, while masking non-error characters works better. This is because masking error characters can reduce the amount of the real errors within the training data, which the model needs for learning correction. Additionally, we find that adding learnable prompts can further improve the performance of ReLM. The results in Table 6 and Table 7 are based on ReLM with learnable prompts.

We also investigate the impact of the mask rate. From Table 7, we find that the performance on ECSpell keeps improving when the mask rate grows from 0% to 30%, and 30% is the best choice. While it still relies on specific data, it shows that the mask rate for ReLM is higher than that for MLM (Devlin et al. 2019). This is because ReLM is essentially a further refinement of PLMs, which allows a higher mask rate.

We illustrate the superiority of ReLM over tagging through a number of cases selected from the evaluation results of masked-fine-tuned BERT-tagging and ReLM in Figure 4.

For the first case (Director of People’s Hospital in Daxingan Ridge.), BERT-tagging overly corrects a geopolitical place “Daxingan Ridge” to “Daxingan Mountain”, which is due to the fact that it doggedly memorizes a previous edit pair “ridge” $\rightarrow$ “mountain” during training. However, we can see that ReLM does not make this mistake.

For the second case (Worried it develops towards muscle and bone.), it highlights the ability of ReLM to utilize the semantics of global context. Two expressions “like” and “towards” are all locally correct, while to reach the correct result, the model should refer to the word “develop” located at the end of the sentence (the Chinese order).

The third case is quite puzzling (Judicial power is not execution, but judgemental power.), especially the first error. The correct answer not only necessitates the semantics but also a legal principle that “judicial power is judgemental power”, which can only be attained through the pre-training process. We find that the tagging model does not possess such expertise and its answer is “law enforcement power is judgemental power”. It suggests that ReLM effectively inherits the knowledge of PLMs, while the tagging model does not even enhanced with masked-fine-tuning.