Improving Chinese Spelling Check by Character Pronunciation Prediction:
The Effects of Adaptivity and Granularity

Similar to recent CSC approaches that leverage multimodal information (Liu et al., 2021; Xu et al., 2021), we use ChineseBERT (Sun et al., 2021) as the encoder to extract semantic, phonetic, and morphologic features as well for the CSC task. ChineseBERT is a pre-trained language model that incorporates both the pinyin and glyph information of Chinese characters. Specifically, for each character $x_{i}$ in the input sentence $X$, the encoder first produces its char embedding, pinyin embedding, and glyph embedding, all with embedding size $D$. These three embeddings are then concatenated and mapped to a $D$-dimensional fused embedding via a fully connected layer. After that, just like in most other pre-trained language models, the fused embedding is added with a position embedding, and fed into a stack of successive Transformer layers to generate a contextualized representation $\bm{h}_{i}\in\mathbb{R}^{D}$ for the input character $x_{i}$. We denote the character representations after this encoding process as $\bm{H}=$ $\{\bm{h}_{1},\bm{h}_{2},\cdots,\bm{h}_{n}\}$. As the encoder is not the main concern of this paper, we just provide a brief sketch of the encoder and refer readers to (Vaswani et al., 2017; Sun et al., 2021) for details.

This decoder is to predict the characters in the correct sentence $Y$ based on the encoding output $\bm{H}$. Specifically, given each input character $x_{i}$, we first project its encoding output $\bm{h}_{i}$ into a character-specific feature space:

and then predict the corresponding correct character $\hat{y}_{i}$ based on the projection output:

Here $\bm{W}^{(c)}\in\mathbb{R}^{D\times D}$, $\bm{b}^{(c)}\in\mathbb{R}^{D}$ are learnable parameters of the character-specific feature projection layer; $\bm{W}^{(y)}\in\mathbb{R}^{V\times D}$, $\bm{b}^{(y)}\in\mathbb{R}^{V}$ are learnable parameters of the character prediction layer; $V$ is the vocabulary size.

This decoder is to predict the fine-grained pinyin, i.e., the initial, final, and tone, of each character in the correct sentence $Y$ based on the encoding output $\bm{H}$. Again, given each input character $x_{i}$ and its encoding output $\bm{h}_{i}$, we project $\bm{h}_{i}$ into a pronunciation-specific feature space:

and predict the initial $\hat{\alpha}_{i}$, final $\hat{\beta}_{i}$, and tone $\hat{\gamma}_{i}$ of the corresponding correct character based on the projection output:

Here $\bm{W}^{(p)}\in\mathbb{R}^{D\times D}$ and $\bm{b}^{(p)}\in\mathbb{R}^{D}$ are learnable parameters of the pronunciation-specific feature projection layer; $\bm{W}^{(\delta)}\in\mathbb{R}^{U\times D}$, $\bm{b}^{(\delta)}\in\mathbb{R}^{U}$ with $\delta\in\{\alpha,\beta,\gamma\}$ are learnable parameters of the pronunciation prediction layers; $U$ is the total number of pronunciation units (initials, finals, and tones).

We devise an adaptive task weighting scheme to balance the primary CSC and auxiliary CPP tasks during training. Given an input sentence $X$, the CSC task aims to match the predicted characters $\{\hat{y}_{i}\}_{i=1}^{n}$ with the ground truth $\{y_{i}\}_{i=1}^{n}$, while the CPP task aims to match the predicted fine-grained pinyin $\{(\hat{\alpha}_{i},\hat{\beta}_{i},\hat{\gamma}_{i})\}_{i=1}^{n}$ with the ground truth $\{(\alpha_{i},\beta_{i},\gamma_{i})\}_{i=1}^{n}$. Their loss functions are respectively defined as:

where $\mathcal{L}^{(c)}_{i}$, $\mathcal{L}^{(p)}_{i}$ are the character and pronunciation prediction losses associated with the $i$-th character in the sentence, and the pronunciation prediction loss $\mathcal{L}^{(p)}_{i}$ is averaged over the initial, final, and tone prediction.

Then as we have discussed earlier in the introduction, the auxiliary CPP task might provide different levels of benefits given different input characters. The more similar the input and target characters are in their pronunciation, the more likely there would be a spelling error caused by phonetic similarity. And to this case the CPP task might provide greater benefits and should be assigned a larger weight. To calculate such adaptive weights, we feed the target correct sentence $Y$ to the encoder and the followup pronunciation-specific projection layer. Then we calculate for each input character $x_{i}$ and its target character $y_{i}$ a cosine similarity ${\rm cos}(\bm{h}^{(p)}_{x_{i}},\bm{h}^{(p)}_{y_{i}})$ based on their pronunciation-specific feature representations $\bm{h}^{(p)}_{x_{i}}$, $\bm{h}^{(p)}_{y_{i}}$ (see Eq. (3)), and accordingly define the adaptive weight at the $i$-th position as:

The higher the cosine similarity ${\rm cos}(\bm{h}^{(p)}_{x_{i}},\bm{h}^{(p)}_{y_{i}})$ is, the larger the weight $w_{i}$ will be. Finally, the overall loss is defined as the CSC loss with an adaptively weighted CPP loss:

where $\mathcal{L}^{(c)}_{i}$ and $\mathcal{L}^{(p)}_{i}$ are the character-specific CSC and CPP losses defined in Eq. (7) and Eq. (8), respectively. There are two points worth noting here: (1) The branch of encoding and mapping the target sentence $Y$ is employed solely in the forward pass to calculate adaptive weights, and will be detached in the backward pass. (2) The auxiliary CPP task, as well as the adaptive weighting scheme, is introduced solely during training. At inference time, we use the CSC decoder alone for prediction.

As in previous work (Cheng et al., 2020; Liu et al., 2021; Xu et al., 2021), our training data is a combination of (1) manually annotated training examples from SIGHAN13 (Wu et al., 2013), SIGHAN14 (Yu et al., 2014), SIGHAN15 Tseng et al. (2015), and (2) 271K training examples from Wang et al. (2018) automatically generated by ASR- and OCR-based methods. We employ the test sets of SIGHAN13, SIGHAN14, SIGHAN15 for evaluation. The statistics of the used datasets are shown in Table 2. The original SIGHAN datasets are in traditional Chinese. We follow previous work (Cheng et al., 2020; Xu et al., 2021) to convert them to simplified Chinese using OpenCC⁴⁴4https://github.com/BYVoid/OpenCC. We further use pypinyin⁵⁵5https://pypi.org/project/pypinyin to obtain the pinyin of each character, and segment it into the initial, final, and tone using a pre-defined vocabulary of initials and finals provided by Xu et al. (2021).⁶⁶6https://github.com/DaDaMrX/ReaLiSe

We use the widely adopted sentence-level precision, recall and F1 as our main evaluation metrics. Sentence-level metrics are stricter than character-level metrics since a sentence is considered to be correct if and only if all errors in the sentence are successfully detected and corrected. Metrics are reported on the detection and correction sub-tasks. Besides sentence-level evaluation, we also consider character-level evaluation and the official SIGHAN evaluation. We leave their results to Appendix A

We compare SCOPE against the following baseline methods. All these methods have employed character phonetic information in some manner, and represent current state-of-the-art on the SIGHAN benchmarks.

• •

  FASpell (Hong et al., 2019) employs BERT to generate candidates for corrections and filters visually/phonologically irrelevant candidates by a confidence-similarity decoder.

• •

  SpellGCN (Cheng et al., 2020) learns pronunciation/shape similarities between characters via GCNs, and combines the graph representations with BERT output for final prediction.

• •

  MLM-phonetics (Zhang et al., 2021) jointly fine-tunes a detection module and a correction module on the basis of a pre-trained language model with phonetic features.

• •

  REALISE (Xu et al., 2021) models semantic, phonetic and visual information of input characters, and selectively mixes information in these modalities to predict final corrections.

• •

  PLOME (Liu et al., 2021) extracts phonetic and visual features of characters using GRU. It also predicts the pronunciation of target characters, but in a coarse-grained, non-adaptive manner.

In SCOPE, the encoder is initialized from ChineseBERT-base⁷⁷7https://github.com/ShannonAI/ChineseBert, while the decoders are randomly initialized. We then conduct further pre-training on wiki2019zh for 1 epoch with a batch size of $512$ and a learning rate of $10^{-4}$. The other hyperparameters are set to their default values as in ChineseBERT (Sun et al., 2021). During this pre-training stage, we do not use the adaptive task weighting scheme, and simply set the auxiliary CPP task weight to 1 for all characters for computational efficiency. After that, we fine-tune on the combined training set. We set the maximum sequence length to $192$ and the learning rate to $5\!\times\!10^{-5}$. The optimal models on SIGHAN13/SIGHAN14/SIGHAN15 are obtained by training with batch sizes of $96/96/$ $64$ for $20/30/30$ epochs, respectively. Other hyperparameters are again set to their default values as in ChineseBERT. All experiments are conducted on 2 GeForce RTX 3090 with 24G memory.

As for adaptivity, we make comparison among the following three diverse task weighting schemes that balance the CSC and CPP tasks.

• •

  Fully-adaptive (Full-adapt) is the scheme we used in SCOPE. It determines the CPP task weights according to phonological similarities between input and target characters, and the similarities are further adjusted dynamically during model training (see Eq. (9)).

• •

  Partially-adaptive (Part-adapt) also decides the CPP task weights according to phonological similarities, but the similarities are static, defined as $w_{i}=1-{\rm norm}({\rm edit\_distance}_{i})$, where ${\rm edit\_distance}_{i}$ is the Levenshtein edit distance (Levenshtein et al., 1966) between the pinyin sequences of the $i$ input and target characters and ${\rm norm}(\cdot)$ is a normalization function. The smaller the edit distance is, the larger the weight will be.

• •

  Non-adaptive (Non-adapt) considers no adaptivity and simply sets the CPP task weight to 1 for all characters ($w_{i}=1$ for all $i$).

We compare the three settings in the SIGHAN fine-tuning stage, starting from the same checkpoint after pre-training on wiki2019zh with a non-adaptive task weighting scheme. Here Full-adapt is equivalent to SCOPE.

As for granularity, we consider and make comparison between two types of CPP tasks.

• •

  Fine-grained (Fine) is the task we employed in SCOPE that predicts the initial, final, and tone of the pinyin of each target character.

• •

  Coarse-grained (Coarse) is a task that predicts the whole pinyin of each target character.

For fair comparison, we also introduce further pre-training on wiki2019zh with a coarse-grained CPP task, and use this checkpoint to initialize the Coarse setting during SIGHAN fine-tuning. In both two settings the CPP task is adaptively weighted as in Eq. (9), and Fine is equivalent to SCOPE.

Table 5 presents the sentence-level performance of these SCOPE variants on the test set of SIGHAN15. The scores of our best performing baseline REALISE as well as SCOPE without the CPP task (denoted as w/o CPP) are also provided for reference. We can see that introducing an auxiliary CPP task always brings benefits to CSC, no matter what level of adaptivity and granularity the task is. As for the adaptivity of task weighting, the Full-adapt scheme that considers dynamic adaptivity performs better than Part-adapt that considers static adaptivity, which in turn performs better than Non-adapt that considers no adaptivity. As for the granularity, a fine-grained CPP task performs better than a coarse-grained one. These results verify the rationality of introducing a fine-grained CPP task with adaptive task weighting to improve CSC.

We conduct ablation studies on SIGHAN15 with the following settings: (1) removing the auxiliary CPP task (w/o CPP); (2) removing further pre-training on wiki2019zh (w/o FPT); and (3) removing the constrained iterative correction strategy at inference time (w/o CIC). The results are presented in Table 6. We can see that no matter which component we remove, the performance of SCOPE drops. This fully demonstrates the effectiveness of each component in our method.

Table 7 further shows several cases from the SIGHAN15 test set to illustrate how the constrained iterative correction strategy (see Section 3.3) can effectively tackle consecutive spelling errors and address the over-correction issue. For consecutive errors, e.g., “户秃” in the first case, this strategy is able to correct them iteratively, one character at a time, e.g., by modifying “秃” to “涂” in the first round and then “户” to “糊” in the second round. For over-correction where the model makes unnecessary modifications, e.g., “他” to “她” in the third case and “隔” to “葛” in the fourth case, the iterative correction strategy can always change them back most of the time.