VSEC: Transformer-based Model for Vietnamese Spelling Correction

Vietnamese spelling correction can be formulated as follows. Given a set of syllable sequences X = $\{x^{i}=(x^{i}_{1},x^{i}_{2}...,x^{i}_{n})\}$ with some errors in $x^{i}$, and a set of syllable sequences Y = $\{y^{i}=(y^{i}_{1},y^{i}_{2}...,y^{i}_{m})\}$, where $y^{i}$ is error-free. The goal is to transform each sequence $x^{i}$ into corresponding sequence $y^{i}$. Following that, the task can be considered as a problem of learning a function $\textit{f}:\textit{X}\rightarrow\textit{Y}$ that satisfies $\textit{f}(x^{i})=y^{i}$.

The state-of-the-art method for Vietnamese spelling correction is to use a statistical language model. Although the method is trainable on a large dataset, it uses a limited context. This motivates us to create a new model which exploits broad context.

Figure 1 illustrates an overview of our proposed model, called VSEC, in training and testing processes. The pipeline of VSEC is composed of three components: a preprocessing module, a tokenization module, and a Transformer-based model. The original data is first preprocessed to remove any noise that might appear in the sentence. The BPE tokenizer then converts each sentence into a sequence of tokens. Finally, the Data Loader feeds the sequence into the Transformer-based model for training.

To ensure that the results are reliable, data quality assurance is critical. Each input sequence needs to be removed noise before proceeding to the next phase. Our preprocessing module, in particular, consists of five steps:

• •

  Step 1: Remove noise characters - In this step, we remove characters that are not useful for learning the context of a sentence such as emojis, line break characters.

• •

  Step 2: Convert uppercase to lowercase - Using both upper and lower case can affect the model’s data density. Therefore, all uppercase characters are converted to lowercase.

• •

  Step 3: Standardize marks - In this step, each syllable is converted to the syllable in the telex typing form. Following that, the data is standardized using a mapping set between the telex syllable and the correct mark syllable.

  For example: Syllable “cuả” =¿ Telex syllable “cuar” =¿ Standardized syllable “của”.

• •

  Step 4: Split merged syllables - The appearance of merged syllables in the dataset may have a detrimental effect on the model’s learning. We use the Peter Norvig word segmentation algorithm to solve this problem [15]. When syllables are merged in the Vietnamese language, they transform into the syllables in the telex typing form. This occurs as a result of the Vietnamese typing tools’ mechanism. Therefore, while calculating probability using the Peter Norvig algorithm, it is critical to convert the telex syllable to the standard Vietnamese syllable.

• •

  Step 5: Merge separated syllables - A syllable that has spaces between its characters is known as a separated syllable. The model is also more difficult to converge due to the appearance of these strange syllables. To solve this problem, we employ the Trie structure [21], which has demonstrated its ability to browse prefixes.

In Vietnamese, each space-separated token is in monosyllabic form. Therefore, we call the word level in English as the syllable level in Vietnamese from now on. At first glance, using syllable level as the input seems like a good idea. However, this level is not well suited for spelling error correction, as we can have difficulties with misspelling syllables or rare syllables (out of vocabulary). It makes the model harder to learn the sentence’s context. One of the solutions to this problem is to use the character level. Nevertheless, breaking syllables into characters will increase the sequence length. As a result, the model is large and slow to converge.

Subword level is between syllable level and character level. It keeps the input length at a reasonable level while addressing the out-of-vocabulary problem. For example, we can split a Vietnamese misspelling syllable “nghành” into two tokens: “ngh” and “ành”, and present “nghành” by vectors of these tokens. The BPE algorithm is used to construct a subword dictionary [17]. Given a large corpus, this tokenization technique groups characters into frequent sequences. It is totally unsupervised and requires no information about the context of the sentence. An example of how BPE obtains vocabulary from raw text is shown in Table 2.

The algorithm of BPE is as follows. Firstly, a special token $/w$ is appended to each syllable to indicate the end position of a syllable. Then, we split all sentences in the corpus into characters. At this point, the vocabulary only contains single characters. After that, we iteratively count all token pairs and merge each occurrence of the most frequent pair (Y, Z) into a new token YZ and add it to the vocabulary. The size of the final vocabulary is equal to the total number of merge operations and initial characters. The number of merge operations is the only parameter of the BPE algorithm. We will have a large vocabulary if this number is large. An example of the BPE tokenization result is in Fig. 2.

Based on the idea of treating Vietnamese spelling error correction as a machine translation problem, the proposed model learns to translate the sentence having spelling errors to the corrected one. Specifically, we use the Seq2seq architecture based on Transformer [22] as our baseline. The Transformer encodes a misspelling sentence to a context hidden state using a stack of L encoder blocks, each of which employs a multi-head self-attention layer and a feed-forward network. The decoder uses the encoder’s hidden states and the sequence of previous target tokens to generate the target hidden state by applying a stack of L decoder blocks. The decoder block has the same architecture as the encoder one, except it has an extra attention layer over the encoder’s hidden states.

The goal of Transformer is to predict the next token $y_{t}$, given the source tokens $(x_{1},x_{2},...,x_{n})$. The formulas of this process are:

The embedding matrix is represented by $\textbf{E}\in\textbf{\text{R}}^{d\times|V|}$, where d is the embedding dimension, and $|V|$ is the size of the vocabulary. The value of $x_{i}$ represents the position of the i-th token in the vocabulary. The encoder’s hidden states are denoted by $\textbf{h}^{src}_{1..n}$ and the target hidden state of the next token is denoted by $\textbf{h}_{t}$. After obtaining the target hidden state, the model determines the next token to be generated by feeding $\textbf{h}_{t}$ into the fully connected (dense) layer behind. Particularly, the fully connected layer has $|V|$ hidden units activated by the Softmax function, which produces scores whose total is 1.0. These values correspond to the generation probability distribution of the next token.

To generate a dataset for training the proposed model, we created a process for artificially adding errors to non-error sentences in Vietnamese. This process is referred to as Error Generator. We began by extracting 5 million sentences from a Vietnamese news corpus¹¹1https://github.com/binhvq/news-corpus, which was crawled from several prominent Vietnamese websites. A fusion table was also constructed, in which each syllable is linked to a group of other candidates, to present common types of Vietnamese spelling errors such as mistyped errors, consonant errors. Then, at random, we selected 8% of the syllables in the sentences to artificially generate errors, with 90% of them being replaced with other syllables, 5% being removed, and 5% being duplicated. The difference between VSEC Error Generator and others is the use of add and delete operators, which represents errors when the typists often use copy and paste.

In addition, we developed a realistic dataset for testing. We sampled the contents of 618 documents at Tailieu²²2https://tailieu.vn, an educational material website. To ensure that the dataset includes a significant amount of incorrect sentences, we sampled documents from lower quality texts, and thus the error rate of the dataset higher. Three people handled three phases of labeling to carefully correct spelling errors in the texts. The dataset includes 9,341 sentences, which contain 11,202 spelling errors in 4,582 different types³³3https://github.com/VSEC2021/VSEC.

We utilized syllable-level precision, recall, and F1 score which are common in the community [12, 9, 14]. In addition, we evaluated the accuracy of both detection and correction tasks. Specifically, we used six metrics:

The BPE Tokenizer is used in the experiments based on HuggingFace’s library.⁴⁴4https://github.com/huggingface/tokenizers Specifically, the BPE Tokenizer was trained on news corpus to build a subword-level vocabulary. We set the vocab size to 30,000 and kept other default hyperparameters.

In the training phase, we use Adam optimizer with Cross-Entropy Loss to train the neural network model with Transformer architecture. Through the experiment, the model achieved the best results with hyperparameters are shown in Table 3.

To conduct an informative experiment, we rebuild the N-gram model as a single baseline for comparison. Comparison to other approaches [14, 13, 11] is not conducted due to two main reasons. First, they are proposed for domains different from ours [14, 13]. Some studies only focus on OCR spelling correction. Apparently, it is not directly comparable to the methods for general solutions like VSEC. Second, some studies primarily focus on a specific sort of Vietnamese spelling errors, such as consonant misspell errors [11]. Thus, it is unfair to compare VSEC with methods in these studies. For the above reasons, we evaluate and analyze the current state-of-the-art method, N-gram, to ensure fairness and generality of the experiment.

In addition, the BiLSTM Seq2seq model with attention mechanism and the Transformer models at different token levels are also trained and tested according to the same process. The Vietnamese news corpus is still being used to build these baseline methods.

Table 4 shows the experimental results of all methods on the test dataset. From the table, we can see that the proposed model substantially outperforms the baseline methods. Particularly, in the detection phase, our proposed method performs much better than the baselines in terms of all metrics. The result for recall of correction task on the test dataset is greater than 76%, implying that more than 76% of errors will be fixed.

The N-gram method achieves the highest precision of correction because it can reduce false corrections by using an additional parameter, error_threshold. The use of this parameter is effective with more than 89% of precision in both evaluation criteria. However, precision and recall are a tradeoff. Increasing the precision of the N-gram method entails lowering the recall. On the other hand, the proposed method shows more balance, when the values of the precision and recall measurements are not significantly different. Specifically, the proposed method reaches 86.8% with the F1 score measure in the error detection task and 81.5% in the error correction task, while the N-gram method only reaches 81.2% and 79.3%, respectively.

For the methods of using Seq2seq architecture, the subword-level Transformer model performs better than the other baselines, while the method of the BiLSTM Seq2seq model with attention mechanism performs fairly poorly. This indicates that, despite ignoring traditional recurrent architectures, the Transformer-based models are still able to outperform the LSTM-based models. Furthermore, the subword-level model can beat the models at other token levels. This demonstrates that the subword-level model can handle the out of vocabulary problem better than one at the syllable level and it also performs effectively than the character-level model.

We also investigate the effect of the vocabulary size and the data size. Table 5 shows that the proposed method reaches its best performance with the data size is 5 million. This indicates that the more training data the higher performance can achieve.

A larger vocabulary size means fewer syllables split into two or more tokens. Table 6 presents the results of the proposed method in different values of the hyperparameter vocabulary size. The highest F1 score is obtained at the vocabulary size equal to 30,000. That is to say, having a larger vocabulary does not guarantee a higher F1 score.

We observed that the proposed method is able to make more effective use of context information than the N-gram method. When there are two or more errors next to each other in a sentence, N-gram usually detects one and leaves the others undetected. The proposed method, on the other hand, can detect both of the errors. For example, there are 2 two errors in the sentence “Chuẩn bị sãn sang hợp đồng ký kết” (Prepare the contract to be signed). The syllables “sãn” and “sang” are incorrect and they should be written as “sẵn” “sàng”, which form a word meaning “ready”. The N-gram method only corrects one syllable that is “sãn” while the proposed method can correct both of them. It is because the N-gram method relies on the context provided by nearby syllables, specifically two syllables before and two syllables after the target.

We also found that the proposed method has three major types of false detections. For statistics of errors, we sampled 100 false detections from the test set. We noticed that 32% of errors are foreign words and acronym words, 28% of errors are due to a lack of domain-specific knowledge, and the remaining 40% of errors have no specific type.

Foreign words and acronym words are the first type of false detection. These words are sometimes converted to Vietnamese syllables by the model. For example, in the sentence “TH đã luôn tiếp cận sản xuất theo chuỗi đồng cỏ sạch” (TH has always approached clean grassland chain production), the acronym word “TH” (a Vietnamese company) is converted to “Thì” syllable. This indicates that in order to make more reliable detections, the models must have a stronger way to determine what the special syllables are.

The second type of false detection is due to a lack of domain-specific knowledge. For example, in the sentence “Đồ thị của hàm số bậc hai” (Graph of quadratic function). The model turned the word “Đồ thị” (Graph) into “Đô thị” (City). This happens due to the fact that the test set is inclined to scholarly language while it is not much in the training data created from the news corpus. This problem is still very challenging for the existing model to determine this type of error.