A Combination of BERT and Transformer
for Vietnamese Spelling Correction

This section briefly presents the characteristics and differences from English of the Vietnamese language. Unlike neighbor countries, the Vietnamese does not use hieroglyphic letters, but a modified Latin (Roman) alphabet. The Vietnamese alphabet uses 29 letters, unlike the English alphabet, it does not use 4 letters ’w’, ’f’, ’j’, ’z’ and uses 6 more vowel letters (with special mark): ’ă’, ’â’, ’ê’, ’ô’, ’ơ’, ’ê’, and the letter ’đ’ [3, 6]. Along with the above 6 types of diacritics, it forms up to 67 separate letters (nearly 3 times larger than the number of letters in English). Therefore, spelling mistakes are much more common in Vietnamese than in English.

In this section, the concept of common error type in the Vietnamese language are presented. Due to the lacks of scientific public research or national survey constructed on this topic, various types of Vietnamese error type from previous related work [18, 19, 20] are summarized and divided them into six groups:

• •

  Abbreviation: There are a wide variety of abbreviation for common words in Vietnamese writing. Despite its convenience, this style of writing may raise misunderstanding, make the writing less formal and not accepted by most people. To determine this error cases, a list of most common abbreviation substitutions in Vietnamese is compiled from the Internet.

• •

  Region: The region error type is the most complicated type to analyze owing to its variety of happening contexts. The region error type comes from different region pronunciation across the Vietnam territory. When people tend to write a word the same way they pronounce it, this error occurs. Many adults may mistake this type of error if not a native speaker or do not have enough knowledge of the Vietnamese language. An incorrect word with region type stands alone, may still have meaning. Some examples of region error type are described in table 1.

  Original	Usually mistake for	Original	Usually mistake for
  ch-	tr-	-nh	-n
  tr-	ch	c-	k-
  -n	-ng	k-	c-
  -ng	-n	ngh-	ng-
  g-	gi-	gi-	g-
  …	…	…	…

• •

  Teencode: Teencode (or Teen-code) is a method of writing used by teenagers on social media or through messaging. Those teenagers put words into special encryption so the adults can not understand.

• •

  Telex: Telex is a convention for encoding Vietnamese text in plain ASCII characters, used initially for transmitting Vietnamese text over telex systems. Forgetting to turn on the language encoder or entering the wrong Vietnamese Telex rules leads to this type of error.

• •

  Fat Finger Fat Finger, also known as the clumsy finger, means when typing through a cell phone or computer keyboard, the user’s finger mistypes the surrounding key instead of the target key, causing the wrong words.

• •

  Edit Distance Edit Distance is a pseudo error generation strategy in which several characters equal to a ’distance’ to the original are randomly replaces. Although this error rarely happens logically, a low percentage amount is still generated in our data set.

For the convenience of observation, a list of examples corresponding to the type of error is presented in table 2.

BERT is a language representation model based on multi-layer bidirectional transformers encoder architecture. There is a wide variety of challenging natural language tasks that BERT can handle and achieve state-of-the-art performance, from classification, question answering, and sequence-to-sequence learning task, etc. BERT can represent sentence effectively by its encoding mechanism, including various embedding step as token embeddings, sentence embeddings, and transformers positional embeddings. Then, this BERT’s last hidden presentation output from BERT is used as the input into the transformers architecture.

In this study, two well pre-trained BERT on the Vietnamese: the Google Multilingual BERT²²2Github: https://github.com/google-research/bert (bert-base-multilingual-cased) and VinAI/phoBERT³³3Github: https://github.com/VinAIResearch/PhoBERT are considered. They are both trained with extensive Vietnamese corpus, while the multilingual BERT is the BERT base model, and phoBERT is a RoBERTa model [15] (which is a modified version from the base model).

Before the transformers architecture, Encoder-Decoder architecture using RNN/ LSTM/ GRU (Gated Recurrent Unit) cell is used widely in machine translation and sequence-to-sequence tasks. This Encoder-Decoder architecture, also known as the Seq2Seq model, uses several RNN cells to encode the input tokens to hidden states and then sum all hidden states up before sending them to the decoder. Thanks to this hidden state, the decoders receive all previously encoded information and use it for the output token prediction task. Despite large capacities in handling sequence-to-sequence tasks, the Seq2Seq decoder may fail to fully capture the meaning and context of the last hidden presentation from the encoder. That means the more extended and more complex the input sequence, the less effective the hidden presentation can represent, which is known as the bottleneck problem.

While the attention mechanism takes several inputs simultaneously, construct weight matrices captured from each hidden presentation input to calculate a weighted sum of all the past encoder states. The decoder will then take the inputs and the provided attention weights, and through that, the decoder knows how to ’pay much attention’ to which hidden presentation and vice versa. Another limitation of Seq2Seq architecture is that it handles the input sequentially, which means to compute for the current token at time t, we need the previous hidden state t-1 and so on. Therefore, especially in the spelling correction task, if many erroneous tokens stick together, the correction of the last tokens can be poorly affected. The transformer architecture and attention mechanism come to cross those boundaries of those previous architectures. Based on the Encoder-Decoder architecture [26], the transformers architecture [28] uses stacked multi-head self-attention and fully connected layers. The transformer is designed to allow parallel computation and reduce the drop in performance due to long dependencies. It uses positional embeddings and multi-head self-attention to encode more information about the position of a token and the relation between each token.

As Shun has provided an advanced insights to incorporating pseudo data into the spelling correction task [12]. Consequently, a vast pseudo training dataset can be generated so that not only can the transformer maximize its parallelization ability but also BERT can represent its rich contextual embedding vectors.

As mentioned in section 3.3, BERT is capable of deep language understanding by capturing contextual embedding of different words in a sequence. In addition, the Transformer model has been proved to be more efficient than popular Encoder-Decoder architectures, especially in the Machine Translation problem.

Some recent studies have treated the spelling correction problem as a machine translation job where the error sentence is the source sequence and the corrected sentence is the target sequence. And the combination of BERT and Transformer achieves state-of-the-art results for the English spelling correction task [9]. However, to the best of our knowledge, there has not been any research combining BERT and Transformer for Vietnamese spelling correction problem. The combination can be briefly summarized in the following steps:

• •

  Step 1: Let the input sentence notated as X = ($x_{1},...,x_{n})$, where $n$ is the number of its tokens; $x_{i}$ is the i-th token in X. BERT receives the input sequence tokens, and through its layers, BERT extracts them to hidden presentations notated as $H_{B}$ = ($h_{1},h_{2},...,h_{n})$, where $H_{B}$ is the output of the last layers in BERT.

• •

  Step 2: The Encoder will take $H_{B}$ from the previous step and encode the representation of each l layer $H_{E}^{l}$. The final contextual representation of the last encoder layer $H_{E}^{L}$ is the output of the Encoder. The Encoder components consist of the multi-head self-attention mechanism, position-wise fully connected feed-forward network. A residual connection around each of the two sub-layers, followed by layer normalization. A Multi-head Attention is a component allowing the model to jointly attend to information from different representations and helps the encoder look at other words in the input sentence as it encodes a specific word for better-capturing contextual embedding.

• •

  Step 3: The Decoder receives the representation $H_{E}^{L}$ from the Encoder and decodes through its layers into final representation $H_{D}^{L}$. Similar to the Encoder, the Decoder possessed the same components of the Encoder. These Encoder $H_{E}^{L}$ are to be used by each Decoder in its “encoder-decoder attention” layer which helps the Decoder focus on appropriate places in the input sequence.

• •

  Step 4: Finally, the Decoder final representation $H_{D}^{L}$ is mapped via a linear transformation and softmax to get the t-th predicted word ŷ. The decoding process continues until meeting the end-of-sentence token.

An illustration of our proposed method is shown in Figure 1.

This section describes the process of creating our training and testing set based on the Binhvq News Corpus⁴⁴4Github: https://github.com/binhvq/news-corpus which contains 14,896,998 Vietnamese news crawled from the Internet and preprocessed, including steps like HTML tag removal, duplicate removal, NFC standardization, and sentence segmentation. The corpus is gathered from reputable news and media sites in Vietnam, so the data is very reliable in terms of spelling. For the purpose of training and evaluating spelling correction, our newly constructed dataset must consist of two fields that can be described as a pair of correct and incorrect spelling sentences.

To the best of our knowledge, there is no specific survey as well as assessment on the rate of error types appearing in Vietnamese. However, Vietnamese often has common spelling mistakes: Region, FatFinger, Telex. Besides, some other types of errors are concerned, such as Edit-Distance, Abbreviation, Teencode, but rarely happened in practice. Therefore, error rate is reproduced based on our experience. Details of the rates of error types in the generated data set are listed in table 3.

The training set is composed by randomly selecting 4,000,000 sentences from the Binhvq corpus and then apply the pseudo error generator to these correct ones. All the sentences must have an average word count between 50-60 words per sentence. Similarly, validating set and testing set are generated with the number of correct sentences chosen from the above corpus 20,000 and 6,000 respectively. Details of the dataset can be summarized in table 4. The testing set is public and can be downloaded at the following link ⁵⁵5Github: https://github.com/tranhamduong/Vietnamese-Spelling-Correction-testset.

In the perspective of a spelling correction task, many traditional approaches used Accuracy, Precision, Recall, and F1 for evaluation [18, 19]. These metrics require the predictions’ words to have the same length as labels’ words. Recently, BLEU score is chosen, especially in deep learning models because of its ability to adapt to different prediction lengths [9, 31]. Therefore, BLUE [23] is selected for the evaluating task. BLEU, or the Bilingual Evaluation Understudy, is a score for comparing a candidate translation of text to one or more reference translations. Although developed for translation, it can be used to evaluate text generated for a suite of natural language processing tasks. Our BLEU configuration uses four n-grams settings because the spelling correction task critically requires the order of words in the sentence.

Where $BP$ stands for Brevity Penalty. $c$, $r$ is the length of the predictions and labels, respectively. BP will penalty cases where the model failed to propose correction, or the change happens more than allowed (as the number of words need to be corrected must be equal to the actual corrected). $p_{n}$ stands for modified n-gram precision, using n-grams up to length N and positive weights $w_{n}$ summing to one. The n-gram precision can be simply understands as ’the number of corrected words which occur in reference sentence (ground-truth)’ divided by ’the number of words after sentence transformed’. Therefore, the BLEU metrics has potential to not only to keep track strictly of word ordering by measuring n-gram (up-to-4) overlapping but also evaluate how a sentence has been corrected from the original despite the action (remove, edit, add more words).

Our models are implemented by fairseq toolkit [22] which is an re-implementation on the base Transformer architecture [28]. To find the appropriate hyperparameters for the proposed model, experiments with multiple model designations has been reviewed and the configuration of Jinhua work [31] are selected. Training details with hyperparameter settings are in the table 5:

In this phase, we compared with the Google Docs spell checking tool ⁶⁶6The tool can be found on the Google Docs website (https://docs.google.com/). We collected samples by using a web browser behavior simulator based on Selenium framework that manipulate the Google spell checking tool to correct all of its possible suggestions. and other methods. From the results showed in Figure 6, two versions of our model, Transformer+vinai/ phoBERT and Transformer+BERT-multi-cased, achieved better results than the previous methods. This partly reinforces our hypothesis that using a pre-trained language model BERT brings two benefits to the spelling correction problem: being applicable in the spelling correction task and taking advantage of contextualized word embeddings. Firstly, as mentioned in the BERT paper, tasks such as Text Classification, Question and Answering, Sentence Tagging, etc, are recommended to be used in the fined-tuning phase but the spelling correction task. Due to our modification, at the first step, BERT is verified to be beneficial for the correction task. Secondly, when correcting an error word, the action of choosing a suitable candidate based on context words is the main characteristic of the spelling correction problem. Concretely, BERT produces contextualized word embedding (the same word for different contexts have different embeddings) helps the models to better utilize word embedding at correcting phase. Besides, the pre-training of BERT on a huge data set also makes fine-tuning for our model easier because of no need to re-train from the beginning, taking advantage of the knowledge from the language model.

For the objective of comparison and practical application, there are a few patterns that our excellent model gain out performance: telex and edit-distance error types, compared to the google docs spellchecking tool. This happened partially because we designated more of these types of error to achieve our goal. More tuning is needed in future work on the error type distribution to improve performance for other types of errors.

The google docs spellchecking tool has another advantage over our model is the ability to restrict unnecessary correction. Additionally, the emergence of proper nouns also makes our model ineffective. When it comes to a proper noun, especially Vietnamese proper names, our model tends to correct them, which should not be the case. To overcome this weakness, some supporting components can be developed to the proposed architecture: Applying a name entity recognition component or an independent spellchecker to determine to correct a word or not.