Hierarchical Transformer Encoders for Vietnamese Spelling Correction

The cost of building a big and high-quality training set would be prohibitive. Fortunately, training data for the spelling correction problem can easily be generated through mistake-free texts which are available online and simple analysis of common spelling mistakes. For a mistake-free corpus, we collect news texts published from 2014 to date from various Vietnamese news sources. Those texts were thoroughly revised and spelling checked by the publishers, so they only have a reasonable amount of spelling mistakes. We also include Vietnamese Wikipedia²²2https://vi.wikipedia.org/wiki/Wikipedia articles in the data. In addition, we consider oral texts by collecting high-quality movie subtitles, which only contains spoken language. In total, there are up to 2 GB of news/Wikipedia text data and 1 GB of oral text data.

Vietnamese spelling mistakes can be easily encountered in everyday life. We classify the most common mistakes into three categories:

• •

  Typos: this type of mistakes is generated while people are typing texts. It includes letter insertions, letter omissions, letter substitutions, transportation, compounding errors, or diacritic errors. Most people type Vietnamese in two styles: VNI and Telex, which use either numbers or distinctive characters to type tones or special characters. For example, ”hà nội”(English: Hanoi, the capital city) is often mistyped as ”haf nooij”when one forgot to turn on Vietnamese mode or was typing too fast.

• •

  Spelling errors: this type of mistakes is due to official conventions or regional dialects in Vietnam. For example, people who live in the north of Vietnam typically have trouble differentiating d/r/gi, tr/ch, s/x, n/l sounds. They often mispronounce or misspell the word ”hà nội”to ”hà lội”(English: no meaning). This type includes both letter or diacritic errors.

• •

  Non-diacritic: although this sometimes is intentional we take the diacritic restoration problem into account. Non-diacritic Vietnamese texts are much faster to type and fairly understandable under specific circumstances and contexts. However, these texts sometimes cause misunderstanding and are very informal. Therefore we would like to offer diacritic restoration to improve further the functionality of the model.

During the training data generation, we try to employ a random rule-based generator that covers all above types of spelling errors in diversity over the corpus. At the end, a generated sentence can have a lot of spelling errors or no errors at all. This ensures the model not only learn the core orthography of Vietnamese but also capture the surrounding context to make accurate spelling detection and correction.

For the most realistic performance measurement of the Vietnamese spelling correction problem, we would like to build up a test set that characterizes general behaviors of spelling mistakes while people are typing. We find out that drafts of Vietnamese Wikipedia articles are often submitted without much spelling revision. Therefore, we collect Wikipedia drafts from varied domains. We select ones that have a substantial number of spelling errors and then manually detect the errors and suggest reasonable substitute words. In total, we revised more than 100 articles with about 1,500 spelling errors and 14,000 sentences (approximately 1,300 sentences with errors). A typical sample in the data set can be found in Table 1.

The dataset is available at GitHub repository³³3https://github.com/heraclex12/Viwiki-spelling under the Attribution 4.0 International (CC BY 4.0) license. The dataset is stored in JSON lines and each document contains the ids, text contents, current revision ids, previous revision ids, Wikipedia page ids, and mistakes.

First of all, the inputted piece of text is broken down into tokens, or says, words based on white-space characters. These tokens are further broken down into characters, and then both these tokens and their characters will be passed on to our model as inputs. Normally, WordPiece [11] is a popular Byte Pair Encoding (BPE) sub-word tokenization [2], which would be used to reduce the exceedingly large amount of vocabulary. However, we do not go for sub-word options for the following reasons. Firstly, Vietnamese is a monosyllabic language whose tokens consist of only one syllable, and that sub-words only stand for phonetic features rather than the meaning. Secondly, Vietnamese has a fairly small amount of tokens. In fact, there are only 7184 unique syllables actually used, and they cover more than 94% of the content⁴⁴4http://www.hieuthi.com/blog/2017/04/03/vietnamese-syllables-usage.html. Lastly, in the spelling error problem, there are a lot of unexpected tokens that may appear due to typos or foreign languages. If sub-word tokenization is applied, resulting out-of-vocab sub-tokens may lead to unknown tokens which either causes loss of character information or redundant computation to deal with too many sub-words. Therefore, we decide to use whitespace tokenization with an additional character-level layer to keep character information. Moreover, we use a special token called UNK to mask out-of-vocab words that are likely to be incorrectly spelled.

For the neural network architecture, at both character and word levels, we adopt Transformer encoder architectures to encode information. The character-level encoder is composed of 4 self-attention layers of hidden size 256 while the word-level one is bigger, with 12 self-attention layers of hidden size 786, which is the same size as conventional BASE-size BERT. Tokenized inputs to the components are vectorized through two embedding tables (i.e. word embedding as $E_{string}$ and character embedding tables as $O$) and are summed with learned positional embeddings $E_{n}$ where $n\in[1,2,...192]$. In addition, the word embeddings are also concatenated with the corresponding character-level output vectors.

The context and character-aware representations of each word generated by the word-level encoder are used to classify if the token is a mistake and also to predict a substitute in case the token is an actual mistake. The outputting representations are then forwarded to two classifiers, one for detection and the other for correction. Each of two classifiers is composed of two fully-connected layers with the softmax activation function on the last layer. Besides, the correction classifier shares the same weights as the word embedding of the embedding layer.

After going through the detection classifier, each token of the sequence of tokens is predicted to be incorrect or not. If a token is incorrect, the correction classifier will suggest a suitable replacement.

The objective function for the learning process is the sum of two cross-entropy losses of the two classifiers. As accurate tokens do not need to be corrected, we remove those tokens out of the correction loss function. More clearly, given a sequence of words $x$ as input, the loss function formulated as follows, where $V$ is the vocabulary size; $N$ is the length of sequence of words; $E$ $(E<=N)$ is the number of true spelling mistakes; $y_{ij}$, $p_{ij}$ denote the ground-truth labels and the probability of predictions, respectively.

The experiment of our model was implemented based on ALBERT [4] Transformer Encoders. We use LAMB [12] optimizer with learning rate of 1.76e-3 and the power of poly decay is 1.0. The model is trained for 500,000 steps with batch size equal to 512. Word and character vocabulary size are 60,000 and 400 respectively, which include the most common words and characters in the training data. In order to verify performance of our model, we trained three models, one is traditional word-level Bi-LSTM (Bidirectional long short-term memory) approach, one is Bi-LSTM with the combination of word and character, and an other is word-level Transformer Encoder to see the effectiveness of our hierarchical model. Bi-LSTM is a special kind of recurrent neural network, which are made of a set of forward LSTM and backward LSTM. LSTM first introduced in [3] alleviates the vanishing gradient problem when captures long-term dependencies by having a set of memory blocks, each of the memory blocks has three different gates including input gate, forget gate, and output gate. Moreover, we also compare our proposed model with an external tool which is obtained from Viettel Group’s NLP⁵⁵5https://viettelgroup.ai/en/service/nlp.

Our experiments were evaluated using recall, precision and f1-score of the positive label (is-error label) for error detector. In order to measure correction performance, we computed accuracy score in two aspects:

• •

  Correction accuracy in the detected errors is formed by the ratio of exact correction and total correction.

  $$Accuracy_{in\ \%\ detected}=\frac{n_{exact\ correction}}{n_{exact\ correction}+n_{wrong\ correction}}$$

• •

  Correction accuracy in total errors is counted by exact correction divided by the sum of total correction and wrong detection.

  $$Accuracy_{in\ total}=\frac{n_{exact\ correction}}{n_{exact\ correction}+n_{wrong\ correction}+n_{wrong\ detection}}$$

The results of evaluating on the Wiki corpus are shown in Table 2. First, we observe that the hierarchical Transformer model themselves are quite useful (based on f-1 Score, correction accuracy on total and detected errors) to detect and suggest corrections for spelling errors. It achieves better scores in those metrics than both Bi-LSTM and external tool. Meanwhile, The Transformer model with multiple encoder layers (character and word) achieves higher scores on f1-score and correction accuracy on total errors than the Transformer model with encoders at word level but fails in correction accuracy on detected errors. To further clarify this, a higher detected recall score means the model predicts more errors than actual errors, which can lead to a low precision; therefore, we can observe that the Transformer model with char-word level encoders achieves a higher f-1 score. However, word level Transformer model is worse than char-word level Bi-LSTM model. The main reason is that the mistakes are not part of the vocabulary, and they will be split into sub-words and characters and use the shared embedding, meanwhile, combining character embedding and word embedding utilize more characteristics. Of those errors detected, both Transformer models suggest a high accuracy for spelling error corrections with 0.35% higher for the model with encoders at word level. This difference is insignificant, since it is less than 1%. Moreover, when weighting accuracy on total actual errors, Transformer model with multiple encoder layers at char-word levels achieves a significantly higher score than both Transformer model with encoders at word level and Bi-LSTM model with char-word layers (64.29% compare to 33.28% and 36.62% respectively). The hierarchical Transformer model utilizes both character and word level encoders to detect Vietnamese spelling errors and make corrections outperformed related contemporary models (Transformer and Bi-LSTM) and the external tools available for the task.

In the subtitle corpus, the hierarchical Transformer with multiple encoders also proved to be efficacious in restore missing diacritics in short conversation text with 98.50% in correction accuracy in the detected errors and 99.75% in detector f1-score. Proving that our model is also practical for informal text. The results show in Table 3.