Bangla Grammatical Error Detection Using T5 Transformer Model

Currently, BERT (Bidirectional Encoder Representations from Transformers) and its variants are the best-performing models on tasks such as token classification[12] and sentence classification[13]. BanglaBERT reproduced this finding when trained specifically on Bangla corpus[11]. Although GED can be formulated as either a token classification problem or a sentence classification problem, both possess several challenges. When presented as a token classification task, punctuation becomes a particular issue since most punctuations represent a pause and are hard to distinguish. Another challenge is tokens which are missing altogether. It can be hypothesized that BERT does have the ability to detect the logical inconsistency in a sentence that arises from missing tokens due to its deep encoder architecture but marking the position of missing tokens is a challenge. On the other hand, when posed as a sentence classification problem, we find that BERT can classify sentences are either error-free or with errors well but cannot mark the erroneous section itself. Recently, sequence-to-sequence (seq2seq) models such as the T5 [JMLR:v21:20-074] have achieved state-of-the-art performance on standard Grammatical Error Correction (GEC) benchmarks[14]. Such models [6] [7] have been trained specifically on synthetic GEC datasets (as opposed to general translation datasets). But since the model must generate the entire output sequence, including the parts which were correct to begin with, inference is slow. The BERT-based GECToR [3] presents another way for GEC - a token classification approach where errors are mapped to the 5000 error-correcting transformations (one for each token in vocabulary and some token independent transformations) which correct the errors algorithmically. The resulting model is up to 10 times faster than comparable seq2seq models but as before, this requires a synthetic pretraining corpus.

For Bangla Grammar Error Detection we decided on the small variant of BanglaT5[1] with 60M parameters. The smaller model allowed for larger batch sizes, faster experimentation and hyper-parameter tuning when compared to the standard BanglaT5 model with 220M parameters while delivering similar performance on our training set (9385 pairs). Experimentation on the larger T5 models using the full available dataset (19385 pairs) and evaluating a BERT-based approach similar to GECToR is left for future work.

The training set consisted of 19385 sentence pairs in total, containing both error-free sentences and sentences with errors. The major error types are: The major error types are:

1. 1.

   Single word error

2. 2.

   Multi-word error

3. 3.

   Wrong Punctuation

4. 4.

   Punctuation Omission

5. 5.

   Merge Error

6. 6.

   Form/Inflection Error

7. 7.

   Unwanted space error

8. 8.

   Hybrid

The errors are each bracketed by a designated symbol $ and are not differentiated from each other.

We used DataSetFold1 for the fine-tuning of the T5 model and both DataSetFold1 and DataSetFold2 for the crucial post-processing steps.

We collected a word list of 311 archaic Bangla verb words which were consistently marked as errors in the training dataset. We collected said word list with the aid of Github Copilot. This data was used in our regular expression based approach to GED.

Not wanting to shift the distribution of the train set from the test set, we kept pre-processing to a minimum. The sentences were normalized and tokenized using the normalizer and tokenizer used in pretraining [1]. One notable point is that we omitted newline characters when present inside sentences since it interfere with the way the T5 model reads in sentences.

Through experimentation on an 80-20 split of DataSetFold1 between training and validation set and using an effective batch size of 128, we determined 120 epochs to be a good stopping point before the model starts to over-fit. Then we used the entirety of DataSetFold1 for 120 epochs of training. Since the task was to predict 5000 test sentences while training only on 9385 training pairs, we determined that keeping any significant segment for validation and early stopping would be detrimental to overall performance.

A naive attempt to train the model on the combined DataSetFold1 and DataSetFold2 dataset did not improve our score on the test set. This is likely because introducing new data requires re-tuning the model hyper-parameters. For now, we leave this as future work.

The T5 model was built on the paradigm of unifying all NLP tasks under text-to-text classification and on that front, T5 achieves state-of-the-art results on many GLUE tasks. However, this paradigm does have its shortcoming. Of particular importance in the task of GED when judged by Levensthein Distance is the tendency of T5 models to spell words differently or sometimes change entire words with a close synonym because reproducing the input sequence exactly is as important as marking the errors. This is a particular problem in Bangla GED since the language is still evolving and multiple spellings of the same word are in use concurrently. Furthermore, there exist several unicode characters representing the same Bangla alphabet or symbol, further complicating the reconciliation of the T5 output sequence with its input sequence.

To transform the raw T5 output to a form as close as possible to the input sequence, we present two algorithms. As an optional post-processing of all outputs, we present a third algorithm that does simple error word detection that the model might have missed.

The first is for respelling and correcting the T5 output by comparing it character by character with the input sequence. Beginning with an empty string as corrected output, if the next character is a $ symbol, it is appended to the corrected output string. If the next character of the input and output sentence match, it is appended. If they do not match, the next character of the output string is looked up in a table and if present, the value from the lookup table represents the correction and is appended. This lookup table has been constructed manually by observing common t5 errors. Constructing the table automatically is left for future work.

If character-level corrections fail, the algorithm attempts to make word-level corrections by replacing entire words in the t5 output and then character-level correction is attempted again.

The second algorithm is a regular expression-based approach to GED in case the first algorithm fails to correct the T5 output. Certain common errors are learned from the training dataset and identified in the test set using sub-string replacements.

These two algorithms work in tandem to correct the t5 output. However, should a test sentence already be present in the training dataset, then the error-marked sentence is directly pulled from the training dataset using another lookup table. In a real-world scenario, having a lookup table of the most commonly mistaken sentences or phrases can significantly speed up GED since the need for a large deep learning model is bypassed entirely.

The pseudo-code for the algorithms is in the Appendix.

function t5_output_correction(t5_output, t5_input):
    corrected_output = ""
    i1 = 0
    i2 = 0
    attempt_word_corr = True
    while True:
        if i1 == len(t5_input) and i2 == len(t5_output):
            return corrected_output

        if t5_input[i1] == t5_output[i2]:
            corrected_output += t5_input[i1]
            i1 += 1
            i2 += 1
            continue

        if t5_output[i2] == "$":
            corrected_output += "$"
            i2 += 1
            continue

        if t5_output[i2] in character_lookup.keys():
            if t5_input[i1] == character_lookup[t5_output[i2]]:
                corrected_output += t5_input[i1]
                i1 += 1
                i2 += 1
                continue

        if attempt_word_corr == True:
            t5_output = word_correction(t5_output, t5_input)
            i1 = 0
            i2 = 0
            attempt_word_corr = False
            continue

        return regex_correction(t5_input)

function regex_correction(sentence):
    for word in common_errors_set:
        sentence = sentence.replace(word, "$"+word+"$")

    for rule in regex_rules:
        sentence = sentence.replace(rule, "$"+rule"$")