Hindi to English: Transformer-Based Neural Machine Translation

Machine translation is one of the oldest tasks taken up by computer scientists and the development in this field has been going on for more than 60 years. The research in this field has made remarkable progress to develop translator systems to convert source language to target language while maintaining the contextuality and fluency. In earlier times, the translation was handled by statically replacing words with the words from the target language. This dictionary look-up led technique led to inarticulate translation and hence was made obsolete by Rule-Based Machine Translation (RBMT) [1]. RBMT is a system based on linguistic information about the source and target languages derived from dictionaries and grammar including semantics and syntactic regularities of each language [14]. With the absence of flexibility and scalability to incorporate new words and semantics and the requirement of human expertise to define numerous rules, rule-based machine translation systems could only achieve accuracy on a subset of languages. To overcome the issues of the RBMT system, a new approach called Statistical Machine Translation (SMT) was introduced. Instead of having rules determine the target sequence, SMT approaches leverage probability and statistics to determine the output sequence. This approach made it feasible to cover all types of language within the source and target language and to add new pairs. Most of these systems are based on Bayesian prediction and have phrases and sentences as the basic units of translation. The main issue faced by this approach is the requirement of colossal amounts of data, which is a huge problem for low resource languages.

Due to these prevailing issues, there is a demand to explore alternate methods for creating a smarter and more efficient translation system. The development of various deep learning techniques and the promising results shown by the combination of these techniques with NLP created a new approach called NMT. NMT’s advantage lies in two facts that are its simplistic architecture and its ability to capture long dependencies in the sentence, thereby indicating its huge potential in emerging as a new trend of the mainstream [2]. Conceptually speaking, NMT is a simple Deep Neural Network (DNN) that reads the entire source sentence and produces an output translation one word at a time. The reason why NMT systems are appealing is that they require minimal domain knowledge which makes it well-suited for any problem that can be formulated as mapping an input sequence to an output sequence. Also, the inherent nature of the neural networks to generalize any input implies that NMT systems will generalize to novel word phrases that are not present in the training set.

Moreover, almost all the languages in the world are continuously evolving with new words getting added, older words getting amended and new slangs getting introduced very frequently. Even though NMT systems generalizes the input data well, they still lack the ability to translate the rare words due to their fixed modest-size vocabulary which forces the NMT system to use unk symbol for representing out-of-vocabulary (OOV) words [9]. To tackle this issue, a subword tokenization technique called Byte Pair Encoding (BPE) was introduced. BPE divides the words such that the frequent sequence of letters is combined thereby forming a root word and affix. This approach alone handles the OOV words by merging the root word and the different combinations of affixes thereby creating the rare word [4].

In this paper, we present the experimental setup and the state-of-the-art results obtained after training the Transformer model on the IIT Bombay CFILT English-Hindi dataset of 1.5 million parallel records. The paper is organized as follows: section 2 describes the motivation behind our work. Section 3, describes the model that we have implemented. In section 4 we present the details of the experimental setup for training the model. In section 5 we display a comparative analysis of the results obtained by training the model in different configurations. Finally, section 6 concludes the paper.

With the power of deep learning, Neural Machine Translation has arisen as the most powerful approach to perform the translation task.

In [5] a model called Transformer was introduced which uses the encoder-decoder approach where the encoder first converts the original input sequence into its latent representation in the form of hidden state vectors. The decoder then tries to predict the output sequence using this latent representation. RNN’s and CNN’s inherently handle sequences word-by-word sequentially which is an obstacle to parallelize. The Transformer achieves parallelization by replacing recurrence with attention and encoding the position of the symbol in the sequence. It reduces the number of sequential operations needed to relate the two symbols from the input/output phrases to a constant number O(1). This is achieved by using the multi-head attention mechanism which allows to model the dependencies regardless of their distance in input or output sentence.

In [7], they show that they can improve the machine translation quality of NMT systems by mixing monolingual target sentences into the training set. They investigate two different methods to fill the source side of monolingual training instances: using a dummy source sentence and using a source sentence obtained via backtranslation, which they call synthetic data and conclude that the latter gives better results.

In [8], they investigate NMT models that operate on the level of subword units. Their main goal is to model open-vocabulary translation in the NMT network itself, without requiring a back-off model for rare words. In this paper, byte pair encoding has been adapted. BPE is a compression algorithm that is used for word segmentation. It facilitates the representation of an open vocabulary via a fixed-size vocabulary that contains variable-length character sequences, thus it serves as a highly suitable strategy of word segmentation for neural network models.

Motivated with the results obtained by the transformer for machine translation on various languages, we created a translation system that translates a Hindi sentence to English. Since we use a low resource Hindi language for which the amount of good quality parallel corpus is limited, we applied back-translation to increase the quantity of training data. To overcome the problem caused by out of vocabulary words we used BPE.

We evaluate the quality of translation of our model on the test set using the Bilingual Evaluation Understudy (BLEU) score [12] and the Rank-based Intuitive Bilingual Evaluation (RIBES) score [11]. For depicting the performance of subword level tokenization, we have divided the test set into 2 subsets. The first set (Set-1) consists of sentences whose words are present in the vocabulary generated from word level tokenization. This set consists of 1694 sentences. The second set (Set-2) is the complete test set consisting of 2478 sentences.

In Table 3, after adding the first batch of 0.5M parallel back-translated records with the original training data, the BLEU score increased by 3.79 and with the subsequent addition of other 2 batches the BLEU and the RIBES score reached a maximum of 24.79 and 0.741 respectively. However, when the 4th batch of 0.5M back-translated records was added with the previous batches, the scores decreased by a small margin indicating convergence with respect to the addition of back-translated data.

Similar to the results obtained with word level tokenization, in Table 4, after adding the first batch of back-translated records the BLEU score increases by 4.78 and with the subsequent addition of other 2 batches the BLEU score reached a maximum of 25.87. After adding the 4th batch, the BLEU scored decreased by 0.13 however the RIBES score increased by 0.003.

When compared with word level tokenization, subword level tokenization achieves a better BLEU score which can be attributed to the fact that it has the advantage of not having an out-of-vocabulary case and also to learn better embeddings for rare words since rare words can enhance the learning from its subwords that occur in other words. This fact is further strengthened in Table 5 which shows the BLEU and RIBES score for the Transformer with Batch4 model using word and subword level tokenization on Set-2. The decrease in the BLEU and RIBES score as compared to Table 3 and Table 4 is due to the fact that the Set-2 consists of additional sentences as compared to Set-1 which contain rare words that are not included in the vocabulary for word level tokenization. When subword level tokenization is used, the model performs reasonably well even in the presence of rare words which is not the case for word level tokenization.

The transformer model has displayed promising results for neural machine translation involving low resource languages as well. We saw that after adding the back-translated records the performance was certainly improved, however when the amount of generated data increases beyond a certain level, there is no further improvement in the performance. Using a combination of the transformer model, back-translation technique and a subword tokenization method like BPE, we achieved a BLEU score of 24.53 which is the state-of-the-art on this dataset to the best of our knowledge. In the future, we can try to incorporate state-of-the-art Natural Language Processing models like BERT [3] into NMT to further improve the quality of translation.