An In-depth Walkthrough on Evolution of Neural Machine Translation

Machine Translation is the branch of computational linguistics which focuses on concepts dealing with translation of text or speech from one language to another (nowadays from one ’form’ to another). These concepts have proved to be a very significant leap in computational linguistics and have a rich set of use cases including cross platform applications, international communication, medical domain, etc. The origins of machine translations come from substituting the tantamount words from one language to other. This was followed by rule based approaches which include eliciting several protocols based on the respective grammatical senses for the translation process. These methods just quite serve the purpose but substitution of words may change the meaning of the context or equivocal words may add to the confusion. With the rise of Machine Learning paradigms and statistical modelling, Statistical Machine Translation was born. These concepts use information theory to predict a word in the target language from existing bilingual corpora with the help of probabilities. However, the collection of these corpora may be expensive. Also these methods have been proved to fail to an extent while translating between languages that do not follow similar grammatical patterns (e.g. ordering of verbs / adjectives / nouns).

The state of the art Language Modelling concepts were engendered and disseminated with Neural Networks. In the age of Deep Learning, Artificial Neural Networks have known to outperform almost all the statistical models in Machine Translation Domain. Neural Machine Translation is hence Machine Translation using Artificial Neural Networks. In this paper, we discuss the major leaps in NMT, various architectures, evaluation metrics, benefits, shortcomings and comparison among the models.

In Regular Feed Forward Neural Networks, the inputs are multiplied with respective weights to obtain intermediate hidden representations (layers) and finally the output. These hidden layers basically imply a very complex mathematical non-linear function. Hence a vanilla Feed Forward Neural Network maps the inputs to the outputs by establishing a mathematical relation between them. However here, the output is the function of just the current inputs. On the contrary, in Recurrent Neural Networks (RNNs), the hidden layers along with the current inputs, also consider the previous hidden layer outputs. These previous outputs introduce a notion of ’memory’ in the Neural Network architecture. The use case of this type of architecture is when the inputs are in a form of a series where the inputs as well as the previous ’timestep’ contribute to the output. A typical use case of this can be finding trends in a ’Time Series’. The backpropagation in these neural nets is called as ’Backpropagation Through Time’ (BPTT). However, with BPTT, a major issue of Vanishing Gradients[1] arose. These issues were overcome by Long Short-Term Memory Networks and its variants. So these recurring properties of RNNs can be exploited for Natural Language Processing tasks since in any Language Model, the words in a given context definitely depend on the previous one and the similarly for sentences, the pattern follows. This is the motivation behind utilizing the architectural benefits of RNNs in Neural Machine Translation.

Vanilla RNNs can be used in NMT, but it would be similar to a word-to-word mapping model, just using neural networks. The intuition to this can be obtained from the example of English to Chinese translation; the length of the English sequence may not be equal to the corresponding Chinese sequence. Also, the translation of an English word at some position in the sentence may not be at the same position in the Chinese context. Hence for such translations [2] proposes the Sequence to Sequence architecture. It is an encoder-decoder architecture, i.e. the input sequence (English sentence in the above example) will be encoded first into an intermediate format (vector) and then using the encoded vector the target sequence (Chinese sentence) will be decoded. In the architecture proposed in [2], there are two RNNs (LSTMs), one as encoder and other as decoder respectively. The depth of the network maybe varied by increasing the number of LSTMs at each side. The cell states of LSTMs are are updated taking one word per timestep (depending on the use case), the output sequence of the encoder LSTM is superfluous, the cell states are passed on to the decoder and using those states the decoder predicts the target sequence one word per timestep. The intuition to this model can be obtained by reconciling it with a human translator; for translating a sentence, rather than translating each word of the sentence individually, one would first ’listen’ to the complete sentence (sequence) and then translate it.

The Sequence to Sequence architecture is the base architecture for the state of the art NMT models. In [2] the main dataset used is WMT’14 English to French; the BLEU[3] score obtained was 34.81 by directly extracting translations from an ensemble of 5 deep LSTMs (with 384M parameters and 8,000 dimensional state each) using a simple left-to-right beam-search decoder. This was the best result for direct translations using Neural Network at the time.

The Attention Mechanism is an addition on the Sequence to Sequence Architecture[2]. The most common implementation of attention is Bahadanau Attention[4]. The problem with sequence to sequence is that it is incapable of remembering long sequences. The intuition to this can be obtained by relating this to the fact that just a single cell state vector is being passed on through various timesteps of a LSTM cell and is updated at every timestep. In such a vector which is overwritten multiple times (in case of longer sequences), the data from the initial cell states may start to disappear. The name attention too, comes from the human tendency to pay ’attention’ only on certain portions of a context while making translations from one context to another. Hence this suggests that rather than focusing on "How much to remember?", attention answers "What to remember?". The idea behind implementation of attention is that, all the intermediate context vectors should contribute to the final context vectors rather than just the last vector.

In NMT many of the words in the corpus are generalised by the ’<unk>’ (unknown) representation so as to reduce the vocabulary size to scale and discard very less frequent words from the vocabulary. However in many cases this depletes the performance of the model. In [16], the architecture used is plain encoder-decoder[2] with attention[4] and there are two softmax layers to predict the next word in the conditional language model; one to predict the location of the word in the model and the second to predict the word in the shortlist vocabulary. The decision of using which softmax is made by an Multi-Layered Perceptron or Feed Forward Neural Net. The intuition to this nmechanism is the human tendency to pointing at objects when not knowing their name.

The model was tested on newstest2011 French to English dataset and gives better results (BLEU score 23.76) as compared to traditional attention based NMT model (20.19) on the same dataset for sequence length 50. 30,000 tokens for both the source and target language shortlist vocabularies were used. The corpus contains 134,831 English and 153,083 French words. A word level translation dictionary is maintained for 15,953 words that belong to none of the vocabularies. About 49,490 words are shared between English and French corpora of Europarl.

GNMT[17] introduces a parallelism in model training and speeding up the training process. Along with parallelism, GNMT also introduces noble segmentation approaches to address out of vocabulary (OOV) words including wordpiece model which was originally developed o solve a Japanese/Korean segmentation problem for the Google speech recognition system[18]; and the mixed word/character model.

The first layer of encoder is a bi-directional layer, which computes on a GPU in parallel first and then the rest of the 7 unidirectional layers are trained in parallel on separate GPUs. The output is fed to the attention module which passes the output to the 8 decoder layers. The model consists of residual connections so as to enable training very deep neural network architectures[14]. The first encoder layer is bidirectional as some of the grammars in different languages exhibit positional resemblance in the words and hence feeding the same input in reverse sometime helps.

GNMT was trained on the publicly available WMT’14 English-German as well as English-French datasets. The results are compared to several models as well as human evaluation. On the En-Fr with WPM-32K, the BLEU score was 38.95 and with Mixed Word/Character model it was 38.39. On En-De with WPM-32K the BLEU was 24.61 and with Mixed Word/Character model it was 24.17. For model ensemble, WPM-32K (8 models) on En-Fr had BLEU 40.35 and for RL-refined model, BLEU was 41.16; for En-De, the WPM-32K BLEU with 8 models was 26.20 and RL-refined WPM-32K (8 models) was 26.30. Human side-by-side evaluation scores of WMT En-Fr models has an average score of 4.82 whereas the NMT model with and without RL has an average score of 4.44 and 4.46 respectively.

[5] proposes the well known transformer model. This is one of the most successful implementations of Self-Attention. Although, the transformer introduces a scaled-dot product mechanism paired with multi-headed self-attention. The architecture proposed in [5] is:

The major modules of the transformer model that make up the architecture include the encoder and decoder multi-head attention, feed forward respectively. The inputs initially are the word embeddings of the words being processed by the model and the inputs to the subsequent layers are the outputs of the preceding layers. The pictorial representation of scaled dot-product and multi-head attention is as follows:

One head of the attention module includes self attention with a scaling add-on; the Key-Query-Value structure stays. The Query and Key dot product is scaled by a value $\sqrt{d_{k}}$ where $d$ is the dimension of the input and is merely a scaling parameter to avoid explosion of values after the dot-product. The dot-product is then softmax-ed and operated with the value vector. This is essentially the math for calculation of the vector ${a_{i}}$ from equation (2) in the self-attention fashion. However, in transformer multiple ${a_{i}}$s are calculated with different weights. This gives a notion of extracting multiple feature maps as in Convolutional Neural Networks[19]. These vectors are then concatenated and then fed to a feed forward neural network layer. Similarly at the decoder the expected output is fed to the decoder layer and then combined at the ’multi-head-encoder-decoder-attention’ layer with the encoder outputs.

The transformer was trained on WMT’14 English-German dataset, 4.5 million sentence pairs. Sentences were encoded using byte-pair encoding [20], which has a vocabulary of about 37000 tokens at source and targets. For English-French, WMT’14 English-French dataset with of 36M sentences was used. 32000 word-piece vocabulary was used [17]. The transformers yielded state of the art results with the big transformer model scoring 28.4 BLEU on English-German dataset and 41 BLEU on English-French dataset. The transformer model aims not only at achieving state of the art results but also to do that with minimal computational complexity. To measure performance, [5] has calculated FLOPs (Floating Point Operations Per Second) which is 10-100 times better than the state of the art models.

Bidirectional Encoder Representations from Transformers[21] is google’s state of the art language modelling architecture based on the transformers model[5]. BERT essentially encourages the recent trends of transfer learning. The architecture is trained on bidirectional (apparently non-directional) transformer which tends to remove the benchmarks set by the recurrent models. Since the goal was to train a language model that could be modularized into the requirement by fine tuning the pretrained model, BERT essentially is a language model that introduces a novel approach called MLM (Masked LM). This involves ’masking’ 15% random words in a given sentence and using the bidirectional model to predict them.

While training Next Sentence Prediction (NSP) in BERT, two inputs are fed to the model, the source sentence (current sentence) and the target sentence (next sentence). For this, the authors of [21] trained with 50% of the sentences trained with wrong target sentences (i.e. a sentence which is out of context). The reason behind this being that the model should learn to be incisive in telling if the sentence belongs in the context of the given context.

BERT fine tuning implies appending an use case on the pretrained language model. [22], [23] discusses implementation of BERT at NMT. In [22] a concerted training framework(CTNMT) is introduced that serves the integration of the pretrained language model with NMT. CTMT consists of 3 modules viz. asymptotic distillation to make sure that the NMT model retains the previous pretrained context; dynamic switching gate to avoid forgetting of pretrained context; strategy to adjust the learning paces according to a scheduled policy[22]. In asymptotic distillation, [22] essentially the mean squared error between the final hidden state of the language model and the NMT model.

This loss is used in conjunction with the traditional cross-entropy loss of the NMT model by a weighted average on a hyper-parameter $\alpha$. This can be summarized mathematically as:

To address the problem of forgetting either the language model context or the NMT context, [22] introduces dynamic switch, which involves taking a weighted average of the hidden states of the language model and NMT respectively. The mathematical representation of the final hidden state is:

In addition to the aforementioned methods, the paper also introduces rate scheduled learning rate strategy to tune each component with different learning rate.

The key is to first converge the NMT model and then jointly train it with the language model and moderately tune only the NMT parameters to make sure the language model data isn’t forgotten.

The dataset to evaluate CTNMT used is WMT English-German translation task and also English-French, English-Chinese. CTNMT with only rate scheduling obtained a BLEU score of 29.7 on En-De dataset which outperforms the transformer (big) with almost 1 BLEU point. The same model with En-Fr and En-Zh yielded BLEU of 41.6 and 38.4 respectively. WIth CTMT and dynamic switch En-De has a BLEU score of 29.4, with En-Fr 41.6 and 38.6 with En-Zh. With Asymptotic Distillation, CTNMT gives 29.2, 41.6 and 38.3 on En-De, En-Fr, En-Zh respectively. Finally, with all the methodologies combined, BERT CTNMT gives state of the art results with BLEU scores 30.1, 42.3 and 38.9 on En-De, En-Fr and En-Zh respectively.

BLEU [3] is the most widely used evaluation metric in NMT tasks. It is marked by simplicity in calculation and closeness to human calculation. The cardinal task in calculating a BLEU score is simply comparing the reference sequence n-gram with the candidate sequence n-gram. BLEU shows resemblance to calculating precision score. To calculate precision, one may take the number of tokens in the candidate sequence that occur in the reference sequence and divide it to total number of tokens in the candidate sequence. However, candidates can overgenerate reasonable words and hence produce a high precision score. Thus, arises the need for exhausting a given reasonable token once it is matched to a token in the reference sequence. This is explained in the modified unigram precision method in [3]. Additionally, the proposed metric uses combined modified n-gram precision along with sentence brevity penalty. The calculation of BLEU can be summarized as :

First the mean of the test corpus’ modified precision scores is taken. Then the brevity penalty is calculated.

In addition to BLEU there are several metrics mentioned in this discussion viz. ROUGE score [10] to evaluate summarizers and glue score [24] to evaluate language models. Although these are out of scope for this discussion.

In this section, results from various previously discussed models are compared. In most of the approaches, the dataset used for evaluation is the WMT’14 En-Fr and En-De. other datasets used are WMT’16 English-Romanian, newtest2011 Fr-En, WMT’14 En-Zh. Along with these, some models are discussed that address to other use cases to delineate the Self-Attention Mechanism; these include evaluation on Stanford Treebank Dataset for binary sentiment analysis, Stanford Natural Language Inference (SNLI) Dataset for NLI [6], [7] and Textual Entailment[8], Twitter Tweets for author age prediction[8], Yelp for sentiment analysis[8] and CNN Daily Mail and New York Times data for abstractive text summarization[9]. The results obtained in the models discussed so far are summarized in Table I.

The comparisons in Table I clearly depict the evolution in Neural Machine Translation. However, certain additions to the current state of the art models may apparently increase the performance by decent figures. One add-on can be integrating the Memory Networks architecture which has been discussed in [25], [26], [27]. The intuition to this can be obtained by visualising as an external data structure implemented to maintain the contextual information and using a weighted sum approach like in any other attention mechanism select the most relevant parts from those context data structures. For example, in case of [22], an external data structure can keep the language model context for various similar sequences and a weighted average / sum can be taken from these external sources followed by application of the approaches implemented for forgetting NMT / LM contexts.

With Neural Machine Translation, almost all other (statistical, probabilistic, etc.) models for Machine Translation have been ruled out. To summarize, the approaches started with RNN-based methodologies which was later replaced by non-RNN methodologies which included considering all the tokens in a given sequence to preserve long term context and in a way eliciting an end-to-end notion in the model architecture. The state of the art language model, BERT has outperformed almost all the existing models in most of the NLP tasks by fine tuning the pretrained language model. The CTNMT model from [22] has outperformed all the discussed models in NMT tasks by appreciable differences in the evaluation scores. Neural Machine Translation tasks have use cases in a wide range of use cases including Government, Education, Medical domains and these demand exacting results. These models certainly can meet the expectations with room for improvement of course.