Softmax Tempering for Training Neural Machine Translation Models

We experimented with the Asian Languages Treebank (ALT),¹¹1http://www2.nict.go.jp/astrec-att/member/mutiyama/ALT/ALT-Parallel-Corpus-20190531.zip comprising English (En) news articles consisting of 18,088 training, 1,000 development, and 1,018 test sentences manually translated into 11 Asian languages: Bengali (Bn), Filipino (Fil), Indonesian (Id), Japanese (Ja), Khmer (Km), Lao (Lo), Malay (Ms), Burmese (My), Thai (Th), Vietnamese (Vi), and Chinese (Zh). We focused on translation to and from English to each of these 11 languages. As a high-resource setting, we also experimented with the WMT 2019 English-to-German (En$\rightarrow$De) translation task.²²2http://www.statmt.org/wmt19/translation-task.html For training, we used the Europarl and the ParaCrawl corpora containing 1.8M and 37M sentence pairs, respectively. For evaluation, we used the WMT 2019 development and test sets consisting of 2,998 and 1,997 lines, respectively.

We evaluated softmax tempering on top of the Transformer model (Vaswani et al., 2017), because it gives the state-of-the-art results for NMT. More specifically, we employed the following models.

• •

  En$\rightarrow$XX and XX$\rightarrow$En “Transformer Base” models where XX is an Asian language in the ALT dataset.

• •

  En$\rightarrow$De “Transformer Base” and “Transformer Big” models.

We modified the code of the Transformer model in the tensor2tensor v1.14. For “Transformer Base” and “Transformer Big” models, we used the hyper-parameter settings in transformer_base_single_gpu and transformer_big_single_gpu, respectively where label smoothing of 0.1 is used by default. We used the internal sub-word tokenization mechanism of tensor2tensor with separate source and target language vocabularies of size 8,192 and 32,768 for low-resource and high-resource settings, respectively. We trained our models for each of the softmax temperature values, 1.0 (default softmax), 1.2, 1.4, 1.6, 1.8, 2.0, 3.0, 4.0, 5.0, and 10.0. We used early-stopping on the BLEU score (Papineni et al., 2002) for the development set which was evaluated every after 1k iterations. Our early-stopping mechanism halts training when the BLEU score does not vary by over 0.1 BLEU over 10 consecutive evaluation steps. For decoding, we averaged the final 10 checkpoints, and evaluated beam search with beam sizes (2, 4, 6 ,8, 10, and 12) and length penalties (0.6, 0.7, 0.8, 0.9, 1.0, 1.1, 1.2, 1.3, and 1.4) and greedy search.

We evaluated translation quality of each model using BLEU (Papineni et al., 2002) provided by SacreBleu Post (2018). The optimal temperature (${T_{\mathit{opt}}}$) for the tempered model was determined based on greedy search BLEU score on the development set. On the other hand, two hyper-parameters for beam search, i.e., beam size and length penalty, are not searched. The best beam search results reported in this paper are determined after computing BLEU scores on the test set, i.e., oracle. We used statistical significance testing³³3https://github.com/moses-smt/mosesdecoder/blob/master/scripts/analysis/bootstrap-hypothesis-difference-significance.pl to determine if differences in BLEU are significant.

Table 1 shows the greedy and beam search BLEU scores along with the optimal temperature (${T_{\mathit{opt}}}$) for translation to and from Asian languages and compare them against those obtained by non-tempered models. Except for a few language pairs, the greedy search BLEU scores of the best performing tempered models are higher than the beam search BLEU scores of non-tempered models.

Figure 1 shows how the greedy and beam search results vary with the temperature, taking En$\rightarrow$Ms translation as an example.⁴⁴4We show only one translation direction due to lack of space. Kindly, refer to our supplementary material for all translation directions. As the temperature is raised, the greedy search BLEU score increases and begins approaching the beam search score. At temperature values between 2.0 and 5.0, not only does the greedy search score exceed that of a non-tempered model, but that the greedy and beam search scores are barely any different. However, increasing the temperature beyond 10.0 always has a negative effect on the translation quality, because it leads to a highly smoothed probability distribution, quantified by high entropy, that does not seem to be useful for NMT training.

Consequently, we conclude that training with reasonably high temperature (between 2.0 and 5.0), softmax tempering has a positive impact on translation quality for extremely low-resource settings.

We also computed the similarity between the greedy and beam search (beam size 4, length penalty 1.0) translations by computing the BLEU score between them, regarding the greedy search results as references. Table 2 reports on the greedy-beam search similarities for 4 translation directions for several different values of temperature. For non-tempered models, the BLEU scores were around 30 to 60, and as the temperature increases, so does the BLEU score. This indicates that greedy and beam search results grow to be similar and the most likely reason is that training with softmax tempering forces the model to be extremely precise.

Table 3 gives the BLEU scores for the high-resource En$\rightarrow$De translation task. The results indicate that compared to the low-resource settings, relatively lower temperature values are effective for improving translation quality. While the improvements in translation quality are not as large as those in the low-resource settings, greedy and beam search improve by 0.8 to 2.3 BLEU points for temperature values around 1.2 to 1.6. We noticed that higher temperature values do bridge the gap between greedy and beam search performances. However, since they also reduce translation quality, we do not recommend using high temperature values in high-resource settings.

For the models trained only on the Europarl corpus (EP), the greedy and beam search performances of the Transformer Base model starts approaching those of the Transformer Big model.

Although training with softmax tempering makes it difficult for a model to over-fit the label distributions, we did not notice any impact on the training time. This indicates that the improvements are unrelated to longer training times.

With regard to decoding, in low-latency settings, we can safely use greedy search with tempered models given that it is as good as, if not better than, beam search using non-tempered models. Thus, by comparing the greedy and beam search decoding speeds, we can determine the benefits that softmax tempering brings in low-latency settings. Greedy search decoding of the Vi$\rightarrow$En⁵⁵5For ALT tasks, decoding times are very similar when translating into English due to it being a multi-parallel corpus. test set requires 37.6s on average, whereas beam search with beam sizes of 4 and 10 require 56.4s and 138.2s, respectively. For non-tempered models, where beam search scores are higher than greedy search scores by over 2.0 BLEU points, the best BLEU scores are obtained using beam sizes between 4 and 10. Given the improved performance with greedy search, we can decode anywhere from 1.5 to 3.5 times faster. Subjecting softmax tempering to model compression methods, such as weight pruning, might further reduce decoding time.

Why can regularization improve translation quality? One possible explanation is that the constraints imposed by regularization techniques force the model to more effectively utilize its existing capacity, indicated by the model parameters. This is especially important in low-resource settings. It is thus worth verifying the impact of softmax tempering, a regularization technique, on models with significantly reduced capacity.

In particular, we experimented with models that share parameters between the stacked layers, so-called recurrently stacked (RS) models (Dabre and Fujita, 2019), known to suffer from reduced translation quality for NMT. While we could have trained shallower models and/or models with small hidden sizes, they inevitably require experiments with combinations of hyper-parameters, i.e., number of layers and hidden size. In contrast, RS models are relatively recent and different from non-RS ones with regards to parameter sharing and thereby simplifying exploration.

Compare the first and second blocks of Table 4 for non-RS models and their RS counterparts. It is clear that RS models are always weaker than non-RS models, but the greedy search with RS models are largely improved by the tempered training. However, unlike in the case of a non-RS model, the beam search quality either remains the same or degrades in most cases. The major difference between RS and non-RS models is in the number of parameters and we can safely say that RS models simply lack the capacity to strongly benefit from softmax tempering. Nevertheless, the greedy search performance of an RS model with an appropriate temperature setting becomes comparable to a non-RS model trained without softmax tempering.

RS models tend to be 50-60% smaller than non-RS models (Dabre and Fujita, 2019). Thus, when one wants RS models for low-memory settings, we recommend to train them with softmax tempering.

By training a multilingual model, we share parameters between multiple language pairs thereby reducing the amount of model capacity available for individual language pairs. Additionally, using multilingual data is a way to simulate a high-resource situation. While softmax tempering is not always useful in unidirectional high-resource translation settings (see Section 4.5), it may benefit simulated high-resource settings realized though multilingual models for extremely low-resource language pairs.

We trained a one-to-many NMT model (Dong et al., 2015) for English to 11 Asian languages by concatenating the training data of the ALT corpus after prepending each source sentence with an artificial token indicating the target language as in Johnson et al. (2017). The ALT corpus is 12-way parallel, and thus the English side contains the same sentences 11 times. We used a vocabulary with 8,192 sub-words⁶⁶6In our preliminary experiment with a 32,768 sub-word vocabulary for English, we obtained lower BLEU scores than those of unidirectional models. We realized that the reason was that increasing the number of language pairs did not increase the English side vocabulary and thus these sub-words resulted in a word-level vocabulary which negatively affects translation quality. for English and another one with 32,768 sub-words for all target languages combined. As in Section 4, we trained Transformer Base models with different values for temperature.

Comparing the non-tempered models ($T=1.0$) in the first and third blocks of Table 4, it is clear that multilingual models already outperform corresponding unidirectional models by up to 6.0 BLEU points (En$\rightarrow$Ja), even though En$\rightarrow$Id and En$\rightarrow$Ms, which originally have the highest BLEU scores, suffer slightly. This highlights the utility of multilingualism in itself, since our training data is multi-parallel and introducing a new language pair does not introduce new contents. With softmax tempering, temperature values between 1.2 and 2.0 brought improvements in translation quality by up to 6.8 BLEU points over the best unidirectional models (marked bold in the first block). Note that the optimal temperature values, ${T_{\mathit{opt}}}$, are similar to those in the case of high-resource En$\rightarrow$De translation task. Although multilingualism simulates a high-resource setting, we observed that softmax tempering is more effective compared to a unidirectional modeling in the real high-resource setting. We thus recommend training multilingual models with softmax tempering, especially when the individual language pairs are low-resource.

The bottom block of Table 4 show results when we push parameter sharing to its extreme limits combining both RS and multilingualism. Comparing them with the third block, multilingual RS models without softmax tempering always outperform its non-RS counterpart by up to 1.9 BLEU points. However, multilingual RS models are negatively impacted by softmax tempering: the greedy search translation quality increases slightly (if at all), while the beam search translation quality degrades. Multilingual models already share parameters for 11 translation directions, significantly lowering the capacity per translation direction than unidirectional models. RS of layers further increases the burden on the multilingual model. Thus, temperature puts further burden on the model’s learning which has a negative impact on performance.

Ultimately, multilingual tempered models give the best possible translations. We thus recommend to train (a) multilingual RS models without softmax tempering aiming at extreme compactness or (b) multilingual non-RS models with softmax tempering for higher translation quality and faster decoding with greedy search.

Softmax tempering is a kind of regularization, since it makes the model predictions noisy during training, aiming to enable better learning. We have so far experimented with softmax tempering in combination with dropout. To this end, we experimented with both low-resource and high-resource settings using softmax tempering with and without dropout to explicitly examine their complementarity.

For low-resource settings, we randomly chose Bn$\rightarrow$En and Vi$\rightarrow$En translation directions, and additionally trained unidirectional Transformer Base models using softmax tempering in various temperature settings but without dropout. We also trained Transformer Base and Big models for the En$\rightarrow$De task without dropout. To be precise, to disable dropout, we set attention, embedding, and layer dropouts in the model to zero during training.

The results are shown in Table 5. It is known that disabling dropout significantly deteriorates the translation quality. However, in our Vi$\rightarrow$En and Bn$\rightarrow$En settings, higher temperature values are able to compensate for the drops. This shows that dropout and softmax tempering are complementary and that the latter acts as a regularizer similarly to the former. Note that softmax tempering alters the softmax layer, whereas dropout is applied throughout the model. This can explain why dropout has a larger impact than softmax tempering.

In the high-resource setting, in contrast, dropout does not positively impact on the translation quality. In the context of high-resource NMT, the impact of dropout on the Transformer was never explicitly investigated in detail. In such a situation, the large quantities of data might enable enough regularization. However, in the case of Transformer Base, temperature values of 1.2 to 1.6 tend to improve the greedy search performance when dropout is not used, bridging the performance gap between the Transformer Base and Transformer Big models. This shows that softmax tempering can be an alternative to dropout depending on the setting. This warrants further exploration of the relationship between data regularization and model regularization.

We expected that tempering leads to a smoother softmax and loss minimization using such a softmax makes it sharper as training progresses. With softmax tempering, the model will continue to receive strong gradient updates even during later training stages due to the deliberate perturbation of the softmax distribution. Thus, we confirm whether our model truly behaves this way through visualizing the softmax entropies and gradient values.

Figure 2 visualizes the variation of softmax entropy averaged over all tokens in a batch during training. The left-hand side shows the entropy of tempered softmax distribution in Equation (1), where there is no visible differences between charts with different values for temperature, i.e., $T$. Considering that the distribution is tempered with $T$, this indicates that the distribution of logits, $D_{i}$, is sharper when tempered with a higher $T$. The right-hand side plots the entropy of softmax distribution derived from the logits without dividing them by $T$. The lower entropy confirm that the distribution of logits is indeed sharper with higher $T$ and that division by $T$ as in Equation (1) counters the effect of sharpening. This means that the distribution of logits is forced to become sharper and thus produce exactly one word that the model believes the best to generate the rest of the sequence. Given the fact that translation quality is improved by softmax tempering, designing training methods that lead to sharper distribution might be useful.

Figure 3 shows the gradient norms during training with softmax tempering. This revealed that, similarly to ordinary non-tempered training, gradient norms in softmax tempering first increase during the warm-up phase of training and then gradually decrease. However, the major difference is that the norm values significantly decrease for the non-tempered training, whereas they are much higher for training with softmax tempering. Note that we re-scaled the loss for softmax tempering as in Equation (2), which is one reason why the gradient norms are higher. Larger gradient norms indicate that strong learning signals are being back-propagated and this will continue as long as the softmax is forced to make erroneous decisions because of higher temperature values.

We can thus conclude that the noise introduced by softmax tempering and subsequent loss re-scaling strongly affect the behavior of NMT models which eventually have an impact on translation quality.