Using Perturbed Length-aware Positional Encoding
for Non-autoregressive Neural Machine Translation

Oka et al. (2020) incorporated random perturbation into the length constraints for length-difference PE (LDPE) Takase and Okazaki (2019). The perturbation is given as a random integer from a uniform distribution within a certain range in the training time. The perturbed LDPE (perLDPE) is added as follows:

$$perLDPE_{(pos,len,2i)}=sin\left(\frac{len-pos+per}{10000^{\frac{2i}{d}}}\right)$$

(1)

$$perLDPE_{(pos,len,2i+1)}=cos\left(\frac{len-pos+per}{10000^{\frac{2i}{d}}}\right),$$

(2)

where $pos$ is the absolute position in the sequence, $2i$ and $2i+1$ represent even and odd dimensions in the PE vector, $d$ is the embedding dimension, $len$ is the given output sequence length, and $per$ is the given random perturbation. The perLDPE is only used in the decoder. Oka et al. (2020) used the predicted length with Bidirectional Encoder Representations from Transformers (BERT) Devlin et al. (2018) as generation length constraints in English to Japanese translation. The translation accuracy improved significantly using oracle length constraints.

Knowledge distillation Hinton et al. (2015) is a method that uses the distilled knowledge learned by a stronger teacher model in the learning of a weaker student model. SKD gives a student model the output of a teacher model as knowledge. SKD propagates a wide range of knowledge by the teacher model to the student model and trains it to mimic its knowledge Kim and Rush (2016). NAT models rely on distilled data from SKD using AT models as teacher models.

A standard autoregressive Transformer is usually used for a teacher model in SKD for NAT. Since a standard Transformer often generates short sentences, this problem generally occurs even with distilled data. We incorporated perturbed length-aware PE into a teacher Transformer model to improve the quality of its outputs for better knowledge distillation because we can use ideal length constraints from the target language sentences during training. The perturbation applies only to training. The perturbed and non-perturbed LDPE are only used in the decoder.

We employed the Levenshtein Transformer Gu et al. (2019) as the NAT model. It has three decoders to insert placeholders and a word into each placeholder token and to delete unnecessary tokens. The encoder and the decoders have position embeddings. Although most NAT models output fixed-length sentences, the Levenshtein Transformer iteratively changes the output length by deletion and insertion. As demonstrated by the empirical study in Section 4.6, the Levenshtein Transformer Gu et al. (2019) often outputs a model that is shorter than the AT model. For this problem, we only incorporate perturbed length-aware PE into the placeholder decoder, which is considered length manipulation without sentence content. This perturbation is used only in the training time, as in the methods mentioned above. Note that the other two decoders still use position embeddings in the proposed method.

We used three translation tasks for the experiments: English to Japanese (En-Ja) using the Asian Scientific Paper Excerpt Corpus (ASPEC) Nakazawa et al. (2016), and English to German (En-De) and German to English (De-En) using WMT14 Bojar et al. (2014). From the ASPEC dataset, we used the first 1 million sentence pairs of the training set with 1,784 and 1,812 sentence pairs for the development and test sets. The WMT14 dataset consisted of 4.4 million sentence pairs for training a pre-processed one distributed by the Stanford Natural Language Processing (NLP) group¹¹1https://nlp.stanford.edu/projects/nmt/. We chose newstest 2013 (3,000 sentence pairs) and newstest 2014 (2,737 sentences) for the development and test sets. All sentences were tokenized into subwords using a SentencePiece model Kudo and Richardson (2018) with a shared subword vocabulary of 16,000 entries in ASPEC and 30,000 entries in WMT14. The length-aware PE used subword-based lengths in all experiments.

All models were implemented based on fairseq Ott et al. (2019). The hyperparameter settings were from fairseq NAT examples ²²2https://github.com/pytorch/fairseq/blob/master/examples/nonautoregressive_translation/README.md for both AT and NAT, except for the number of training epochs (50) and the batch size (18,000).

Figure 1shows the training process using the baseline and the proposed. The models are:

• •

  Teacher AT Transformer (AT baseline, $Transformer$)

• •

  Teacher AT Transformer with LDPE ($Transformer$ $w/perLDPE$)

• •

  Student NAT model and SKD using $Transformer$ (NAT baseline, $LevT$, and $MaskT$)

• •

  Student NAT model and SKD using $Transformer$ $w/perLDPE$

• •

  Student NAT model with LDPE (perturbation in training, $LevT+perLDPE$), and SKD using $Transformer$

• •

  Student NAT model with LDPE (perturbation in training, $LevT+perLDPE$), and SKD using $Transformer$ $w/perLDPE$

We used a standard Transformer with sinusoidal PE as a teacher AT model baseline in the knowledge distillation, a standard Levenshtein Transformer Gu et al. (2019), and a conditional masked language model (CMLM) with Mask-Predict Ghazvininejad et al. (2019) as a student NAT model. Each of the student models shared embedding parameters. We applied the proposed SKD described in 3.1 to two different NAT models: LevT and CMLM with Mask-Predict. We also applied the proposed NAT described in 3.2 using LevT.

We used perturbation ranges of $[-4,4]$ for the teacher AT model and $[0,2]$ for the student NAT models. We did not give negative perturbations to the student model to encourage long outputs. We also compared two different conditions in the embedding parameter matrices in the student NAT models: shared and independent.

The proposed NAT model described in Section 3.2 needs length prediction. For the length constraints of our student models in inference, we used BERT-based length prediction Oka et al. (2020) for En-Ja and a proxy by input length Lakew et al. (2019) for En-De and De-En.

We used BLEU Papineni et al. (2002) as our main quality evaluation metric. All BLEU scores were calculated by sacreBLEU Post (2018). In the En-Ja translation, the translation results were re-tokenized by MeCab Kudo (2005) after subword detokenization. We also investigated the LR of the output and reference sentences in the subword level to evaluate the output length.

Table 2 shows the BLEU and LR results. The latter showed that the baseline Levenshtein Transformer with the standard SKD resulted in shorter sentences than the Transformer. In the En-Ja and De-En experiments, the proposed method outperformed the baseline Levenshtein Transformer. However, BLEU decreased when we used shared embeddings in the En-Ja translation, although BLEU and LR improved when we used the independent embeddings. The De-En experiment results were different from the others; BLEU improved even with shared embeddings. The baseline Levenshtein Transformer resulted in a smaller LR in De-En than in the other tasks and many under-translations occurred. However, the proposed method did not outperform the baseline in the En-De translation.

On the other hand, the Mask-Predict model with the proposed SKD outperformed the baseline in the En-Ja translation; however, it was ineffective in the other tasks. Further investigation will be conducted in our future research.