Contrastive Learning for Context-aware Neural Machine Translation Using Coreference Information

Context-aware machine translation has been vigorously studied to exploit the crucial context information in surrounding sentences. Recent works have shown that contextual information can help the model to generate not only more consistent but also more accurate translation Smith (2017); Voita et al. (2018); Müller et al. (2018); Kim et al. (2019).

In particular, Voita et al. (2018) introduced a context-aware Transformer model which is able to induce anaphora relations, Miculicich et al. (2018) showed that a model using cross-sentential contextual information significantly outperforms in document-level translation tasks, and Yun et al. (2020) insisted that context-aware models record the best performance especially in spoken language translation tasks where mandatory information tend to be sparse over multiple sentences.

The simplest method for context-aware machine translation is to concatenate all surrounding sentences and treat the concatenated sequence as a single sentence Tiedemann and Scherrer (2017). Although the concatenation strategy boosted Transformer architectures in multiple tasks Tiedemann and Scherrer (2017); Voita et al. (2018); Yun et al. (2020), it lagged behind efficiency as the Transformer architecture has limited long-range dependency Tang et al. (2018).

To improve the efficiency, an additional encoder module is introduced to encode only the context sentences Voita et al. (2018); Jean et al. (2017). Additionally, hierarchical structures also have been introduced because the context sentences do not have the same significance as the input sentences Miculicich et al. (2018); Yun et al. (2020).

The difference in coreference expressions among languages Zinsmeister et al. (2017); Lapshinova-Koltunski et al. (2020) gives MT systems a challenge on pronoun translation. Several recent works have attempted to incorporate coreference information Ohtani et al. (2019). The closest work to ours is Stojanovski and Fraser (2018) which also adds noises on creating a coreference-augmented dataset, while we do not add oracle coreference information directly to the training data.

One of the most common methods for data augmentation in NMT is back-translation that generates pseudo-parallel data from monolingual corpora using intermediate NMT models Sennrich et al. (2016a). Generally, back-translation is conducted at sentence-level, however, several works have proposed document-level back-translation Sugiyama and Yoshinaga (2019); Huo et al. (2020).

On the other hand, sentence corruption by removing or replacing word(s) has also been widely used for improving model performance and robustness Lample et al. (2018); Voita et al. (2019). Inspired by these works, we choose sentence corruption for contrastive learning.

Contrastive learning is to learn a representation by contrasting positive and negative (contrastive) examples. It has been succeed in various machine learning fields including computer vision Chen et al. (2020) and natural language processing Mikolov et al. (2013); Wu et al. (2020); Lee et al. (2021).

Recently, several approaches on contrastive learning for NMT have also been studied. Yang et al. (2019) proposed strategies for generating word-omitted contrastive examples and leveraging contrastive learning for reducing word omission errors on NMT. Pan et al. (2021) applied contrastive learning for multilingual MT and employed data augmentation for obtaining both the positive and negative training examples.

While these works have been conducted on sentence-level NMT settings, we focus on extending contrastive learning on context-aware NMT.

Generally, constrastive learning encourages a model to discriminate ground-truth and contrastive (negative) examples. In existing works, a number of approaches have been studied for obtaining contrastive examples:

• •

  Corrupting the sentence by randomly removing or replacing one or more tokens in the sentence. Yang et al. (2019)

• •

  Choosing irrelevant example in the batch or dataset. Pan et al. (2021)

• •

  Perturbations on representation space. Usually output vector of encoder or decoder is used. Lee et al. (2021)

CorefCL basically takes a similar approach as the first one, by the sentence corruption. However, unlike previous works that modify the source sentence, CorefCL modifies the contextual sentences to form contrastive examples. Specifically, we corrupt cross-sentential coreference mentions which occur between the source and its contextual sentences. This is based on the intuition that coreference is one of the core components of coherent translation.

More formally, steps to forming contrastive examples in CorefCL are as follows (see also Figure 2):

1. 1.

   Annotate the source documents automatically. We use the NeuralCoref¹¹1https://github.com/huggingface/neuralcoref to identify the coreference mentions between the source and its previous sentences as contextual sentences

2. 2.

   Filter the examples with cross-sentential coreference chain(s) between the source and contextual sentences. Around 20 to 30% of the training corpus is annotated in this way. See Section 5.1 for details

3. 3.

   For each coreference chain, mask every word in the antecedents with a special token. We also keep the original examples for training

4. 4.

   Masked words are replaced randomly with other words in vocabulary (word replacement), or omitted (word omission)

In the experiments, we take both of the corruption strategies. Precisely, the masked words are removed with a probability of 0.5, or randomly replaced otherwise. We found that this method is more effective compared to the methods using only one of the two corruption strategies. Please refer to the ablation study in Section 5.5 for more details.

Context-aware NMT models can implicitly capture dependencies between the source and contextual sentences. CorefCL introduces a max-margin contrastive learning loss to train the model to explicitly discriminate inconsistent contexts. This contrastive loss also encourages a model to be more sensitive to the contents of contextual sentences.

Formally, given the source $\mathbf{x}$, target $\mathbf{y}$, $n$ contextual sentences $C=[\mathbf{c}_{1},\cdots,\mathbf{c}_{n}]$ in the data $\mathcal{D}$, we first train the model by minimizing a negative log-likelihood loss, which is a common MT loss:

Once the model is trained with MT loss, we fine-tune the model with a contrastive loss. With a contrastive version of context $\tilde{C}$, our contrastive learning objective is minimizing a max-margin loss Huang et al. (2018); Yang et al. (2019):

Minimizing $\mathcal{L}_{CL}$ encourages the log-likelihood of the ground-truth to be at least $\eta$ larger than that of the contrastive examples. In our formulation, we want the model to be more sensitive to the subtle changes in the contextual sentences.

The contrastive loss is jointly optimized with MT loss since we empirically found that the joint optimization has yielded better performance than minimizing CL loss only as similar to Yu et al. (2020):

where $\alpha\in[0,1]$ is a weight for balancing between contrastive learning and MT loss. For simplicity, we fixed $\alpha$ during fine-tuning.

We experimented with CorefCL on various document-level parallel datasets: i) 3 English-German datasets including WMT document-level news translation²²2http://www.statmt.org/wmt19/translation-task.html Barrault et al. (2019), IWSLT TED talk ³³3https://wit3.fbk.eu/home Cettolo et al. (2017), OpenSubtitles’18⁴⁴4https://opus.nlpl.eu/OpenSubtitles-v2018.php Lison et al. (2018), and ii) our web-crawled English-Korean subtitles corpus.

For all tasks, we take every 2 preceding sentences as contextual sentences and we only consider sentences only within the same document (article, talk, movie, one episode of TV programs, etc.) of the source sentence. If split of the validation and the test set is not presented in the data, we apply document-based split to ensure that training and validation/test data is well-separated. Details of datasets are listed as follows:

WMT We use a set of parallel corpora annotated with document boundaries which is released in WMT’19 news translation task. Specifically, we combine Europarl v9, News Commentary v14, and MODEL-RAPID to form a training set containing 3.7$M$ examples and 0.85$M$ with cross-sentential coreferences. For validation and test sets, we used newstest2013 and newstest2019 which contain 3.05$k$ and 2.14$k$ examples respectively.

IWSLT The IWSLT dataset consists of transcriptions of TED talks in a variety of languages. We used the 2017 version of the training set, a combination of dev2010, tst2010, tst2015 as a validation set, and tst2017 as a test set. The resulting dataset consists of 232$k$ (50.3$k$ with cross-sentential coreferences), 3.5$k$, 1.2$k$ examples of train, dev, test sets respectively.

OpenSubtitles We also choose the English-German pair of OpenSubtitles2018 corpora. The raw corpus contains 24.4$M$ parallel sentences. We follow the filtering methods in Voita et al. (2019) by removing pairs that have a time overlap of subtitle frames less than 0.9. We also use separate documents for validation / test sets, resulting 3.9$M$ (1.01$M$ with cross-sentential coreferences), 40.7$k$, 40.5$k$ examples for train / validation / test sets respectively.

En-Ko Subtitles For English-Korean experiments, we first crawled approximately 6.1$k$ bilingual subtitle files from websites such as GomLab.com. Since sentence pairs of these subtitles are already soft-aligned by the creators so we applied a simple time-code based heuristics to filter examples. The final data contains 1.6$M$ (0.24$M$ with cross-sentential coreferences), 155.6$k$, and 18.1$k$ examples of consecutive sentences in the training, validation, and test sets respectively.

For preprocessing, all English and German corpus is tokenized first with Moses Koehn et al. (2007) tokenizer⁵⁵5https://github.com/moses-smt/mosesdecoder. We then apply the BPE Sennrich et al. (2016b) using SentencePiece⁶⁶6https://github.com/google/sentencepiece, and the size of the merge operation is approximately 16.5$k$. We also put a special token [BOC] at the beginning of contextual sentences to differentiate them from the source sentences.

We use model hyperparameters, such as the size of hidden dimensions and the number of hidden layers as same the transformer-base Vaswani et al. (2017), since all of the compared models are based on Transformer. Specifically, we set 512 as the hidden dimension, the number of layers is 6, the number of attention heads is 8, and the dropout rate is set to 0.1.

All models are trained with ADAM Kingma and Ba (2014) with different learning rates for each dataset. We employ early stopping of the training when the MT loss on the validation set does not improve. We start training each baseline model from scratch with random initialization and document-level dataset. Note that all the baseline models are not trained using iterative training as Zhang et al. (2018); Huo et al. (2020) which first trains the model from sentence-level task first, then document-level task. All the evaluated models are implemented on top of the transformer⁷⁷7https://github.com/huggingface/transformers framework.

We measure the translation quality by the BLEU score Papineni et al. (2002). For scoring BLEU, we use the sacreBLEU Post (2018) case-sensitive, detokenized scores for En-De, and case-insensitive scores with intl tokenizer for En-Ko task. We also report case-insensitive char-level scores on En-Ko for comparison.

We display the corpus-level test BLEU scores of all compared models on different tasks on Table 1. Among the baseline systems, all context-aware models show moderate improvements over the sentence-level (sent-level) baseline. These results are comparable to that of Huo et al. (2020) on the IWSLT task except for multi-enc-hier, and Yun et al. (2020) on OpenSubtitles task. One exception is a single-encoder model (concat) on WMT task, which seems due to the longer average sentence length.

We evaluated CorefCL by fine-tuning the context-aware models. Results show that models with CorefCL outperformed their vanilla counterparts, with the BLEU gain of up to 1.4 in En-De tasks, and 1.6/2.8 (detokenized/char-level BLEU) in the En-Ko subtitles task.

We observed that while CorefCL consistently improves BLEU on all tasks, it achieves better results on IWSLT and En-Ko subtitles tasks. Since improvements on much larger datasets like WMT and OpenSubtitles are smaller, we suggest that CorefCL also works as a regularization.

To assess how CorefCL improves the ability to deal with pronoun-related translations more in detail, we experiment our method with ContraPro⁸⁸8https://github.com/ZurichNLP/ContraPro. ContraPro is a contrastive test suit for En-De pronoun translation introduced by Müller et al. (2018). The evaluation is done by letting the model scores the German sentence with correct and incorrect pronoun translation, given the source and contextual English sentence. The accuracy is calculated by counting the number of correctly scored examples (i.e. correct examples that received a higher score than their incorrect counterpart).

We evaluate the models trained with WMT and OpenSubtitles tasks. We also list BLEU scores of En-De translation using the English source text in ContraPro. As shown in Table. 2, CorefCL significantly improves the baselines in scoring accuracy for all models by up to 5.5%, as well as slight improvements in BLEU scores.

One interesting finding is that CorefCL also achieved substantial accuracy gain on the models trained on WMT. Since the ContraPro is created from OpenSubtitles, WMT-trained models would yield lower performance because of domain shift between training and testing. Table.2 clearly shows the performance drop in BLEU, nevertheless, moderate improvements in accuracy can also be observed on WMT-trained models.

Ablation Study CorefCL uses the two corruption strategies for generating contrastive coreference mentions; word omission and word replacement. To make a better understanding of influence of these strategies, we evaluate CorefCL of different settings of these strategies.

As shown in Table.3, using both types of corruptions results in better performance. Removing one of the two strategies slightly degrades both the pronoun resolution accuracy and BLEU. Although not being significant, removing the word replacement has more impact on accuracy. This suggests that a standard context-aware model, at least for multi-enc-hier is less sensitive to the word substitution. The word replacement strategy can complement this behavior as resulted in better performance.

Qualitative Example We display a sample from ContraPro corpus and its translations made by multi-enc-hier model trained with OpenSubtitle task. In this example, since "coat" is translated as Mantel which is a masculine noun thus Er would be adequate translation of "It" instead of Sie which is feminine. While multi-enc-hier incorrectly translated "It" as Sie, the model fine-tuned with CorefCL correctly resolved it as Er.

In practice, context-aware models do not leverage target-side contexts struggle to maintain these kinds of coreference consistency Müller et al. (2018); Lapshinova-Koltunski et al. (2019) because of the asymmetric nature of grammatical components and data distributions. Results show that CorefCL can complement the limitation of source-only context-aware models.