Adam Mickiewicz University at WMT 2022:
NER-Assisted and Quality-Aware Neural Machine Translation

For our initial experiments, we used transfer learning Aji et al. (2020); Zoph et al. (2016) from the high-resource Czech$\rightarrow$English language pair. We used only the parallel data provided by the organizers to train the model in this direction. In this case, we created a single joint vocabulary for three languages (Czech, English, Ukrainian), consisting of 32,000 units. The Czech$\rightarrow$English model was fine-tuned for the Ukrainian$\rightarrow$Czech and Czech$\rightarrow$Ukrainian language directions. Our later experiments showed that there were no gains in translation quality compared with models trained from scratch using separate vocabularies for source and target languages – the upside was that the models took less time to converge during training.

We used models created by the transfer learning approach to produce synthetic training data through noisy back-translation Edunov et al. (2018). Specifically, we applied Gumbel noise to the output layer and sampled from the full model distribution. We used monolingual data available in the constrained track, which included all ~59M Ukrainian sentences after filtering and ~60M randomly selected Czech sentences.

After training the model with concatenated parallel and back-translated corpora, we replaced the training data with filtered parallel data and further fine-tuned the model. We kept the same settings as in the first training pass, training the model until it converged on the development set.

Translation in domains such as news, social or conversational texts, and e-commerce is a specialized task, involving such challenges as localization, product names, and mentions of people or events in the content of documents. In such a case, it proved helpful to use off-the-shelf solutions for recognizing named entities. For Czech, the Slavic BERT model Arkhipov et al. (2019) was used, with which entities such as persons (PER), locations (LOC), organizations (ORG), products (PRO), and events (EVT) were tagged. Due to the lack of support for the Ukrainian language in the Slavic BERT model, the Stanza Named Entity Recognition module Qi et al. (2020) was used to detect entities in the Ukrainian text, recognizing persons (PER), locations (LOC), organizations (ORG), and miscellaneous items (MISC). With these ready-made solutions, the parallel and back-translated corpora were tagged. The named entity categories were then numbered to assign appropriate source factors to words in the text, supporting the translation process. The source factors were later transferred to subwords in a trivial way.

Source factors Sennrich and Haddow (2016) have previously been used to take into account various characteristics of words during the translation process. For example, morphological information, part-of-speech tags, and syntactic dependencies have been added as input to neural machine translation systems to improve the translation quality.

In the same way, it is possible to add information about named entities found in the text Modrzejewski et al. (2020), making it easier for the model to translate them correctly. However, the AMU machine translation system does not distinguish between inside-outside-beginning (IOB) tags Ramshaw and Marcus (1995), treating the named entity tag names as a whole. Specifically, we introduce the following source factors:

• •

  p0: source factor denoting a normal token,

• •

  p1: source factor denoting the PER category,

• •

  p2: source factor denoting the LOC category,

• •

  p3: source factor denoting the ORG category,

• •

  p4: source factor denoting the MISC category,

• •

  p5: source factor denoting the PRO category,

• •

  p6: source factor denoting the EVT category.

An example of a tagged sentence is shown in Figure 1.

Models were trained in two settings: concatenation and sum. In the first setting, the factor embedding had a size of 16 and was concatenated with the token embedding. In the second setting, the factor embedding was equal to the size of the token embedding (1024) and was summed with it.

As shown in Table 4, we observe an increase in the string-based evaluation metrics (chrF and BLEU) while COMET scores remain about the same. This is in accordance with Amrhein and Sennrich (2022), who show that COMET models are not sufficiently sensitive to discrepancies in named entities.

Table 2 presents the numbers of recognized named entity categories in the training, development and test data.

Our work on document-level translation is based on a simple data concatenation method, similar to Junczys-Dowmunt (2019) and Scherrer et al. (2019).

As our training data, we use parallel document-level datasets (GNOME, KDE4, TED2020, QED), as well as synthetically created data, concatenating random sentences to match the desired input length. Specifically, we merge datasets created in the following ways as a single, large dataset:

• •

  Curr $\rightarrow$ Curr: sentence-level parallel data,

• •

  Prev + Curr $\rightarrow$ Prev + Curr: previous sentence given as a context,

• •

  50T $\rightarrow$ 50T: a fixed window of 50 tokens after subword encoding,

• •

  100T $\rightarrow$ 100T: a fixed window of 100 tokens after subword encoding,

• •

  250T $\rightarrow$ 250T: a fixed window of 250 tokens after subword encoding,

• •

  500T $\rightarrow$ 500T: a fixed window of 500 tokens after subword encoding.

By concatenating such datasets, we allow the model to gradually learn how to translate longer input sequences. It is also capable of sentence-level translation. To separate sentences from each other, we introduced a <SEP> tag. An example of a document-level input sequence is shown in Figure 2. All data used to train the document-level model were tagged with NER source factors, including the back-translated data.

We created a weighted ensemble of four best-performing models. It consisted of the following models:

• •

  (A) sentence-level models trained with NER source factors (concat 16),

• •

  (B) sentence-level model trained with NER source factors (sum),

• •

  (C) document-level model trained with NER source factors (concat 16).

In this case, the document-level model was used only for the sentence-level translation. The optimal weights for each model were selected using a grid search method. For the specific language pairs, we used the following model and weight combinations:

• •

  Czech $\rightarrow$ Ukrainian: 1.0 $\cdot$ (2$\times$A) + 0.8 $\cdot$ (B) + 0.6 $\cdot$ (C),

• •

  Ukrainian $\rightarrow$ Czech: 1.0 $\cdot$ (2$\times$A) + 0.8 $\cdot$ (B) + 0.4 $\cdot$ (C).

Having the final model ensemble, we created an n-best list containing 200 translations for each sentence with beam search. Then we merged it with a second n-best list containing 50 translations for each sentence, created by a single document-level model with document-level decoding. This enabled the use of quality-aware decoding Fernandes et al. (2022).

We applied a two-stage quality-aware decoding mechanism: pruning hypotheses using a tuned reranker (T-RR) and minimum Bayes risk (MBR) decoding Kumar and Byrne (2002, 2004), as shown in Figure 3.

First, we tuned a reranker on the development set, using as features NMT model scores, as well as existing QE models based on TransQuest Ranasinghe et al. (2020) and COMET Rei et al. (2020), which are based on Direct Assessment (DA) Graham et al. (2013) scores or MQM Lommel et al. (2014) scores. Specifically, we used:

• •

  model ensemble log-likelihood $\log p_{\theta}(y|x)$ scores,

• •

  TransQuest QE model trained on DA scores (monotransquest-da-multilingual),

• •

  COMET QE model trained on MQM scores (wmt21-comet-qe-mqm),

• •

  COMET QE model trained on DA scores (wmt21-comet-qe-da).

We tuned the feature weights to maximize the COMET reference-based evaluation metric value using MERT Och (2003).

After tuning the reranker, we used it to prune the n-best list from 250 to 50 hypotheses per input sentence. The resulting n-best list was used for minimum Bayes risk decoding, using the COMET reference-based metric as the utility function. Minimum Bayes risk decoding seeks, from the set of hypotheses, the hypothesis with the highest expected utility.

Equation 1 shows that the expectation can be approximated as a Monte Carlo sum using model samples $y^{(1)},\ldots,y^{(M)}\sim p_{\theta}(y|x)$. In practice, the translation with the highest expected utility can be chosen by comparing each hypothesis $y\in\bar{\mathcal{Y}}$ with all other hypotheses in the set.

The described two-stage quality-aware decoding process allowed us to further optimize our system for the COMET evaluation metric, which has been shown to have a high correlation with human judgements Kocmi et al. (2021).

The final step involved post-processing. We applied the following post-processing steps for each best obtained translation:

• •

  transfer of emojis from the source to the translation using word alignment based on SimAlign Jalili Sabet et al. (2020),

• •

  restoration of quotation marks appropriate for a given language,

• •

  restoration of capitalization (e.g. if the source sentence was fully uppercased),

• •

  restoration of punctuation, exclamation and question marks (if a source sentence ends with such a mark, we make the translation do likewise),

• •

  replacement of three consecutive dots with an ellipsis,

• •

  restoration of bullet points and enumeration (e.g. if the source sentence starts with a number or a bullet point),

• •

  deletion of consecutively repeated words.

The General MT Task tests the MT system’s performance on multiple domains. Therefore, we investigated the possibility of improving our translation system with the on-the-fly domain adaptation method.

This experiment was based on Farajian et al. (2017). Our idea was to retrieve similar sentences from the training data for each input sentence and to fine-tune the model on their translations. After the translation of a single sentence is complete, the model is reset to the original parameters. We used Apache Lucene McCandless et al. (2010) as our translation memory to search for similar sentences. We indexed all of the training data and used the Marian dynamic adaptation feature. We compared the translation quality with and without the retrieved context. The experiments were carried out with a different similarity score used to choose similar sentence pairs for the fine-tuning process. We empirically modified the learning rate and the number of epochs to find optimal values that improved the translation quality.

Table 3 shows the results of the aforementioned experiments on the full development set. We found that only a small number of sentences in the training data were similar to those present in the development set. The results showed that tuning the model on similar sentences from the training data did not significantly improve translation quality. In the end, we decided not to use this method in our WMT 2022 submission.