Extending Word-Level Quality Estimation for Post-Editing Assistance

According to Specia et al. (2020), word-level QE shown in Figure 1(a) is a task that takes a source sentence and its machine-translated counterpart (MT) as input. It then outputs tags for source words, MT words and gaps between MT words (MT gaps).¹¹1For convenience, source tags and MT word tags are collectively known as word tags. MT word tags and MT gap tags are collectively known as MT tags. All those tags are expressed either as OK or BAD. BAD indicates potential translation errors that post-editors should correct. We refer to such a task as original word-level QE.

Original word-level QE is not efficient enough for post-editing assistance because BAD is ambiguous. For example, in Figure 1(a), tag of “white” indicates a replacement of the mistranslation “黑” (black), but tag of “dogs” indicates an insertion into the gap between “猫” and “吗”. It is impossible to distinguish between these indications unless one attend to both entire sentences, which makes post-editing assistance meaningless.

We formally define a novel concept called extended word alignment between source sentence and MT. Ordinary word alignment indicates word-to-word relations between a pair of semantically equivalent sentences in two languages. Any word can be theoretically aligned with another semantically equivalent word on the other side. In contrast, in extended word alignment, translation errors in MT are considered. Specifically, a source word is allowed to be aligned with its mistranslation (wrong word choice) and a word is allowed to be aligned with nothing, namely null-aligned.

Extended word alignment can disambiguate BAD tags, overcoming the disadvantage of original word-level QE. When a BAD-tagged source word is aligned with a BAD-tagged MT word, it is clear that a replacement is needed. Likewise, a null-aligned BAD-tagged source word indicates an insertion and a BAD-tagged MT word is a deletion.

To make our idea more user-friendly, we formally propose a novel task called refined word-level QE by incorporating extended word alignment with original word-level QE. Besides extended word alignment, following refined tags are also included as the objectives.

• •

  REP is assigned to a source word and its mistranslation (wrong word choice) in MT, indicating a replacement.

• •

  INS is assigned to a source word and the gap where translation should be inserted in, indicating an insertion.

• •

  DEL is assigned to a redundant MT word, indicating a deletion.

In addition, we include correspondences between INS-tagged source words and MT gaps to express the insertion points. Those source-gap correspondences along with extended word alignment are collectively referred to as word-level correspondences.

Figure 1(b) is an example of our proposal. Compared with Figure 1(a), Figure 1(b) points out the replacement of “黑” (black), the insertions of “and” and “dogs” to the insertion point, and the deletion of “吗” (an interrogative voice auxiliary).

Extracting extended word alignment is non-trivial. Traditional unsupervised statistical tools(Och and Ney, 2003; Dyer et al., 2013) cannot work well because they expect semantically equivalent sentence pair as input. After trying several neural methods (Garg et al., 2019; Dou and Neubig, 2021), we empirically adopt the supervised method proposed by Nagata et al. (2020).

Specifically, extended word alignment extraction is regarded as a cross-lingual span prediction problem similar to the paradigm that utilizes BERT (Devlin et al., 2019) for SQuAD v2.0 (Rajpurkar et al., 2018). mBERT is used as the basic architecture. Given a source sentence with one word marked $S=[s_{1},s_{2},...,M,s_{i},M,...,s_{m}]$ ($M$ stands for a special mark token) and the MT $T=[t_{1},t_{2},...,t_{n}]$, mBERT is trained to identify a span $T_{(j,k)}=[t_{j},...,t_{k}](1\leq j\leq k\leq n)$ that is aligned with the marked source word $s_{i}$. Cross entropy loss is adopted during training.

$\mathcal{L}^{align}_{s_{i}}=-\frac{1}{2}[\mathrm{log}(p_{j}^{start})+\mathrm{log}(p_{k}^{end})]$

Because of the symmetry of word alignment, similar operations will be done again in the opposite direction. During testing, following Nagata et al. (2020), we recognize word pairs whose mean probability of both directions is greater than 0.4 as a valid word alignment. The image of the model is illustrated in Figure 2.

Nagata et al. (2020) demonstrated that the mBERT-based method significantly outperforms statistical methods in ordinary word alignment extraction. According to them, extracting word alignment for each word independently is the key to outperform other methods. Traditional methods model word alignment on a joint distribution, so that an incorrect previous alignment might cause more incorrect alignments like dominos. Our experiments prove that their method consistently works for extended word alignment.

For Original tags, we conduct sequence tagging with multilingual pre-trained language models including mBERT and XLM-R. Figure 3 shows the image. Input sequence is organized in the format of “[CLS] source sentence [SEP] MT [SEP]” without any mark tokens. Two linear layers followed by Sigmoid function transform output vectors into scalar values as respective probabilities of being BAD for each token. Formally, for a source sentence $S=[s_{1},s_{2},...,s_{i},...,s_{m}]$ and an MT $T=[t_{1},t_{2},...,t_{j},...,t_{n}]$, the total loss is the mean of binary cross entropy of all word tags.

$\mathcal{L}^{tag}_{s_{i}}=-[y_{s_{i}}\mathrm{log}(p_{s_{i}})+(1-y_{s_{i}})\mathrm{log}(1-p_{s_{i}})]$ $\mathcal{L}^{tag}_{t_{j}}=-[y_{t_{j}}\mathrm{log}(p_{t_{j}})+(1-y_{t_{j}})\mathrm{log}(1-p_{t_{j}})]$ $\mathcal{L}^{tag}=\frac{1}{m+n}(\sum\limits_{i=1}^{m}\mathcal{L}_{s_{i}}+\sum\limits_{j=1}^{n}\mathcal{L}_{t_{j}})$

We have also implemented our models with classification top-layers⁴⁴4Classification top-layers refers to a binary classification linear layer with Softmax., but we find that regression models are consistently better since we can adopt flexible threshold to offset the bias caused by imbalance of reference tags.

We use extended word alignment to refine the original word tags. Following the rules described in Section 3.3, we can refine word tags as Figure 4 shows. In practical situation, some BAD-tagged words are likely to be aligned with OK-tagged words. In that case, we change OK into BAD encouraging more generation of REP.

For gap tags, we adopt a method similar to the one described in Section 4.1. Specifically, we model source-gap correspondences as alignment between source words and MT gaps. We train a model that aligns an INS-tagged source word with a two-word span in MT where corresponding gap is surrounded. Figure 5 illustrates our idea. During testing, when a valid source-gap correspondence is confirmed, we tag the MT gap as INS⁵⁵5As for source words involved, we do not change their tags and trust the refinement based on extended word alignment because we believe extended word alignment is easier to model..

It is natural if we use such a method to determine gap tags based on the INS-tagged source words prediction from previous workflow. However, in experiment, we notice that absolute value of accuracy of INS-tagged source words is not high. In order not to be influenced by the previous wrong predictions, instead of treating this task as a downstream one, we conducted it independently.

We make full advantage of the En-De and En-Zh datasets of the shared task of original word-level QE in WMT20⁶⁶6http://www.statmt.org/wmt20/quality-es
timation-task.html. There are 7,000, 1,000, and 1,000 sentence pairs with annotation of tags respectively for training, development, and test set. Since the original datasets do not contain refined objectives, we additionally annotate the original development sets with all the objectives for refined word-level QE. Those annotated 1,000 pairs are further divided into 200 pairs for evaluation and 800 pairs for fine-tuning.

All the experiments are conducted with modified scripts from transformers-v3.3.1⁷⁷7https://github.com/huggingface/transfo
rmers on an NVIDIA TITAN RTX (24GB) with CUDA 10.1. For pre-trained models, we use bert-base-multilingual-cased for mBERT and xlm-roberta-large for XLM-R from Huggingface.

To train the model for original tags described in Section 4.2, we use the 7,000-pair training set provided by WMT20. 800 pairs of manually annotated data whose refined tags are degenerated into original tags are used for further training. Learning rate is set to 3e-5 and 1e-5 for mBERT and XLM-R respectively and both models are trained for 5 epochs. All the other configurations are remained unchanged as the default.

To train the models extracting extended word alignment described in Section 4.1, we utilize AWESoME (Dou and Neubig, 2021) to generate pseudo alignment data based on 7,000-pair WMT20 training set. We also use extra 800 sentence-pair annotated alignment data for fine-tuning. Models are pre-trained for 2 epochs and fine-tuned for 5 epochs with a learning rate of 3e-5. Most configurations are remained unchanged as the default but max_seq_length and max_ans_length are set to 160 and 15 following Nagata et al. (2020).

To train the model extracting source-gap correspondences described in Section 4.3, similar to what described above, we firstly adopt 7,000 sentence-pair WMT20 training set, generating pseudo data by randomly dropping out some target words in PE⁸⁸8Provided [P]ost-[E]dited sentence from MT in WMT20 dataset. It is regarded as the correct translations.. Then we link gaps where words are dropped with their source counterparts according to the source-PE alignment extracted by AWESoME. Also, 800 sentence-pair of gold source-gap correspondences are used for fine-tuning. All model configurations and training settings are kept identical as those of the model for extended word alignment extraction.

In Figure 7(a), our system basically succeeds in detecting errors caused by incorrect use of punctuations. Our system correctly suggests the replacements for the second comma and the half-width period. As for the first comma, the translation is still natural and acceptable if we delete the comma following the system’s suggestion. Moreover, our system successfully detects mistranslations of “passes” and “touchdowns”. In MT, those football terminologies are respectively translated as “通行证” (a pass to enter somewhere) and “摔倒” (falling down). It is noteworthy that those two mistranslations are not revised in post-edited corpus provided by WMT21. It implies that our system performs surprisingly well as it even succeeds in detecting mistranslations that is not noticed by human annotators.

In Figure 7(b), our system still works well in detecting incorrect use of half-width punctuations. However, compared with the reference, “abdominal aneurysm” is mistranslated and our model failed to detect it because both are tagged as OK during the prediction of original tags. A premature prediction of OK prevents a word from being refined into REP/INS/DEL later.

We believe that an inappropriate threshold mainly leads to such an issue. Predicted probabilities of “腹部” and “动脉瘤” are respectively 0.103 and 0.134, but the optimized threshold used is 0.88 as we searched it to maximize the MCC on the whole set. Meanwhile, probabilities of all other OK-tagged MT words are actually smaller than 0.01. As a result, if we set a threshold between 0.01-0.10 for this sentence pair, we could have obtained the perfect result. In the future, we plan to investigate into methods that can determine fine-grained optimized threshold for each sentence pair.

Figure 7(c) shows a typical En-De case that our model handles well. In German, it is more natural to indicate actions took place in the past in perfect tense rather than past tense. In this case, English verb “drafted” should be modified to “haben … ausgewählt”. Our model correctly suggests a correspondence between “drafted” and the MT gap in front of the period. As there are many cases need similar modification that inserts a particular word (like particle or infinitive for clause) before the period in MT, it is easier for our model to learn such laws. It probably explains the relatively good accuracy of INS in En-De experiments.

In contrast, Figure 7(d) is an En-Zh example showing that our model tends to align many source words with the gap right before or after their translation in MT even the translation is correct and needs no extra insertions. The word “dissected” is unnecessarily aligned with the gap around its translation “解剖”. Two human names are also unnecessarily aligned with gaps. As a result, four gaps are incorrectly tagged as INS. We observed the annotated dataset and noticed that many Chinese words in MT are slightly modified by adding prefixes and suffixes during post-editing. For example, “成年 海龟” (adult sea turtle) is modified to “成年的海龟” (adding “的” as a suffix for adjective). “演讲” (the speech) is modified to “这一演讲” (emphasizing “this” speech). Generally, those modifications are not necessary because of the free Chinese grammar. However, existence of those modifications might mislead the model into preferring to unnecessarily align a word with the gap around its translation like Figure 7(d). To address this issue, we plan to restrict the annotation rules to exclude meaningless modification in En-Zh training data in the future.