Understanding Pre-Editing for Black-Box Neural Machine Translation

To collect fine-grained manual pre-editing instances, we adopted the protocol formalised by Miyata and Fujita (2017), in which a human editor incrementally and minimally rewrites an ST on a trial-and-error basis with the aim of obtaining better MT output. An original ST (Org-ST) and its pre-edited versions are collectively called a unit. Using an online editing platform we developed, editors implement the protocol in the following steps:

Step 1.

Evaluate the MT output of the current ST based on a 5-point scale criterion shown in Table 1. If the quality of the MT output is satisfactory (i.e., “Perfect” or “Good”), go to Step 4; otherwise, go to Step 2.

Step 2.

Select one of the versions of the ST in the unit to be rewritten and go to Step 3. If none of the versions are likely to become satisfactory through further edits, go to Step 4.

Step 3.

Minimally edit the ST³³3We operationally defined “to minimally edit” as “to modify an ST with a small edit that is difficult to be further decomposed into more than one independent edit, without inducing ungrammaticality in the edited sentence.” while maintaining its meaning, referring to the corresponding MT output. The MT output for the edited ST is automatically generated and registered in the unit. Return to Step 1.

Step 4.

Select one version of the ST that achieves the best MT quality (Best-ST) from among all the versions in the unit, and terminate the process for the unit.

The pre-editing instances in a unit collected through this protocol form a tree structure as shown in Figure 1. We refer to the shortest path between the Org-ST and the Best-ST as Best path. An important extension to the work in Miyata and Fujita (2017) is that our platform provides editors with a visualisation of the tree representation of the pre-editing history. This can facilitate the selection of ST versions in Step 2.

To extensively investigate pre-editing phenomena, we prepared the following conditions:

Translation directions:

We targeted Japanese-to-English (Ja-En), Japanese-to-Chinese (Ja-Zh), and Japanese-to-Korean (Ja-Ko) translations.

MT systems:

As black-box MT systems, we adopted Google Translate⁴⁴4https://translate.google.com/ and TexTra.⁵⁵5https://textra.nict.go.jp/ Both are general-purpose NMT systems that are prevalently used for translating Japanese texts into other languages.

Text domains:

We selected four text domains, whose linguistic characteristics, such as mode and sentence length, are different from each other (see Table 1 for details).

We randomly selected 25 Japanese sentences for each of the four text domains, and used the resulting ST set consisting of 100 sentences for all of the six combinations of translation direction and MT system. We assigned one editor to each translation direction. Each editor was asked to work with both MT systems, without being informed of the type of MT system used in the task. All editors were professional translators with sufficient writing skills in Japanese and experience for evaluating MT outputs. Before the commencement of the formal tasks, we trained the editors using example sentences so that they could become accustomed to the task and platform.

The Ja-En task was implemented from November to December 2019; the Ja-Zh and Ja-Ko tasks were implemented from December 2019 to February 2020.

Table 3 shows statistics for the pre-editing instances collected through the protocol described above. In general, the numbers of collected instances for the hospital and municipal domains were smaller than those for the bccwj and reuters domains, reflecting the influence of sentence length of the Org-ST. In other words, the shorter the sentence is, the fewer parts there are to be edited.

A notable finding is that while only about 11% (69/600) of the MT output for the Org-ST was of satisfactory quality, 95% (571/600) of the MT output of the Best-ST was satisfactory. This means that almost all the ST can be pre-edited into a form that can lead to satisfactory MT output, demonstrating the potential of both pre-editing and NMT.

The number of collected instances can be interpreted as the editing efforts required to obtain the Best-ST from the Org-ST. In most of the settings, the median number of collected instances for a unit falls in the range of 5 to 10. It is thus necessary to optimise the pre-editing process for an intended MT system. The length of the Best path approximates the minimum editing efforts needed to obtain the Best-ST. The total number of pre-editing instances in the Best path was 2,443, while the total of all instances is 6,652. This implies that there is substantial opportunity for reduction of the pre-editing efforts.

To quantify structural complexity, we used the following three indices:

1. (1)

   sentence length: the number of words per sentence⁶⁶6If ST instance includes multiple sentences, we averaged the scores.

2. (2)

   attachment distance: the averaged distance of all attachment pairs of the Japanese base phrases in a sentence

3. (3)

   dependency depth: the maximum distance from the root word in the dependency tree

We used the Japanese tokeniser MeCab⁷⁷7https://taku910.github.io/mecab/ to calculate (1) and the Japanese dependency parser JUMAN/KNP⁸⁸8http://nlp.ist.i.kyoto-u.ac.jp/index.php?KNP to calculate (2) and (3).

The first three blocks in Table 4 show the results for these indices. It is evident on all indices, the Org-ST exhibits the lowest scores. In other words, the length and surface complexity of the sentences generally increased through the pre-editing operations. This is a counter-intuitive finding in that most previous pre-editing practices have axiomatically assumed that shorter and less complex sentences are better for MT. We further delve into this in §5.

The remaining two blocks in Table 4 present statistics for the lexical characteristics of the STs. The results for lexical diversity indicate that both the total number of word types and the Token/Type ratio increased from the Org-ST to the Best-ST for all the conditions. This suggests that though the diversity of words increased slightly, the word distribution became peakier through pre-editing.

We also calculated the word frequency rank with Wikipedia as the reference.⁹⁹9We used the whole text data of Japanese Wikipedia obtained in October 2019 (https://dumps.wikimedia.org/). To assess the status of word frequency in relation to MT, it would be ideal to use the training data for each MT system, but such data are unavailable in black-box MT settings. Therefore, we decided to use Wikipedia as a convenient way to observe general word frequency. Lower numbers indicate higher word frequencies in Wikipedia. The 50th and 75th percentile values in the datasets imply that pre-editing induced the avoidance of low-frequency words.

To further inspect the differences between the Org-ST and the Best-ST, we extracted the word types (a) that appeared only in the Org-ST and (b) that appeared only in the Best-ST. Figure 2 illustrates the rank distributions of (a) and (b) for each condition. It is clear that low-frequency words with a frequency rank of around 10,000 decreased in the Best-ST, while words with a frequency rank of around 2,000–4,000 increased in the Best-ST. As Koehn and Knowles (2017) demonstrated, low-frequency words still pose major obstacles for NMT systems. Our results endorse this claim from a different perspective and can provide general strategies for word choice in the pre-editing task.

To understand the diversity of edit operations for pre-editing, we manually annotated the collected pre-editing instances in terms of linguistic operations. Given that the Best path contains effective editing operations for improved MT quality, we focused on the pairs of ST versions in the Best path (e.g., the pairs {1$\to$3, 3$\to$7, 7$\to$8} in Figure 1). We randomly selected 10 units for each of the 24 combinations of translation direction, MT system, and text domain, resulting in a total of 961 pre-editing instances. We then excluded 26 instances that could be decomposed into multiple smaller edits¹⁰¹⁰10Only 2.7% of the edits were not regarded as minimum, which demonstrated satisfactory adherence to our instructions, compared with the implementation by Miyata and Fujita (2017), in which 568 pre-editing instances were finally decomposed into 979 instances. and classified the remaining 935 instances, each of which consists of a minimum edit of ST, based on the typology proposed by Miyata and Fujita (2017). Through the classification, we refined the existing typology to consistently accommodate all the instances.

Table 5 presents our typology of editing operations with the number of instances in the different conditions. The typology consists of 39 operation types under 6 major categories, which enables us to grasp the diversity and trends of pre-editing operations. Compared to structural editing, local modifications of words and phrases were frequently used in the Best path. The dominant type is C01 (Use of synonymous words): content words are replaced by another synonymous word. This operation is important for achieving appropriate word choice in the MT output. C07 (Change of content), the second dominant type, includes the addition of information that is inferred by human editors based on the intra-sentential context or even external knowledge. For example, a named entity ‘Nemuro-sho’ (Nemuro office) was changed into ‘Nemuro-keisatsu-sho’ (Nemuro police office) by using the knowledge of the entity. It might be challenging to automate such creative operations.

It is also notable that S01 (Sentence splitting) only amounts to 1.5% of all instances, which supports the observation in §4.1 that in general, sentence length was not reduced, and even increased by pre-editing. Among the 14 cases of this type, nine of the split sentences were 60–67 words in length. These results support the empirical observation by Koehn and Knowles (2017) that NMT systems still have difficulty in translating sentences longer than 60 words, and suggest that sentence splitting may only be promising for such very long sentences.

Towards the effective exercise of pre-editing, we further analysed the pre-editing instances in terms of informational strategies based on the notion of explicitation/implicitation acknowledged in translation studies (Vinay and Darbelnet, 1958; Chesterman, 1997; Murtisari, 2016). Following these studies, we broadly defined explicitation as an act of indicating what is implied in the text to clarify its meaning and implicitation as the inverse act of explicitation. We classified all the instances analysed above except for the E01 and E02 types into three general strategies, namely, explicitation, implicitation, and (information) preservation. The right side of Table 5 shows the classification result. The total numbers of instances classified into each strategy were 329, 88, and 480, respectively. Not surprisingly, this indicates that explicitation is an essential strategy for effective pre-editing.

We also grouped all the 329 instances of explicitation into the following four subcategories.¹¹¹¹11See Appendix A for details.

Information addition

is the strategy of adding supplementary information, such as subjects, modality, and explanation, to clarify the content of the ST. For example, subjects were sometimes inserted as they tend to be omitted in Japanese sentences. This strategy generally corresponds to operation C07 (Change of content) described earlier.

Use of clear relation

includes structural changes and the use of explicit connective markers to make the relation between words, phrases, and clauses more intelligible. For example, the relation between the subject and object can be clarified by using the nominative case marker ‘ga’ in Japanese.

Use of narrower sense

is the strategy of replacing general words with more specific ones. For example, the verb ‘dasu,’ which has multiple meanings such as ‘put,’ ‘take,’ and ‘send,’ was replaced with the verb ‘teishutsusuru,’ which has a narrower range of meaning and was correctly translated as ‘submit.’

Normalisation

includes the use of authorised or standardised expressions, style, and notation. For example, elliptic sentence-ending was completed to construct a normal structure.

These strategies can be used as concise pre-editing principles for human editors and can guide researchers in devising effective tools for pre-editing. We also emphasise that these general informational strategies are not specific to the Japanese language and could be applied to other languages.

To grasp the general tendency, using all the collected pre-editing instances (see Table 3), we first calculated the correlation coefficients (Pearson’s $r$ and Spearman’s $\rho$) between the amount of edits (the TER and the number of edits) in the ST and in the MT. More formally, let $\mathtt{ST^{\prime}}$ be the pre-edited versions of $\mathtt{ST}$. For TER, the correlation is between $\mathtt{TER(ST,ST^{\prime})}$ and $\mathtt{TER(MT(ST),MT(ST^{\prime}))}$. For the number of edits, the correlation is between $\mathtt{EditCount(ST,ST^{\prime})}$ and $\mathtt{EditCount(MT(ST),MT(ST^{\prime}))}$.

As shown in Table 6, most coefficients are in the range of 0.15–0.25, suggesting a very weak correlation. This means that the change in NMT output is hardly predictable based on the amount of edits in the ST. For example, the replacement of a single particle in the ST sometimes caused drastic changes of lexical choices in the MT output.

The Japanese-to-Korean translation is an exception; in particular, the correlation coefficients of the TER for the Google NMT system, i.e., 0.580 for Pearson’s $r$ and 0.574 for Spearman’s $\rho$, indicate a moderate positive relationship between the changes in the ST and those in the MT. This is partly attributable to the fact that the syntactic structures of Japanese and Korean, including the word order and usage of particles, are substantially close. Thus, it is relatively easy to build sufficiently accurate MT systems.

Finally, using the pre-editing instances in the Best path analysed in §5, we further investigated to what extent each type of minimum editing operation affects the MT output. At this stage, we focused on the 28 editing types that have at least 10 instances, considering that it is difficult to derive reliable insights from fewer data.

Figure 3 presents the distribution of the degree of changes in the MT output when an ST is pre-edited, measured by $\mathtt{TER(MT(ST),MT(ST^{\prime}))}$. Most of the structural edits (S01–S04) resulted in sizeable changes in the MT. This is reasonable since structural modifications in the ST tended to cause major changes in the MT as well, leading to high TER. In contrast, many of the editing types that include local modifications of functional words and orthographic notations (F01–F03, F05, F06, O01, O02) did not have major impacts on the MT results.

It is worth noticing that P03 (Phrase reordering) did not drastically affect the MT output. In other words, recent NMT systems in practical use manage to retain the phrase-level equivalence even when the position of a phrase is shifted. The influence of P02 (Use/disuse of chunking marker(s)) is fairly significant. For human readers, the use of chunking markers, such as double quotes and square brackets, does not greatly affect the sentence parsing, but for NMT, it might seriously impinge on the tokenisation result, eventually leading to a large change in the final output.