Interplay of Machine Translation, Diacritics, and Diacritization

We collect an extensive set of $55$ language pairs where the target language is always English under different train sizes (five sizes for African languages and nine sizes for European languages, detailed in Section 4.2). For every pair of train size and language pair, e.g. ($125$k, fr-en) and ($5$k, bex-en) , we build four types of models as illustrated in Figure 1. We list each model type along with the corresponding research question in Table 1. For our single-task setting, there are three types of models: (i) models that perform MT and are trained with undiacritized source (OnlyMT_undia), (ii) models that perform MT and are trained with diacritized source (OnlyMT_dia), and (iii) models that perform diacritization (OnlyDia). The only distinction between the two OnlyMT models lies in whether diacritics are incorporated into the source sequences. For the multitask setting, (iv) a DiaMT model is trained to perform both diacritization and translation.

We use BLEU score Papineni et al. (2002) with SACREBLEU implementation Post (2018)¹¹1https://pypi.org/project/sacrebleu/ to measure the performance of MT. For diacritization, we adopt diacritization error rate (DER) and word error rate (WER) Abandah et al. (2015) with implementation details described in Appendix B.

We adopt transformer architecture Vaswani et al. (2017) for all models and train from scratch with the Fairseq library Ott et al. (2019), each using a single Nvidia A100 GPU. For train sizes $1$k, $2$k, $3$k, $4$k, $5$k, the number of steps is $30$k. For higher train sizes, we use $100$k steps for $25$k, $500$k steps for $125$k, $1.5$M steps for $625$k, and $3$M steps for $1$M train size. We evaluate our test set on the model with the best performance (lowest loss) on development set. Detailed information about hyperparamter settings, software version and license are included in Appendix Table A.3.

We observed code-switching phenomenon in the dataset. For example, a Spanish sentence may include French word(s). To ensure a clean comparison across these languages, we use fasttext tool Joulin et al. (2016b, a) to identify and remove lines with heavy code-switching.³³3lid.176.bin edition of language identification tool with access at https://fasttext.cc/docs/en/language-identification.html Specifically, we remove a line if the model prediction of the respective language is lower than $90\%$.⁴⁴4In spite of this measure, a manual inspection still uncovers a few examples of foreign characters in the data, which we assume have a minimal adverse effect on our experiments. We show the diacritical system extracted from the data in Table A.5 which may include foreign characters and diacritics. For African languages, since the domain is the Bible, we assume there are no foreign or code-switched texts. Therefore, we do not carry out any data cleaning for African languages. Furthermore, we remove overly long and short lines. Specifically, we remove lines with $>500$ or $<6$ characters.

To determine any interaction between performance and data sizes, we experiment with varying amounts of training data across different experiments. We now provide details of these train sizes for African and European languages.

The format of source and target of the processed data can be seen in Table 2. We handle non-English (source languages) and English (target language) data differently. For non-English data with diacritics, we (1) decompose every character carrying diacritic(s) into a base character and independent diacritic(s) with NFKD normalization,⁶⁶6https://unicode.org/reports/tr15/ (2) replace word-boundary whitespaces with the symbol ‘|’ to maintain information of word boundary after tokenization, (3) insert a whitespace between characters in preparation for whitespace tokenization, and (4) employ whitespace tokenization to build character-level vocabulary which includes characters and diacritics as tokens.⁷⁷7An exception is the vocabulary for OnlyMT_undia which has no diacritics because the source side is undiacritized and the target side is English, a language without diacritics Mihalcea (2002). Decomposing text with NFKD to retrieve independent diacritics and build character-level vocabulary enables better generalization of the model for rare combinations of a base character and diacritic(s). In addition, it helps avoid data sparsity that can occur if word or sub-word tokenization is used. For example, the probability distribution of the variants of ‘o’ in the African language Fon (fon) is skewed. The probabilities are about $60.8\%,38.1\%,1.1\%$ for o, ó, ǒ, respectively. Without decomposition, it could be very difficult for the model to learn a decent embedding representation for ǒ since there is a limited number of examples from which the model can capture its linguistic information. By making each diacritic a token, the model may be able to learn a generalized pattern for diacritic ˇ because it can learn its linguistic behavior in not only ǒ but also other characters that carry this diacritic in this language, e.g., ě, ǐ.

For English data, we tokenize it with whitespace to form word-level tokens. We strive to minimize the introduction of uncontrolled variables by utilizing word-level tokenization. Unlike word-level tokenization, BPE Sennrich et al. (2016b) and BPE-related implementations of subword tokenization can introduce additional uncontrolled variables to the experiments. In particular, the frequency component in BPE renders this method dependent on the corpus. The sampling and language model components in SentencePiece Kudo and Richardson (2018), render it both corpus-dependent and non-deterministic. If we adopt these methods, for a piece of text in English, it can be tokenized differently for different (1) language pairs and (2) train sizes. For (1), as an example, the word ‘review’ could be tokenized into [‘rev’, ‘iew’] in the fr-en language pair, but [‘re’, ‘view’] in the es-en language pair. Similarly for (2), ‘review’ can be tokenized differently in 25k and 1M train sizes. We use word-level tokenization to avoid inconsistency in tokenization. With word-level tokenization, a piece of English text is tokenized identically throughout different train sizes and language pairs. This enhances the comparability among different settings.

For DiaMT, we prepend a symbol (and a following whitespace), $\varepsilon$ for diacritization and $\tau$ for MT, at the beginning of every source sequence to prime the model which of the two tasks (translation or diacritization) to perform for a specific input sequence. The source side for both sub-tasks is identical, except the prepended symbol. The potential advantage of this design is that the model may be able to gain positive transfer via attaining cross-task knowledge.

When processing non-English data, we use whitespace to separate characters and the symbol ‘|’ to denote word boundaries. During post-processing for diacritization output, we consolidate the separated characters back into words and substitute the ‘|’ symbol with whitespace to properly indicate word boundaries. It is after this post-processing step that we compute DER and WER metrics. In contrast, when performing MT, post-processing is not required. This is because the output is always in English, a language we process straightforwardly from the outset, thereby eliminating the need for any post-processing adjustments.

To determine (1), we measure Diacritized Character Ratio (DCR), Diacritized Word Ratio (DWR), Diacritized Base character Ratio (DBR), and Diacritized Word Sentence Ratio (DWSR). To formulate the complexity metrics, for a corpus of any given language, let $c,c_{d}$ be the number of characters and diacritized characters; let $w,w_{d}$ be the number of words and words with at least one diacritized character; let $b$ be the number of unique base characters, $b_{d}$ be the number of unique character-diacritic(s) combinations and $s$ be the number of sentences. Then, $DCR=c_{d}/c$, $DWR=w_{d}/w$, $DBR=b_{d}/b$, and $DWSR=w_{d}/s$.

For (2), we measure Average Entropy of Diacritics (AED), and Weighted Average Entropy of Diacritics (WAED). AED serves as an assessment of the challenge faced by a diacritization model in diacritizing a character (including the decision not to diacritize). It is computed by averaging the entropies of the probability distribution of character-diacritic combinations for each base character. The more uniformly distributed they are, the more challenging it becomes for the model to make accurate predictions. WAED is the weighted edition of AED where the weight is the frequency of each base character.

It is important to mention that our proposed complexity metrics are theoretically data-dependent. That is, a single language can have different complexity metric values given different datasets and/or train sizes. However, empirically, as can be seen in Tables LABEL:tab:leveled_dia_complexity_metrics_afri and LABEL:tab:leveled_dia_complexity_metrics_euro, the values are similar across different train sizes for each language. This demonstrates that our proposed complexity metrics are robust among different sizes of training data and can capture the complexity of a diacritical system consistently. The proposed metrics are useful because (1) they provide a quantitative view of the diacritical system, (2) it is straightforward to compute them, and (3) they show high correlation with model performance as discussed later in Section 5.3.

We discuss findings to our research questions based on results reported in Table 4 and the visualization shown in Figure 2. We report a significance test with paired t-test for the performance of each pair of compared models, along with Cohen’s d  Cohen (1977) to estimate the effect sizes, as significance tests alone may not capture the magnitude of the effect Cumming (2013). To interpret Cohen’s d, we refer to the standard proposed in Sawilowsky (2009): 0.01 (very small), 0.2 (small), 0.5 (medium), 0.8 (large), 1.2 (very large), and 2.0 (huge).

RQ1a. Does diacritization benefit MT? As Figure 2 shows, on average, diacritization improves MT performance when train size is $\leq 5$k and harms MT performance when train size is $>5$k. For each individual language, the performance gain is in general positive for both African and European languages as can be seen in Appendix Figures C.1 and C.2.⁸⁸8The BLEU scores and exact percentage changes between DiaMT and OnlyMT_undia are shown in Appendix Tables LABEL:tab:bleu_afri_exact_values and LABEL:tab:bleu_euro_exact_values where some of the languages achieve over $300\%$ gain after adding diacritization when train size is $\leq 5$k. However, for $>5$k train sizes, adding diacritization in general harms MT performance. As the significance tests in Table 4 show, $p(DM,OM_{u})$, the p-values of paired $t$-test between the BLEU scores of DiaMT and OnlyMT_undia are lower than $0.01$ throughout all train sizes and language regions. This supports that adding diacritization will significantly affect MT performance, positively when $\leq 5$k, and negatively when $>5$k. We observe a gradual decrease of effect size from $1$k to $5$k for both African and European languages, and a rapid increase after $25$k for European languages. That is, the benefit of adding diacritization gradually reduces from $1$k to $5$k, and the harm grows rapidly after $25$k from small to huge.

The unexpected negative transfer effect on MT performance following the inclusion of diacritization as an auxiliary task in higher-resource scenarios warrants careful examination. While it might be tempting to attribute this to an inadequately sized model struggling to learn both tasks simultaneously, our analysis, as detailed in RQ1b, reveals a contrary trend. Interestingly, certain languages exhibit enhanced diacritization performance after the incorporation of MT, indicating that the model’s capacity is indeed sufficient to accommodate both tasks. Furthermore, the equitable distribution of data between MT and diacritization tasks, each constituting $50\%$, eliminates data imbalance as a contributing factor. Thus, the observed phenomenon likely originates from external variables, underscoring the need for further studies to pinpoint its underlying cause.

Thompson and Alshehri (2022) also find that when the dataset is large, Arabic diacritization can benefit from the addition of MT as an auxiliary task. Hence, we recommend adding MT to diacritization when training with $\geq 1$M train size because there potentially can be a performance boost. Even if not, the negative effect is manageable.⁹⁹9The DER/WER values and percentage change between DiaMT and OnlyDia are shown in Tables LABEL:tab:der_wer_afri_exact_values and LABEL:tab:der_wer_euro_exact_values.

After studying RQ1a and RQ1b, a notable asymmetry emerges in the relationship between MT and diacritization at higher-resource scenarios when introduced as auxiliary tasks. Specifically, while the inclusion of diacritization adversely affects MT performance, the incorporation of MT may yield benefits for diacritization. To summarize, we propose a guideline of either training in single-task or multi-task fashion in Figure 3, tailored to varying sizes of the training set.

Nonetheless, our experimental results show that the MT system perform indifferently regardless of diacritics of source language being kept or removed. The mean difference of BLEU scores between OnlyMT_undia and OnlyMT_dia is consistently around zero throughout all train sizes and languages of both regions as can be seen in Figure 2. As shown in Table 4, the $p$-values between the BLEU scores of OnlyMT_undia and OnlyMT_dia are consistently larger than $0.05$ when $<125$k for both African and European languages. When $\geq 125$k, there is inconsistency in the significance test results where we observe $p$ values being less than $0.05$ at $125$k and $1$M, but larger than $0.05$ at $625$k. At $125$k, $625$k, and $1$M, $95\%$, $50\%$, and $89\%$ of language pairs have better performance when source is diacritized, respectively. It seems that when $\geq 125$k, the existence of diacritics may benefit translation performance. However, with a closer look into Table LABEL:tab:bleu_euro_exact_values, the percentage changes of the two models for each language are in general around zero at $1$M train size. That is, the performance differences between two models are minimal at $1$M. Despite that the paired t-test shows significance at $125$k and $1$M, the Cohen’s d for $125$k and $1$M are $0.17$ and $0.08$, respectively. Both of them are between very small to small, indicating that the effects are little.

We speculate two potential reasons of the absent effect when diacritics are removed: (1) the contextual clues provided by adjacent words may enhance machine translation quality as effectively as the inclusion of diacritics. That is, MT systems are capable of inferring the missing information based on the contexts. As suggested in Adelani et al. (2021), an MT system may be capable of learning to disambiguate and generate correct translation even when diacritics are absent at the source side. (2) The infrequent incidence of ambiguity resulting from the removal of diacritics makes it negligible when assessing the performance difference between retaining and removing diacritics.

Despite that we observe minimal impact on MT performance whether diacritics are removed or retained as discussed in our RQ2, the comparison is between OnlyMT_dia and OnlyMT_undia among languages with all types of diacritical functions. To further explore the effect, we investigate whether the way diacritics function in each language influences model performance of MT. This is motivated by linguistic studies which find a reading cost in humans when diacritics that perform lexical functions are mismatched Labusch et al. (2023). We split the diacritical functions into lexical function, where diacritics influence the lexical semantics of a word and grammatical function, where the diacritics can change the grammatical structure of a sentence. Due to limited research on diacritics in African languages, our analysis concentrates on European languages. An overview of diacritical functions in these languages is provided in Appendix Table A.6. To conduct an analysis, we categorize European languages into three groups: lex only, gra only, lex+gra, which represent that diacritics have only lexical function, only grammatical function, and both, respectively. We inspect how different groups of diacritical functions will affect translation quality when diacritics are removed by comparing the average BLEU scores produced by OnlyMT_dia and OnlyMT_undia for each group at different train sizes.

We hypothesize that the removal of diacritics would harm languages whose diacritics have lexical function more than those having grammatical function, based on the assumption that grammatical information can be easier to infer from the contexts, compared to lexical information. Hence, we speculate that the differences between mean BLEU scores of OnlyMT_dia and OnlyMT_undia would be lex+gra $>$ lex only $>$ gra only where lex+gra having the largest difference because diacritics perform both functions for languages in this group and removing diacritics may lead to heavier loss in information compared to the other two groups. Experimental results, as can be seen in Figure 4, show that for train sizes $\leq 5$k, the differences of average BLEU scores are all around zero among the three different groups without an obvious pattern. However, for $\geq 25k$, there is a somewhat consistent order of lex+gra $>$ lex only $>$ gra only, except that the difference for lex only is slightly higher than lex+gra at $625$k; and gra only is slightly higher than lex only at $1$M. In part, the experimental results align with our hypothesis.

Although the results show a tendency of performance loss after removing diacritics being lex+gra $>$ lex only $>$ gra only, it is noteworthy that this finding does not guarantee that languages categorized in these three groups will always follow the order. This is due to the fact that the differences for all three groups are consistently around zero, within the range of 0.66 to -0.78 BLEU score, reflecting the effect of removing diacritics is minimal as discussed in RQ2. Furthermore, this analysis is not conclusive for two reasons: (1) The categorization into groups may overlook subtle but significant linguistic nuances, as languages within the same group might exhibit distinct linguistic characteristics despite their shared classification. (2) A thorough investigation with a representative dataset specifically designed to include ample instances of lexical ambiguity and sentences prone to grammatical ambiguity, after removal of diacritics, is necessary to definitively ascertain the relationship between diacritical functions and MT performance. That is, additional research in this area is needed.

We propose two classes of complexity metrics as discussed in Section 4.5. The complexity metrics quantify the complexity of the diacritical system of a given language and anticipate that the higher the values of complexity metrics, the more difficult to restore diacritics (i.e. the worse the performance metrics: DER and WER). As for correlation analysis, the proposed complexity metrics exhibit a consistently positive correlation with diacritization performance metrics across both African and European languages at all train sizes. For instance, the substantial difference in complexity metric DCR between Gidar (gid) at $0.001$ and Ndogo (ndz) at $0.258$ corresponds to a divergent performance metric DER of $0.097$ for gid and $0.330$ for ndz.¹⁰¹⁰10OnlyDia model at $5$k as shown in Table LABEL:tab:der_wer_afri_exact_values. We use the Train and Dev sets to compute complexity metrics while we measure performance on the Test set alone. We ensure that the data used to measure the complexity metrics and the data used to evaluate model performance are non-overlapping.

To assess the significance of these correlations, three measures, namely Pearson, Kendall, and Spearman correlations, were computed. The resulting p-values, which are predominantly lower than $0.05$ across African and European languages and different train sizes, indicate statistical significance. Examples of profoundly high correlations between complexity and performance metrics include (DCR, DER) with pearson correlation at $0.885$, and (WAED, WER) at $0.788$ at $1$M train size. A high correlation observed with larger training sizes bolsters confidence in the efficacy of the proposed complexity metrics. This finding solidifies the belief that the proposed metrics effectively quantify the complexity of the diacritical system of a language. The correlations between the proposed complexity metrics and DER/WER are detailed in Appendix Table E.3 for African languages and Table E.4 for European languages.

There are two exceptions to the strong correlations: DBR and AED. These metrics occasionally exhibit lower correlation with DER and WER. We speculate that (1) DBR in European languages can be biased due to the inclusion of foreign text, as discussed in Section 4.1. This may bring about the lower correlation between DBR and performance metrics. (2) The absence of taking character occurrence frequency into consideration may negatively influence the effectiveness of AED. To support this speculation, WAED, the weighted version of AED which takes frequency into consideration shows a high correlation with performance metrics across all train sizes and both language regions.