Reconsidering SMT Over NMT for Closely Related Languages: A Case Study of Persian-Hindi Pair

In the current state of NLP affairs, the performance of attention-based Bahdanau et al. (2014); Vaswani et al. (2017) MT systems reaches BLEU scores of almost one. However, the underlying Neural Network (NN) architectures of such high-performing models, assume that the language pairs have a humongous diverse parallel corpora to achieve such desired performance. Of course, there are certain high-source language pairs such as English and French which benefit from those solutions but the MT system of other natural languages that utilize NN architecture without meeting the architecture’s assumptions are on the disadvantage side and will generate translations that are way far being accepted by native speakers.

Beyond the need for large datasets, another great concern when using NNs is their high power consumption and the environmental impact they leave behind—through processes such as training, inference, and experimentation, which all contribute to carbon footprints Faiz et al. (2023).

To walk through an efficient alternative path, we looked at the big picture of natural languages, focusing on their linguistic families and the factors that group languages. We have observed that the property of linguistic closeness of less divergent languages can be exploited. The key contributions of our paper are:

• •

  The first attempt to build a general domain MT system of Persian-Hindi languages.

• •

  We demonstrate that for structurally close language pairs having a moderate-sized (1M+ sentences) high-quality parallel corpus, SMT outperforms a Transformer-based NMT model.

• •

  Suggesting alternative paths to build computationally and environmentally efficient MT systems.

The upcoming sections are structured as follows: section 2 presents a review of the literature, followed by a detailed description of the parallel corpus used in our experiments and analysis in section 3. Section 4 outlines the experimental setup, while section 5 provides a comprehensive analysis of the results. Finally, section 6 concludes the paper and discusses potential directions for future research.

Before the revolutionization of the field by the Transformer architecture Vaswani et al. (2017), the notion of language closeness was being leveraged in various forms for different language pairs. In this section, we will see that our work is not only different from the perspective of studying a new language pair, Persian-Hindi, which to date no formal research has been conducted yet for the pair, but it also varies in terms of past Transformer comparison of the two architectures, NMT and SMT, given that we have access to a moderate amount of parallel sentences.

Previous works such as,Toral and Way (2015), hypothesize that translations between related languages tend to be more literal, with complex phenomena (e.g., metaphors) often transferring directly to the target language. In contrast, these phenomena are more likely to require complex translations between unrelated languages. Other instances such as Rios and Sharoff (2015), Kunchukuttan et al. (2017), and Kunchukuttan and Bhattacharyya (2017) utilize lexical similarities. Except Jauregi Unanue et al. (2018) which shows that for the scenario of low-resource (unlike our scenario which assumes medium resource) languages, SMT performs better than NMT, there is no comparative analysis of the NMT and SMT architectures for structurally similar languages with an assumption concerning the size of the parallel corpus.

In addition to the "Large Scale Colloquial Persian Dataset" (LSCP) Abdi Khojasteh et al. (2020), the other datasets we utilized are primarily sourced from OPUS Tiedemann (2016), a well-known repository for parallel corpora of various domains for a vast number of language pairs.

Table 3 shows the basic statistics related to all the corpora. After downloading those 10.9M+ sentences, we noticed that most of them were of low quality. To filter them we used LABSE Feng et al. (2020) during which Batheja and Bhattacharyya (2022)’s work was of help. Before the LABSE-filtration, the preprocessing steps for each corpus include, the removal of empty lines, punctuations, emojis, and deduplication of repeated parallel sentence pairs, normalization, and tokenization using language-specific libraries, indic-nlp-library Kunchukuttan (2020) for Hindi and ParsiNorm Oji et al. (2021) for Persian.

An additional time-taking step applied to the LCSP corpus was to pair the sentences first and then pass to the preprocessing phase. Then, we performed LABSE filtration which the dramatic reduction of the original corpus is shown in Table 1. In one of our SMT experiments, as we will see the details in the next section, the parallel sentences need to be Romanized, for which we employed uroman library Hermjakob et al. (2018). For all NMT experiments, the first step after receiving the raw data (Table 1) was to apply Byte Pair Encoding (BPE) Sennrich et al. (2016) with 32K merge operations.

It should be mentioned that for the Language Model (LM) component of the SMT model, we used the unfiltered monolingual of the target language, Hindi, and the corresponding numbers are detailed in Table 2.

To evaluate the structural similarity between Persian and Hindi, we analyzed the differences in sentence lengths (measured in token counts) across all parallel sentences. We assumed that if the majority of sentences exhibit a difference of three tokens or less, the alignment achieved through mgiza would represent an optimal one-to-one correspondence. As illustrated in the donut chart in Figure 1, more than 72% of the parallel sentences in our dataset, Table 1, have a length difference of less than three tokens. Moreover, through the language divergence (the phenomenon of languages expressing meaning in divergent ways) setting proposed by Dorr (1993), we studied the fact that the Persian-Hindi pair is almost isomorphic.

From the perspective of structure and syntax of German, Spanish, and English, Dorr proposes a set of seven types of divergences Bhattacharyya (2015). For our pair, we examine some types of syntactic divergence through examples provided in the Appendix A:

The first SMT experiment used the normalized filtered data of Table 1, applying it conventionally to SMT-Moses with the configurations detailed in Section 4. While Moses was processing the data, we simultaneously set up our encoder-decoder Transformer model. The best NMT model achieved 53.7, whereas the initial SMT model had a BLEU score 64.9. To verify the high BLEU score of SMT, we conducted a 4-fold cross-validation the BLEU scores of which are 67.32, 66.32, 64.90, and 66.74, respectively. Therefore, our best SMT model’s BLEU has been marked 66.32- the average of 4-fold’s BLEU. Also, by looking at the algorithmic nature of each architecture, it makes sense for the SMT to perform better than NMT in the existence of moderate data size. Because, SMT uses Expectation Maximization (EM) algorithm for alignment, and since the source and target languages are almost always one-to-one mapping, we need less data-size than that of NN.

Since Persian and Hindi share many common words, in our second experiment, parallel sentences were first Romanized to increase text similarity Hermjakob et al. (2018). However, the BLEU score dropped to 51.21 from 66.7. One highly probable reason is due to the diacritics (mark that is placed above, below, or through a letter to indicate how it should be pronounced) that some Persian words have in order to determine the phoneme of a word and hence the associated meaning. For example, gul (flower) and gel (mud) are two words the sound and meaning of which can be determined from the context (without diacritics) or using diacritics. Romanization often results in the loss of these nuances, leading to ambiguities in alignment and translation.

In the final experiment, we reversed the Persian scripts from right-to-left (RTL) to left-to-right (LTR) to align the writing direction of Hindi, expecting improved alignment quality. Unfortunately, this resulted in a further decrease in the BLEU score, falling to 48.74. This decline can be attributed to the fact that reversing a sentence alters its meaning. Although both Persian and Hindi are classified as free-word-order languages, we observed that the phenomenon where “only constructs that follow each other can be moved to any other position in the sentence while still preserving meaning” is compromised when inversion occurs, leading to a change in interpretation. See Appendix A for examples. Table 4 summarizes the BLEU scores for the different experiments we performed.

This study presents a comparative analysis of SMT and NMT for the Persian-to-Hindi language pair. Our findings demonstrate that SMT yields superior results in closely related languages, attributable to their shared linguistic structures. Additionally, we observed that reversing the order of Persian sentences from RTL to LTR negatively impacted the SMT model’s performance, resulting in a loss of meaning. In contrast, romanizing the input text showed a beneficial effect compared to the inversion experiment.

Future work will focus on deepening our understanding of these languages and exploring alternative approaches, such as translation through common space word embedding, transfer learning, and pivot-based NMT with English as a bridging language.