Automatically Creating a Large Number of New Bilingual Dictionaries

We first develop a direct translation method which we call the DT approach (see Algorithm 1). The DT approach uses transitivity to create new bilingual dictionaries from existing dictionaries and an MT. An existing dictionary Dict(S,R) contains alphabetically sorted LexicalUnits in a source language S and each has one or more Senses in the language R. We call such a sense $Sense_{R}$. To create a new bilingual dictionary Dict(S,D), we simply take every pair <LexicalUnit,$Sense_{R}$> in Dict(S,R) and translate $Sense_{R}$ to D to generate translation candidates candidateSet (lines 2-4). When there is no translation of $Sense_{R}$ in D, we skip that pair <LexicalUnit,$Sense_{R}$>. Each candidate in candidateSet becomes a $Sense_{D}$ in language D of that LexicalUnit. We add the new tuple <$LexicalUnit,Sense_{D}$> to Dict(S,D) (lines 5-7).

An example of generating an entry for a Dict(asm,vie) using the DT approach from an input Dict(asm,eng) is presented in Figure 1.

To handle ambiguities in the dictionaries created, we propose the IW approach as in Figure 2 and Algorithm 2.

For each $Sense_{R}$ in every given LexicalEntry from Dict(S,R), we find all Offset-POSs⁴⁴4Synset is a set of cognitive synonyms. Offset-POS refers to the offset for a synset with a particular POS, from the beginning of its data file. Words in a synset have the same sense. in the Wordnet of the language R to which $Sense_{R}$ belongs (Algorithm 2, lines 2-5). Then, we find a candidate set for translations from the Offset-POSs and the destination language $D$ using Algorithm 3. For each Offset-POS from the extracted $\emph{Offset-POS}s$, we obtain each $word$ belonging to that Offset-POS from different Wordnets (Algorithm 3, lines 2-3) and translate it to D using a MT to generate translation candidates (Algorithm 3, line 4). We add translation candidates to the $candidateSet$ (Algorithm 3, line 6). Each candidate in the $candidateSet$ has 2 attributes: a translation of the word $word$ in the target language D, the so-called $candidate.word$ and the occurrence count or the rank value of the $candidate.word$, the so-called $candidate.rank$. A candidate with a greater rank value is more likely to become a correct translation. Candidates having the same ranks are treated similarly. Then, we sort all candidates in the $candidateSet$ in descending order based on their rank values (Algorithm 2, line 7), and add them into the new dictionary Dict(S,D) (Algorithm 2, lines 8-10). We can vary the Wordnets and the numbers of Wordnets used during experiments, producing different results.

Figure 3 shows an example of creating entries for Dict(asm,arb) from Dict(asm,eng) using the IW approach.

Our approach is general, but to demonstrate the effectiveness and usefulness of our algorithms, we have carefully selected a few languages for experimentation. These languages include widely-spoken languages with limited computational resources such as arb and vie; a language spoken by tens of millions in a specific region within India, viz., asm with almost no resources; and a couple of languages in the UNESCO’s list of endangered languages, viz., dis and ajz both from northeast India, again with almost no resources at all.

We work with 5 existing bilingual dictionaries that translate a given language to a resource-rich language, which happens to be eng in our experiments: Dict(arb,eng) and Dict(vie,eng) supported by Panlex⁵⁵5http://panlex.org/; Dict(ajz,eng) and Dict(dis,eng) supported by Xobdo⁶⁶6http://www.xobdo.org/; one Dict(asm,eng) created by integrating two dictionaries Dict(asm,eng) provided by Panlex and Xobdo. The numbers of entries in Dict(ajz,eng), Dict(arb,eng), Dict(asm-eng), Dict(dis,eng) and Dict(vie,eng) are 4682, 53194, 76634, 6628 and 231665, respectively. Many LexicalEntrys in some of our input dictionaries have no POS information. For example, 100% and 6.63% LexicalEntrys in Dict(arb,eng) and Dict(vie,eng), respectively, do not have POS.

To solve the problem of ambiguities, we use the PWN and Wordnets in several other languages linked to the PWN provided by the Open Multilingual Wordnet project (?): FinnWordnet (?) (FWN), WOLF (?) (WWN) and Japanese Wordnet (?) (JWN). We choose these Wordnets because they are already aligned with the PWN, cover a large number of synsets of the about 5,000 most frequently used word senses and are available online⁷⁷7http://compling.hss.ntu.edu.sg/omw/ for free. Depending on which Wordnets are used and the number of intermediate Wordnets, the sizes and qualities of the new dictionaries created change. The Microsoft Translator Java API⁸⁸8https://datamarket.azure.com/dataset/bing/microsofttranslator is used as another main resource. The Microsoft Translator supports translations for 50 languages.

In our experiments, we create dictionaries from any of {ajz, arb, asm, dis, vie} to any non-eng language supported by the Microsoft Translator, e.g., arb, Chinese (cht), German (deu), Hmong Daw (mww), Indonesian (ind), Korean (kor), Malay (zlm), Thai (tha), Spanish (spa) and vie, as shown in Figure 4.

Ideally, evaluation should be performed by volunteers who are fluent in both source and target languages. However, for evaluating created dictionaries, we could not recruit any individuals who are experts in two appropriate languages⁹⁹9This is not surprising. Considering languages we focus on are disparate belonging to different classes, with provenance spread out around the world, and frequently resource poor and even endangered. For example, it is almost impossible to find an individual fluent in the endangered language Karbi and Vietnamese.. Hence, every dictionary is evaluated by 2 people sitting together, one using the target language as mother tongue, and the other the source language. Each volunteer pair was requested to evaluate using a 5-point scale – 5: excellent, 4: good, 3: average, 2: fair and 1: bad. For selecting the samples to evaluate, we follow the concept of “simple random method” (?). According to the general rules of thumb, we can be confident of the normal approximation whenever the sample size is at least 30 (?, page 310). To achieve reliable judgment, we randomly picked 100 translations from every dictionary created.

To study the effect of available resources used to create our new dictionaries, we first evaluate the input dictionaries. The average score of LexicalEntrys in the input dictionaries Dict(arb,eng), Dict(asm,eng) and Dict(vie,eng) are 3.58, 4.65 and 3.77, respectively. This essentially means that Dict(asm,eng) is of almost excellent quality, while the other two are of reasonably good quality. These are the best dictionaries we could find. The average score and the number of LexicalEntrys in the dictionaries we create using the DT approach are presented in Table 1. We believe that if we find “better” input dictionaries, our results will be commensurately better.

The average scores and the numbers of LexicalEntrys in the dictionaries created by the IW approach are presented in Table 2 and Table 3, respectively. In these tables, Top n means dictionaries created by picking only translations with the top n highest ranks for each word, A: dictionaries created using PWN only; B: using PWN and FWN; C: using PWN, FWN and JWN; D: using PWN, FWN, JWN and WWN. The method using all 4 Wordnets produces dictionaries with the highest scores and the highest number of LexicalEntrys as well.

The number of LexicalEntrys and the accuracies of the newly created dictionaries definitely depend on the sizes and qualities of the input dictionaries. Therefore, if the sizes and the accuracies of the dictionaries we create are comparable to those of the input dictionaries, we conclude that the new dictionaries are acceptable. Using four Wordnets as intermediate resources to create new bilingual dictionaries increases not only the accuracies but also the number of LexicalEntrys in the dictionaries created. We also evaluate several bilingual dictionaries we create for a few of the language pairs. Table 4 presents the number of LexicalEntrys and the average score of some of the bilingual dictionaries generated using the four Wordnets.

The dictionaries created from arb to other languages have low accuracies because our algorithms rely on the POS of the LexicalUnits to find the Offset-POSs and the input Dict(arb,eng) does not have POS. We were unable to access a better Dict(arb,eng) for free. For LexicalEntrys without POS, our algorithms choose the best POS of the eng word. For instance, the word “book” has two POSs, viz., “verb” and “noun”, of which “noun” is more common. Hence, all translations to the word “book” in Dict(arb,eng) will have the same POS “noun”. As a result, all LexicalEntrys translating to the word “book” will be treated as a noun, leading to many wrong translations.

Based on experiments, we conclude that using the four public Wordnets, viz., PWN, FWN, JWN and WWN as intermediate resources, we are able to create good bilingual dictionaries, considering the dual objective of high quality and a large number of entries. In other words, the IW approach using the four intermediate Wordnets is our best approach. We note that if we include only translations with the highest ranks, the resulting dictionaries have accuracies even better than the input dictionaries used. We are in the processes of finding volunteers to evaluate dictionaries translating from ajz and dis to other languages. Table 5 presents the number of entries of some of dictionaries, we created using the best approach, without human evaluation.

It is difficult to compare approaches because the language involved in different papers are different, the number and quality of input resources vary and the evaluation methods are not standard. However, for the sake of completeness, we make an attempt at comparing our results with (?). The precision of the best dictionary created by (?) is 79.15%. Although our score is not in terms of percentage, we obtain the average score of all dictionaries we created using 4 Wordnets and containing 3-top greatest ranks LexicalEntrys is 3.87/5.00, with the highest score being 4.10/5.00 which means the entries are very good on average. If we look at the greatest ranks only (Top 1 ranks), the highest score is 4.69/5.00 which is almost excellent. We believe that we can apply these algorithms to create dictionaries where the source is any language, with a bilingual dictionary, to eng.

To handle ambiguities, the existing methods need at least two intermediate dictionaries translating from the source language to intermediate languages. For example, to create Dict(asm,arb), (?) and (?) need at least two dictionaries: Dict(asm,eng) and Dict(asm, French). For asm, the second dictionary simply does not exist to the best of our knowledge. The IW approach requires only one input dictionary. This is a strength of our method, in the context of resources-poor language.

Our purpose of creating dictionaries is to use them for machine learning and machine translation. Therefore, we evaluate the dictionaries we create against a well-known high quality MT: the Google Translator. We do not compare our work against the Microsoft Translator because we use it as an input resource.

We randomly pick 300 LexicalEntrys from each of our created dictionaries for language pairs supported by the Google Translator. Then, we compute the matching percentages between translations in our dictionaries and translations from the Google Translator. For example, in our created dictionary Dict(vie,spa), “người quảng cáo” in vie translates to “anunciante” in spa which is as same as the translation given by the Google Translator. As the result, we mark the LexicalEntry (“người quảng cáo”, “anunciante”) as “matching”. The matching percentages of our dictionaries Dict(arb,spa), Dict(arb,vie), Dict(arb,deu), Dict(vie,deu), Dict(vie,spa) and the Google Translator are 55.56%, 39.16%, 58.17%, 25.71%, and 35.71%, respectively.

The LexicalEntrys marked as “unmatched” do not mean our translations are incorrect. Table 6 presents some LexicalEntrys which are correct but are marked as “unmatched”.

To achieve better evaluation, we intend to use crowdsourcing for evaluation. We are in the process of creating a Website where all dictionaries we create will be available, along with a user friendly interface to give feedback on individual entries. Our goal will be to use this feedback to improve the quality of the dictionaries.