Milimili. Collecting Parallel Data via Crowdsourcing

The translation task was straightforward where we displayed a source sentence and requested a translation into the target language. The instruction was as simple as ”Translate the sentence from SRCLANG to TGTLANG”.

Only workers who had indicated in their settings that they were proficient in both SRCLANG and TGTLANG were allowed to undertake this task.

In total, we had four translation tasks: from Chechen to Russian, from Russian to Chechen, from Fula to English, and from English to Fula.

Translated sentences were sent to the quality
control task where we asked workers to evaluate whether the translation of the source sentence was good or bad. To ensure the accuracy of the evaluation, we gave the same translation to three workers. The majority vote was selected.

For the quality control task, we had created an additional exam, so performers had to prove their language proficiency. However, the exam was fairly easy. We had ten sentences. Five sentence pairs were correct translations, mined from various parallel resources. The other five sentence pairs were incorrect translations. Two were translations of different sentences, one was a translation into a different language, and two were word-for-word translations collected from an online dictionary, specifically Glosbe³³3https://glosbe.com.

In total, we had the same four tasks for different language directions.

Certain bad translations could be detected automatically, so we didn’t need to send all translated sentences for human verification.

To verify if the translation was in the appropriate language, we used a language detector, specifically Fasttext [Joulin et al., 2016].

We also conducted a length relation check between the source and translated sentences. If the difference exceeded three times, we rejected the translation.

The entire pipeline was prepared so that it could run automatically. However, we carried out manual verification from time to time to check the quality.

We utilized basic crowdsourcing tools to detect poor quality, such as excessively fast responses.

Generally, our method is not recommended for language pairs that are already represented in machine translation systems, as people began using machine translation to complete the task.

Furthermore, we want to highlight that the quality of the translated sentences cannot be compared with professional translation. Therefore, we recommend using crowd-translated tasks only in training sets, excluding test sets.

Addressing the problem of low resource languages can entail significant costs. Let’s consider a scenario where there are approximately 7,000 languages, each needing a million parallel texts. This totals 7 billion sentences. If we estimate the translation of one sentence at around one dollar, the total expense would rise to 7 billion dollars. Clearly, that is a quite substantial amount.

We paid $0.02 for each translation and $0.01 for every set of 10 verifications.

In fact, the total expenditure for all the experiments discussed in this context amounted to approximately $100.

We had taken all the source sentences in Chechen from Wikipedia⁴⁴4https://ce.wikipedia.org. From the raw data, we had discarded the template sentences for frequently repeated words. Additionally, we had filtered sentences in other languages using Fasttext [Joulin et al., 2016].

For the Russian language, we used the data from the WMT21 test set [Akhbardeh et al., 2021], which was presented at the News Task.

All the source sentences in Fula were taken from Wikipedia⁵⁵5https://ff.wikipedia.org, in the same way as the Chechen language. It’s worth noting that even monolingual data for Fula was quite difficult to find on the Internet. One more important point about Fula is that it has several dialects which are significantly different from each other. We attempted to work with Nigerian Fulfulde but didn’t impose strict restrictions.

For English, we also used the WMT21 News Task test set [Akhbardeh et al., 2021].

Table 1 shows the main results of the experiment. The first row is about how many sentences were initially translated. The second row shows how many of them were verified. In the case of Fula, the quantity is less than the number of translated sentences since we don’t have enough users who passed our exam. The final row tells us how many sentences were marked as good and hence were included in the corpus.

All the data is available on GitHub.

Much to our chagrin, we must acknowledge that scaling to other languages is exceedingly difficult due to the scarcity of users who are bilingual and willing to contribute to a crowdsourcing platform. Although we can confidently declare that the proposed method is able to gather sufficient data for training in the Chechen and Russian languages, the overall experiment cannot be deemed a success. This is because we encountered a shortage of contributors for the Fula and English pair. Other language pairs also faced similar challenges.

We believe that the future expansion of crowdsourcing platforms could potentially help ameliorate this issue.

At the moment, we do not have any proof that the quality of the corpus is good enough to be used in machine translation model training. However, we believe that it is at least as good as automatically mined parallel corpora.