Do Large Language Models Speak All Languages Equally?
A Comparative Study in Low-Resource Settings

Recent advances in large language models (LLMs) developed significant interest in natural language processing (NLP) across academia and industry. LLMs are known for their language generation capabilities that are trained on billions or trillions of tokens with billions of trainable parameters. Recently researchers have been evaluating LLMs for various NLP downstream tasks, especially question answering Akter et al. (2023); Tan et al. (2023); Zhuang et al. (2023), reasoning Suzgun et al. (2022); Miao et al. (2023), mathematics Lu et al. (2023); Rane (2023), machine translation Xu et al. (2023); Lyu et al. (2023), etc.

Most of the existing works on the evaluation of LLMs are on resource-rich languages such as English. However, the capabilities and performances of LLMs for low-resource languages¹¹1Refers to the scarcity of datasets and other resources rather than limitations in LLM capabilities. for many NLP downstream tasks are not widely evaluated, leaving a notable gap in the linguistic capabilities of low-resource languages. The most widely spoken yet low-resource languages of South Asia²²2https://simple.wikipedia.org/wiki/Languages_of_South_Asia such as Bangla, Hindi, and Urdu, several researchers are handling the scarcity of datasets and other resources in NLI Aggarwal et al. (2022), Sentiment analysis Hasan et al. (2023b); Sun et al. (2023); Koto et al. (2024) and Hate speech detection Khan et al. (2021); Santosh and Aravind (2019). However, the amount of work that uses LLMs is still very few, mainly due to a few constraints such as dataset scarcity, computational costs, and research gaps associated with low-resource languages. These constraints of low-resource languages require more attention, alongside a focus on high-resource languages, to enhance the applicability of LLMs to general-purpose NLP applications.

To fill the aforementioned gap, we comprehensively analyze zero-shot learning using various LLMs in English and low-resource languages. The performance of LLMs shows that GPT-4 provides comparatively better results than Llama 2 and Gemini. Moreover, the English language performs better on different tasks than low-resource languages such as Bangla, Hindi, and Urdu. The Key contributions are as follows:

• •

  To address the limitation of publicly available datasets for low-resource languages, we present datasets for sentiment and hate speech tasks by translating from English to Bangla, Hindi, and Urdu, thereby facilitating research in low-resource language processing.

• •

  We investigate and analyze the effectiveness of different LLMs across various tasks for both English and low-resource languages such as Bangla, Hindi, and Urdu, which suggest that LLMs perform better when evaluated in English.

• •

  We apply zero-shot prompting using natural language instructions, which describe the task and expected output, enabling constructing a context to generate more appropriate output.

LLMs are proficient in various NLP tasks and highly generalizable across multiple domains. However, their performance remains significant room for improvement, particularly in low-resource languages such as Bangla, Hindi, and Urdu. Previous study Robinson et al. (2023) demonstrates the inability of LLMs such as GPT-4 to perform on low-resource (African) and high-resource languages. However, LLMs perform well in languages (European) that use the same script as English Holmström et al. (2023).

NLP research works, and applications for several downstream tasks mainly focus on high-resource languages. Unlike the English language, the advancement of NLP tasks for low-resource languages made it challenging due to several factors described by Alam et al. (2021). However, there have been some improvements in the last couple of years for Bangla sentiment analysis focusing on resource development Hasan et al. (2020); Islam et al. (2021); Hasan et al. (2023a) that attained attention from many researchers to concentrate on solving this issue. Some of the recent works on NLI Pahwa and Pahwa (2023); Gubelmann et al. (2023), Sentiment Analysis Xing (2024); Zhang et al. (2023b, a), and Hate Speech Detection Hee et al. (2024); García-Díaz et al. (2023) that utilize LLM are mainly carried out in English languages. Moreover, these works opened up the prospects of exploring LLMs for downstream tasks of low-resource languages.

There are few attempts from researchers across different languages to utilize LLM for low-resource languages Hasan et al. (2023b); Kabir et al. (2023); Koto et al. (2024); Kumar and Albuquerque (2021) that show LLMs can achieve similar results to traditional machine learning techniques and transformer-based models. However, existing multilingual benchmarks such as BUFFET Asai et al. (2023), XTREME Hu et al. (2020), XTREME-R Ruder et al. (2021), MEGA Ahuja et al. (2023a), and MEGAVERSE Ahuja et al. (2023b) do not address all four South Asian low-resource languages we are considering in our study. Moreover, BUFFET is limited to binary classification tasks and uses few-shot learning and instruction fine-tuning of smaller LLMs (such as mT5, mT0) and ChatGPT. At the same time, we focus on multi-class classification and use zero-shot learning with SOTA LLMs. The performance of LLMs is not balanced for all languages Huang et al. (2023); Qin et al. (2023), and our study uniquely focuses on comparing resource-rich (English) and low-resource (Bangla, Hindi, and Urdu) languages using SOTA LLMs.

Previous studies have highlighted LLM limitations in low-resource languages, particularly in binary classification, with minimal focus on South Asian languages. These constraints include dataset scarcity, high computational costs, and specific research gaps. To address these challenges, we concentrate on South Asian languages like Bangla, Urdu, and Hindi. We provide datasets for sentiment and hate speech tasks by translating from English. We explore zero-shot learning techniques across English and South Asian languages, thus expanding LLM applications in low-resource settings.

We focused on both open- and closed-source LLMs. We choose three LLMs that are GPT-4 OpenAI (2023), Llama 2 Touvron et al. (2023), and Gemini Pro Team et al. (2023). We select the LLMs based on their performances, parameter sizes, and capabilities. To conduct our experiments, we used the XNLI dataset Conneau et al. (2018) for the NLI task, the official test of SemEval-2017 task 4 Rosenthal et al. (2017) for the sentiment task, and the dataset described in Davidson et al. (2017) for hate speech task. We provide the details of the dataset used and the detailed data preprocessing and evaluation metrics in Appendix B.

English vs Low-resource Languages: Our experiments show that all the LLMs consistently provide superior performances for English languages in all tasks except the performances of Gemini in the sentiment task ( Table 1). In the NLI task, the performance of GPT-4 in English is $18.04$%, $17.38$%, and $22.81$% better than the Bangla, Hindi, and Urdu languages respectively (see Table 1). Although Hindi performs better than Bangla and Urdu, there is still a massive performance gap compared to English. Besides, Llama 2 performance in English is $32.52$%, $31.28$%, and $29.94$% higher compared with Bangla, Hindi, and Urdu respectively. The difference between English and other languages is $\sim$$70$% from their original performance. Although the performance differences of Gemini between English and other languages are comparatively lower than GPT-4 and Llama 2, English is accomplishing approximately $13$% better on average than Bangla, Hindi, and Urdu.

For the sentiment task, English is performing nearly on average $13\%$ better than other languages using GPT-4 (see Table 1). The performance difference of Llama 2 between English and other languages is $\sim 11\%$ on average, and English is consistently doing better than other languages. Despite that, Bangla, Hindi, and Urdu are performing $0.49\%$, $0.89\%$, and $0.60\%$ better than English. The performance of Gemini remains almost the same for all the languages in the sentiment task. Our hate speech task experiments reveal that the performance of GPT-4 in English is approximately, on average, $22\%$ better than low-resource languages (see Table 1). Moreover, the performances in English are $\sim 17\%$ and $\sim 18\%$ better than low-resource languages for Llama 2 and Gemini models.

We postulate the low performance of LLMs in low-resource languages for the following reasons. One of the main reasons is that most of the LLMs are trained on a large amount ($90\%$) of English data, whereas the amount of training data for low-resource languages is small compared with English. Moreover, cultural differences between English-spoken countries and low-resource language countries affect the sentiment and hate speech tasks the most. Lastly, the quality of the translation affects the performance of low-resource languages. However, Hindi performed better than Bangla and Urdu in all tasks among the low-resource languages. The performance difference among the low-resource languages is insignificant across the tasks and LLMs. Our findings from this section conclude that improving LLMs is required for low-resource languages.

Gemini is the only LLM that predicted all the samples of each task. Although we provide a safety setting for the Gemini model, it blocked some data due to the content containing derogatory language. We noticed that the samples from sentiment and hate speech tasks were blocked for containing derogatory language, and those from the NLI task were not blocked. We provide a brief overview of the number of samples that are blocked by Gemini in Figure 2. However, the Urdu language is not supported by the Gemini. Despite that, the Gemini performs strongly in Urdu for the NLI and sentiment tasks. We further investigated the performances of Gemini in the Urdu language. We found that the alphabets of Urdu are derived from the Arabic language family³³3https://en.wikipedia.org/wiki/Urdu_alphabet and many words are adopted from the Arabic language. Arabic is supported by Gemini, and the training data of Arabic shares semantic information with the Urdu language, which is why Gemini exhibits a strong performance in the Urdu language.

In general, GPT-4 shows prominent performances over other LLMs across all the tasks. Although Llama 2 provides better results for hate speech tasks, it struggled to perform well in NLI and sentiment tasks. While Gemini demonstrated strong performances in NLI and sentiment tasks, it delivered worse in hate speech tasks. Despite observing a smaller performance gap in Gemini, significant disparities persist in GPT-4 and Llama-2, indicating that direct translation is less likely to compromise sentiment information. See Appendix B for class-wise experimental results.

In this study, we introduce datasets for sentiment and hate speech tasks by translating from English to Bangla, Hindi, and Urdu to facilitate research in low-resource language processing. Through a comprehensive examination of zero-shot learning across multiple LLMs, notably GPT-4, we uncover performance disparities between English and low-resource languages. Furthermore, our analysis identifies NLI as a task where GPT-4 consistently demonstrates superior capabilities, underscoring avenues for enhancing LLM applicability in general-purpose NLP applications.

In our study, we refrained from utilizing explicit prompting techniques to enhance the performance of large language models (LLMs). Our evaluation primarily focused on assessing LLMs in the context of English and low-resource languages such as Bangla, Hindi, and Urdu, without exploring variations in prompts. Regarding the quality of dataset translations, it is important to note that the translations generated by Google Translator were not subjected to human verification. Consequently, while certain translation errors were overlooked during our analysis, we conducted sampling from each translated dataset to gain insights into the overall translation quality. Our findings underscore the necessity for further refinement in translation methodologies to elevate both the quality and accuracy of translations in future research endeavors.