Mixing it Right: A Comparison of Sampling Strategies to Create a Benchmark for Dialectal Sentiment Classification of Google Place Reviews

Recent advancements in natural language processing (NLP) have led to significant improvements in variety of computational tasks, including sentiment analysis, question answering, and machine translation. State-of-the-art language models, particularly large language models, consistently outperform existing models and achieve high scores on widely adopted benchmarks devlin-etal-2019-bert; achiam2023gpt; dubey2024llama like SuperGLUE wang2019superglue, GSM8K cobbe2021training, and MTEB muennighoff2022mteb. However, benchmarks do not adequately capture linguistic diversity present in the world, particularly when it comes to dialectal variations. Existing work highlights this gap jurgens2017incorporating; joshi2024naturallanguageprocessingdialects as a critical limitation in NLP, indicating the need for cross-dialect generalisation. Dialectal benchmarks such as DIALECT-BENCH faisal2024dialectbench help evaluate these models in more challenging, realistic settings, where variations in vocabulary, syntax, and semantic can significantly impact performance.

A few benchmarks exist specifically for dialectal sentiment classification al2018sentiment; mdhaffar2017sentiment; oussous2020asa. However, most predominantly focus on binary label curation (positive/negative), also lacking the ability to generalize over across dialects harrat2015cross. Further, there is limited work on sentiment datasets for dialects of English ziems2023multi. Dialectal sentiment classification poses a significant challenge due to the informal, context-dependent writing style of user-generated content. For instance, posting an online review is a voluntary activity motivated by a desire to express opinions, which vary widely across different languages and dialects lo2017multilingual; saadany2021challenges. These variations go beyond vocabulary differences and show the use of distinct grammatical structures, idiomatic expressions, and culture-specific connotations eisenstein2010latent. To address these gaps, we curate dialectal data from Google Place reviews and uses self-supervision to assign multi-level labels ($1$ to $5$). Additionally, our work utilizes data sampling techniques i.e., strategically selecting reviews based on label proximity, lexical properties, and semantic content, to challenge these models.

Our work attempts to answer, “What data sampling strategies can be utilized to create an evaluation benchmark for dialectal sentiment classification that is sufficiently challenging?”.

To address this question, we curate reviews from three locales - Austalian English (en-AU), Indian English (en-IN) and British English (en-UK), and use an existing dataset of reviews from Standard American English (en-US), as shown in Figure 1. By applying various data sampling techniques, we show the curation of an effective, challenging benchmark dataset, and present insights from a comprehensive evaluation of model performance on this dialectal data.

The creation of benchmarks that reflect linguistic diversity is crucial for ensuring that language technologies are equitable and do not perpetuate biases against specific linguistic subgroups blodgett-etal-2020-language. Despite these efforts, language models often struggle with tasks involving dialects other than standard US-English, highlighting a significant gap in current evaluation practices joshi2024naturallanguageprocessingdialects; faisal2024dialectbench.

Traditional sentiment benchmarks like the Stanford Sentiment Treebank (SST-2) socher-etal-2013-recursive and the IMDB reviews dataset maas-etal-2011-learning focus on standard language forms and do not account for dialectal variations, limiting their applicability in multi-dialectal settings. Modern benchmarks such as GLUE DBLP:journals/corr/abs-1804-07461 and SuperGLUE DBLP:journals/corr/abs-1905-00537 aim for broader language understanding but still fall short in evaluating dialectal diversity.

A few datasets exist for dialectal sentiment analysis. SANA abdul-mageed-diab-2014-sana provides a multi-genre, multi-dialectal Arabic lexical dataset with over $200,000$ entries labeled for sentiment. A study on Tunisian Dialect Arabic medhaffar-etal-2017-sentiment introduces a $17,000$-comment Facebook corpus, showing improved model performance on dialectal data compared to Modern Standard Arabic. Similarly, ArSAS d3cb00d902eb44a0a0d01b794e862787 offers a $21,000$-tweet corpus over multiple dialects of Arabic annotated for speech-act recognition and sentiment, supporting research in both areas. boujou2021openaccessnlpdataset present a comprehensive dataset of over $50,000$ tweets in multiple Arabic dialects, labeled for dialect detection, topic detection, and sentiment analysis. In contrast, while there are robust resources for Arabic dialects, comparable datasets for English dialects remain limited.

Work in muhammad2023afrisentitwittersentimentanalysis advances dialectal benchmarks by providing a resource with tweet data annotated by native speakers for $14$ African languages and dialects, addressing gaps in existing benchmarked datasets. Similarly, fsih-etal-2022-benchmarking evaluate several Arabic language models against standard benchmarks for sentiment analysis, where it is shown that models trained on dialectal Arabic outperform general models, highlighting the need for benchmarks that reflect dialectal diversity.

In this section, we detail the methodology employed to investigate the impact of linguistic variety on task performance. Our approach includes the collection and pre-processing of a diverse dataset of reviews from multiple locales using the Google Places API. We describe our criteria for selecting cities across Australia, India, and the United Kingdom, and include reviews from New York City to establish a baseline for Standard American English. Finally, we elaborate on our design choices focusing on label proximity, lexical properties, and semantic content to assess their influence on the task performances.

We experiment with three encoder models namely, BERT (bert) devlin-etal-2019-bert, DistilBERT (distil) sanh2020distilbertdistilledversionbert, RoBERTa (roberta) zhuang-etal-2021-robustly. Given a variety, we perform ‘in-sample’ fine-tuning on all models for 10 epochs with early stopping, using Adam optimiser. All experiments are performed using 6 $\times$ NVIDIA V100 GPUs, with a single seed for reproducibility. The statistics of each sample in each variety are reported in Table 3. We report our task performances on three macro-averaged metrics: precision (P), recall (R), and f1-score (F).

Our evaluation of design choices addresses three questions: (a) What is the impact of label proximity on task performance?; (b) What is the impact of introducing dialectal text on task performance; (c) How to sample each subset of text to make the task more challenging for all models?

Table 4 shows the comparison of the average performance of models on reviews from each locale. The highest performance on the task is reported by models fine-tuned on en-AU (simple) with the F1-Score of 97.2. The worst performance is reported by the baseline models, with the average F1-Score of 93.5. Models report a degraded performance across reviews from all locales when the labels are changed from simple to hard, with an average decrease in the F1-score of 13.7. The idea of changing labels based on proximity has been explored earlier, and our results validate these claims. We further evaluate the impact of sampling on task performances.

This paper focused on evaluating the impact of the linguistic variety and various data-related strategies that make sentiment classification on Google Places reviews challenging for large language models. To introduce linguistic variety, we collected reviews from three locales– Australia, India, and the UK. We also utilised the existing dataset of reviews from New York for the baseline. The strategies included changing target labels and data sampling based on length and sentiment density.

Our results show that models reported higher performance on subsets with well-separated labels (simple) compared to those with closer labels (hard). This validates the idea that label proximity impacts model performance on the task, motivating the need for fine-grained analysis in sentiment classification tasks. Our evaluation of data sampling strategies showed that models trained on longer reviews (*-len-25) generally outperformed those trained on shorter ones. This suggests that longer texts provide more contextual information, enabling models to make more accurate sentiment predictions. However, models performed worse on reviews with higher sentiment density (*-sent-25), especially when trained on subsets with hard labels. This indicates that higher sentiment density can introduce ambiguity, leading to degradation in model performances on the task. Models trained on inner-circle English varieties (en-US, en-AU, en-UK) generally outperformed those trained on en-IN, particularly in more challenging tasks (hard labels and samples with long review lengths). These findings from our evaluation highlight the need for incorporating diverse linguistic varieties along with better sampling strategies while creating benchmarks to better reflect real-world complexities.