DarijaBanking: A New Resource for Overcoming Language Barriers in Banking Intent Detection for Moroccan Arabic Speakers

The field of Natural Language Processing (NLP) has gained significant traction, particularly with the emergence and advancement of Large Language Models (LLMs). This surge in interest and implementation is especially notable in industries where customer engagement and relationships are critical. In the realm of retail banking, Financial NLP and LLMs are reshaping the dynamics of client interactions, fostering enhanced personalization, responsiveness, and ultimately, customer loyalty and satisfaction (et al., 2020).

In the ecosystem of generative AI-enhanced customer service, powerful LLMs such as GPT-4 (OpenAI, 2023a) act as the brain of the system, orchestrating various components to deliver nuanced and contextually relevant interactions (et al., 2023b). Among these components, Retrieval Augmented Generation (RAG) (Lewis, 2020) stands out by augmenting the agents’ responses with information retrieved from a dense knowledge base, thereby enriching the quality and relevance of the interactions. However, the linchpin in this sophisticated setup is the Intent Classification Module, a critical aspect of Financial NLP’s Natural Language Understanding (NLU).

Intent classification, also known as Intent Detection, focuses on deciphering the semantic essence of user inputs to elicit the most appropriate response. It involves creating a linkage between a user’s request and the corresponding actions initiated by the chatbot, as outlined by Adamopoulou (2020). Typically framed as a classification challenge, this process entails associating each user utterance with one or, in certain cases, multiple intended actions. The task of intent classification presents notable difficulties. Conversations with chatbots often involve terse utterances that offer minimal contextual clues for accurate intent prediction. Furthermore, the extensive variety of potential intents necessitates a vast dataset for annotation, complicating the detection process due to the expansive label space that must be managed.

In this paper, we introduce DarijaBanking, a comprehensive Darija intent dataset and embark on a systematic comparison of diverse intent classification methodologies, encompassing full fine-tuning of both monolingual and multilingual models, zero-shot learning, retrieval-based strategies, and Large Language Model (LLM) prompting. DarijaBanking is meticulously curated from three foundational English banking datasets, namely:

• •

  Banking77 dataset (Casanueva, 2020), which provides a corpus of 13,083 banking-related queries each mapped to one of 77 distinct intents,

• •

  Banking-Faq-Bot dataset (Patel, 2017), which includes 1,764 questions distributed across 7 intent categories, and

• •

  Smart-Banking-Chatbot dataset (Lakkad, 2018), encompassing 30,100 questions sorted into 41 intents, cumulatively aggregating to a rich repository of 42,000 questions.

The initial phase involved a rigorous cleaning process aimed at intent unification across these diverse sources, addressing various challenges such as generic or colloquially dense sentences, redundancy, and contextual incompatibilities with the Moroccan banking sector. This meticulous refinement resulted in a distilled collection of 1,800 English sentences, subsequently translated into French and Modern Standard Arabic (MSA) using the OPUS MT (Tiedemann and Thottingal, 2020) and Turjuman (Nagoudi, 2022) models, respectively. Subsequently, a crucial translation step involved leveraging GPT-4 (OpenAI, 2023a) to translate these English sentences into Darija.

To ensure the highest level of accuracy and contextual relevance, particularly for the Darija translations, we employed a team of five external human native speakers as annotators. The annotators conducted a thorough manual verification and correction process, focusing on the nuanced aspects of the Moroccan dialect that automated systems might overlook. Their invaluable contributions were instrumental in refining the dataset to accurately reflect the linguistic intricacies of Darija, thereby enhancing the dataset’s utility for our intended applications. It is noteworthy that all utterances underwent manual review, with approximately 47% of them were edited to further ensure their accuracy and idiomatic appropriateness.

The resultant dataset, featuring a total of 7,200 queries across English, French, MSA, and Darija comprising of 1,800 queries for each language. The MSA and Darija subsets serve as the foundation for assessing various intent classification methods, including full fine-tuning of monolingual and multilingual models, zero-shot learning, retrieval-based approaches, and Large Language Models prompting.

The main contributions of our paper can be succinctly summarized as follows:

• •

  The DarijaBanking dataset is introduced, serving as a novel resource for banking intent detection in Darija, featuring over 7,200 multilingual queries across English, French, MSA, and Darija across 24 intent categories. This dataset results from arabizing, localizing, deduplicating, cleansing, and translating existing English banking data.

• •

  A comparative analysis of intent classification methodologies is presented, evaluating monolingual and multilingual model fine-tuning, zero-shot learning, retrieval strategies, and LLM prompting, to assess their effectiveness across the dataset.

• •

  BERTouch our Darija specific BERT-based language model, that is tailored for the intent classification task. BERTouch proved to be competitive with state-of-the-art solution including large language models such as GPT-4. As a contribution to open science, we made BERTouch publicly available in the HuggingFace platform.

• •

  Insights for enhancing Moroccan Darija banking intent detection systems are detailed, highlighting the balance between using generalist LLMs, cross-lingual transfer learning, and the importance of domain-specific data annotation. Our findings indicate the advantage of specialized classifiers with sufficient data labeling and the efficacy of retrieval-based approaches as a budget-friendly option, guiding the development of precise and economical systems.

The rest of the paper is structured to provide a holistic view of this research endeavor. Section 2 delves into the related work, establishing a contextual backdrop. Section 3 unveils the DarijaBanking corpus, detailing the processes of data arabization and localization. Section 4 explores the array of intent classification approaches, including model architectures and training paradigms. Section 5 showcases the empirical results derived from these methodologies. Section 6 is dedicated to a thorough discussion of the results obtained from our experiments, drawing critical insights and identifying the implications of our findings for the field, while also discussing the limitations of our work. Finally, we conclude our paper by summarizing the key takeaways, reflecting on the contribution of the DarijaBanking corpus to the advancement of intent detection in the Moroccan Dialect.

In the realm of NLP, the Arabic language presents unique challenges, primarily due to its rich dialectical diversity and the scarcity of domain-specific labeled datasets. This scarcity significantly impedes the advancement of Arabic NLP applications, including intent detection and conversational systems. Recent research endeavors, as documented by Darwish (2021) and Naser-Karajah (2021), highlight the acute shortage of labeled datasets for Arabic, especially for dialectal and specialized tasks. Similarly, (Fuad, 2022) and (Ahmed, 2022) underscore the resultant stagnation in the development of Arabic conversational machine learning systems, attributing this lag to the inadequate dataset resources. Furthermore, there has been a recent surge in interest in Arabic NLP, as evidenced by the shared task (Malaysha, 2024), which aims to foster further research and development in this area.

Despite these challenges, some strides have been made towards understanding and processing the Arabic language more effectively. A pioneering step in the domain of Arabic intent detection was taken by (Mezzi, 2022), who introduced an intent detection framework tailored for the mental health domain in Tunisian Arabic. Their innovative approach involved simulating psychiatric interviews using a 3D avatar as the interviewer, with the conversations transcribed from audio to text for processing. The application of a BERT encoder and binary classifiers for each mental health aspect under investigation—depression, suicide, panic disorder, social phobia, and adjustment disorder—yielded impressive results, achieving an F1 score of 0.94.

Exploring other facets of Arabic NLP, (Hijjawi, 2013) ventured into the classification of questions and statements within chatbot interactions, leveraging decision trees for this task. Their methodology was later integrated into ArabChat (Hijjawi, 2014), enhancing the system’s ability to preprocess and understand user inputs. Moreover, (Joukhadar, 2019) contributed to the field by creating a Levantine Arabic corpus, annotated with various communicative acts. Their experiments with different classifiers revealed that the Support Vector Machine (SVM), particularly when utilizing 2-gram features, was most effective, achieving an accuracy rate of 0.86.

The quest for understanding Arabic speech acts and sentiments led Elmadany (2018) to develop the ArSAS dataset, encompassing around 21K tweets labeled for speech-act recognition and sentiment analysis. The dataset, marked by its categorization into expressions, assertions, questions, and sentiments (negative, positive, neutral, mixed), provided a fertile ground for subsequent studies. Utilizing ArSAS, Algotiml (2019) employed Bidirectional Long-Short Term Memory (BiLSTM) and SVM to model these nuanced linguistic features, achieving an accuracy of 0.875 and a macro F1 score of 0.615. Lastly, Zhou (2022) demonstrated the potential of contrastive-based learning to enhance model performance on out-of-domain data, testing their methodology across several datasets, including the banking domain (Casanueva, 2020), and showed that it’s possible to improve adaptability without sacrificing the accuracy for in-domain data.

Research on intent detection has progressed beyond Arabic, with notable studies in languages like Urdu and Indonesian. Shams (2019) translated key datasets to Urdu and employed CNNs, LSTMs, and BiLSTMs, finding CNNs most effective for ATIS with a 0.924 accuracy and BiLSTMs best for AOL at 0.831 accuracy. This work was further refined to achieve a 0.9112 accuracy (Shams, 2022). Similarly, in Indonesian, Bilah (2022) utilized ATIS to inform their CNN model, achieving a 0.9584 accuracy.

Expanding the scope, Basu (2022) explored a meta-learning approach with contrastive learning on Snips and ATIS datasets for diverse domains, emphasizing the complexity of intent detection across different contexts.

Addressing the gap in Arabic intent detection, Jarrar et al. (2023) introduced ArBanking77, an arabized dataset from Banking77 (Casanueva, 2020), enhanced with MSA and Palestinian dialect queries, totaling 31,404 queries across 77 intents. A BERT-based model finetuned on this dataset achieved F1-scores of 0.9209 for MSA and 0.8995 for the Palestinian dialect.

In the exploration of NLP for Moroccan Darija, significant strides have been made across two pivotal areas: the development of novel datasets and the advancement in classification tasks. The creation of specialized datasets, as seen in the work of Essefar (2023), introduces the Offensive Moroccan Comments Dataset (OMCD), marking a significant step towards understanding and moderating offensive language in Moroccan Arabic. This initiative fills a gap in resources tailored to the nuances of Moroccan dialect. Similarly, Boujou (2021)’s contribution of a multi-topic and multi-dialect dataset from Twitter, which includes Moroccan Darija, provides a versatile tool for sentiment analysis, topic classification, and dialect identification, underlining the importance of accessible data for diverse NLP applications. On the classification front, El Mekki (2021)’s research leverages a deep Multi-Task Learning model with a BERT encoder for nuanced language identification tasks. This model’s ability to differentiate between various Arabic dialects and MSA showcases the evolving precision in language processing techniques. Furthermore, El Mekki (2022) introduces AdaSL, an unsupervised domain adaptation framework aimed at enhancing sequence labeling tasks such as Named Entity Recognition and Part-of-Speech tagging. Their approach, which capitalizes on unlabeled data and pre-trained models, addresses the challenges posed by the scarcity of labeled datasets in Dialectal Arabic, thereby advancing the field of token and sequence classification within Moroccan Darija and other dialects.

Building on these contributions, our study specifically targets the unexplored area of intent detection for Moroccan Darija within the banking domain. We introduce DarijaBanking, a pioneering dataset crafted to enhance intent detection capabilities. This dataset, developed through the arabization, localization, deduplication, thorough cleaning and translation of existing English banking datasets, comprises over 3,600 queries across 24 intent classes, in both Modern Standard Arabic and Moroccan Darija. In exploring various methods for intent detection, we highlight the success of a BERT-based model specifically finetuned for this dataset, which achieved impressive F1-scores of 0.98 for Darija and 0.96 for MSA.

The DarijaBanking Corpus¹¹1https://github.com/abderrahmanskiredj/DarijaBanking represents a novel endeavour to tailor and enrich banking-related linguistic resources specifically for the Moroccan context, leveraging the foundational structures of three significant English datasets, namely: (1) Banking77 (Casanueva, 2020), which encompasses an extensive collection of 13,083 banking-related queries, each meticulously categorized into one of 77 distinct intents; (2) the banking-faq-bot dataset (Patel, 2017), comprising 1,764 questions distributed across seven intent categories; and (3) the smart-banking-chatbot dataset (Lakkad, 2018), that includes a broad spectrum of 30,100 questions sorted into 41 intents. Collectively, these resources amalgamate into a comprehensive repository of 42,000 questions. Subsequently, we will detail the various stages involved in our corpus’s data collection and validation process.

In this section, we embark on a systematic comparison of diverse methodologies for intent detection, which can broadly be categorized into three distinct approaches: BERT-like models finetuning, retrieval-based strategies, and LLM prompting. The finetuning of BERT-like models Devlin (2019) includes both monolingual and multilingual variations, where we specifically focus on adapting monolingual Arabic models for Arabic and Moroccan Darija, and employ cross-lingual transfer techniques with multilingual models like XLM-RoBERTa (Conneau, 2020) across English, French, Arabic, and Darija datasets. Additionally, we explore zero-shot learning scenarios to evaluate model performance on languages or dialects not explicitly included in the training phase. Meanwhile, the retrieval-based strategy leverages text embedding models to index and match queries to the nearest utterance, thus inferring intent based on semantic similarity. Lastly, LLM prompting involves the utilization of advanced models such as ChatGPT-3.5 OpenAI (2022) and GPT-4 (OpenAI, 2023a), which are prompted to classify intents by providing a comprehensive list of intents and their descriptions. Each of these approaches offers a unique perspective on intent detection, highlighting the versatility and adaptability of current technologies to understand and classify user intents across a range of languages and contexts.

In this section, we present a comprehensive analysis of our experimental studies, conducted to assess the effectiveness of the three distinct approaches previously discussed. Specifically, in subsection 5.1, we delve into the nuances of fine-tuning BERT-like models. This subsection is bifurcated into an examination of both the application of zero-shot learning techniques and the comprehensive fine-tuning process. Subsequently, subsection 5.2 is dedicated to exploring the Retrieval-based Intent Detection method, wherein we benchmark the various models listed to evaluate their performance. In subsection 5.3, we shift our focus to investigating intent detection through the lens of LLM Prompting techniques, offering a fresh perspective on this approach. The culmination of our experimental journey is presented in subsection 5.4, where we discuss the outcomes of our investigations, aiming to extract meaningful insights and implications from the gathered data. To provide a robust evaluation of model efficacy, we report performance metrics on the test set, including macro F1 scores, precision, and recall, thereby ensuring a holistic assessment of each model’s capabilities.

The exploration of three different finetuning techniques for monolingual and multilingual models, in both comprehensive and zero-shot scenarios, including retrieval-based approaches and prompting with Large Language Models, for intent detection in Moroccan Darija within the banking sector, offers critical insights and implications for both future research and practical application.

First and foremost, our study demonstrates that relying solely on cross-lingual transfer learning might not be the most effective approach for addressing the complexities of Moroccan Darija. The initial results reveal a significative variance in F1 scores between MSA and Darija, underscoring the need for dedicated linguistic resources tailored to dialectical Arabic. This gap significantly narrows with the comprehensive integration of languages and fine-tuning, as shown by the enhanced performance of models like Arabertv02-twitter when trained on datasets that encompass Darija. This emphasizes the value of investing in high-quality, domain-specific data labeling to improve model accuracy and efficiency, especially in bounded domains like banking where the complexity and size of the model can be balanced with targeted, high-quality data.

Furthermore, the findings urge caution against an overreliance on powerful LLMs for specialized classification tasks such as intent detection. Despite their impressive generative capabilities, models like ChatGPT-3.5 and GPT-4 exhibit limitations in accurately classifying intents within the DarijaBanking dataset. This highlights the importance of incorporating specialized classifiers for the Intent Classification Module in banking chatbots, where precision in understanding customer queries is paramount. LLMs, while beneficial in generating human-like responses and enhancing the chatbot’s conversational capabilities, should complement rather than replace dedicated classifiers finetuned for specific intents.

However, the study also acknowledges the economic and logistical constraints of data labeling, recognizing that it can be prohibitively expensive or resource-intensive for some organizations. In such scenarios, retrieval-based approaches using pre-trained text embeddings, such as OpenAI’s "text-embedding-3-large" or the "multilingual-e5-base" model, offer a viable alternative. These models demonstrate respectable performance and efficiency, providing a practical solution for intent detection that balances cost and accuracy.

Despite these advancements, it is important to acknowledge the limitations of the study. The intent list used is not exhaustive, and real-world applications will likely require further adjustments to accommodate the full spectrum of banking queries. Moreover, the current approach does not account for the contextual and multi-intent nature of real-world conversations, which could provide valuable signals for more accurate intent classification.

In conclusion, this paper introduces DarijaBanking, an innovative dataset designed to enhance intent detection within the banking sector for both Moroccan Darija and MSA. Through the adaptation and refinement of English banking datasets, DarijaBanking emerges as a vital tool for nuanced language processing, comprising over 3,600 queries across 24 intent classes. Our analysis, spanning model fine-tuning, zero-shot learning, retrieval-based techniques, and Large Language Model prompting, highlights the critical need for tailored approaches in processing dialect-specific languages and underscores the effectiveness of ArabertV0.2 when finetuned on this dataset.

The research emphasizes the importance of domain-specific classifiers and the limitations of relying solely on general-purpose Large Language Models for precise intent detection. It also presents retrieval-based approaches as practical, cost-effective alternatives for scenarios where data labeling poses significant economic and logistical challenges. These approaches provide a pragmatic balance between performance and resource allocation, facilitating the advancement of AI-driven solutions in settings that are linguistically diverse and resource-limited.

However, we recognize the limitations of our study, including the non-exhaustive nature of our intent list and the lack of consideration for the contextual and multi-intent dynamics of real-world interactions. These aspects offer avenues for future research to further refine and enhance the application of AI in understanding and servicing the diverse needs of banking customers.

By contributing to the development of resources like DarijaBanking, this paper aims to support the broader goal of making AI technologies more adaptable and effective across various linguistic contexts. In doing so, we hope to inspire continued efforts towards creating more inclusive digital banking solutions and advancing the field of NLP.