Creating Arabic LLM Prompts at Scale

We created natural language prompts for 78 publicly available Arabic datasets using the PromptSource sourcing tool. The PromptSource UI facilitates a user-friendly interface for the creation of prompt templates, providing the ability to specify the insertion points of inputs within the prompts and to provide potential answers. For each task we developed 2-8 different prompt templates. The time required for creating the prompt templates was typically 30-90 minutes. Figure 1 shows 4 different prompt templates that we developed for one of the datasets¹¹1https://huggingface.co/datasets/Abdelkareem/arabic_tweets_classification. We were keen to diversify the order and language of the prompts. From 78 datasets, we generated 67,488,303 prompts.

We developed a comprehensive data processing pipeline designed to perform the translation, evaluation, and filtering of English datasets into Arabic. The pipeline consists of a series of well-defined steps, summarized as follows:

• •

  Sentence Extraction: Initially, the pipeline splits the English prompts into sentences. We used the Python sentence-splitter library²²2https://github.com/mediacloud/sentence-splitter, which is a re-implementation of the Moses sentence splitter³³3https://www.statmt.org/europarl/. These sentences serve as the source text for translation.

• •

  Machine Translation: Subsequently, we translated English sentences to Arabic using the opus-mt-ar-en Helsinki Machine Translation (MT) model Tiedemann and Thottingal (2020), which is a neural model based on Marian NMT⁴⁴4https://marian-nmt.github.io. We opted to use this model as opposed to commercial translation APIs to lower the overall cost.

• •

  Translation Evaluation: In the pursuit of ensuring translation quality, the pipeline incorporates the COMET-QE reference-free MT evaluation model Rei et al. (2023), which has been shown to correlate well with human judgments. COMET-QE calculates a quality score on a scale from 0 to 1, with 0 indicating the lowest translation quality and 1 signifying the highest quality.

• •

  Threshold-Based Filtering: To enhance the reliability of the translated content, a specified threshold, namely 0.7, is applied. A translated prompt where one or more of its sentences do not meet this quality criterion are filtered out, ensuring that only high-quality translated prompts are retained for further utilization.

• •

  Manual Translation Verification: We manually correct the translation of the instructions for datasets where the instructions were separable from the input.

• •

  Quality Assurance: We manually check a random sample from each dataset to determine the efficacy of translation. Performing this manual check helped us identify (and subsequently drop) datasets that are not amenable for translation such as those that require the identification of English specific grammatical phenomena. It also helped us in identifying systematic translation errors that can be easily handled. For example, some datasets involved the manipulation of a list of numbers, and thus aside from the instructions, the input and output did not require translation. Another example involves the incorrect translation of word “passage”, which was translated to mean a “walk way” instead of a “piece of text”.

Our target was to translate both PromptSource and Super-NaturalInstructions, which are composed of 57 and 1,429 datasets respectively. Prior to translation, we excluded datasets that were English specific such as grammar checking and translation tasks such translation between Urdu and Marathi. In doing so, we excluded 14 and 490 tasks from PromptSource and Super-NaturalInstructions respectively. After running our pipeline on both datasets, we excluded tasks that had less than 10 translated prompts after filtering. While none of the tasks for PromptSource were excluded, an additional 619 tasks were excluded for Super-NaturalInstructions.

From all the prompts that we created, we extracted a test set composed of a random sample from each dataset with a maximum of 50 samples from each dataset. The final size of the test set was 13,462 prompts with ground truth responses that was extracted from 643 datasets that cover 67 different tasks. For fine tuning, we constructed two training datasets composed of 800 thousand prompts and 8 million prompts, which were randomly extracted from all the datasets (and were not in the test set). We opted to create two training datasets to measure the impact of using more (or less) examples for fine tuning. From the training dataset, we randomly picked 1% of the prompts for validation.

As baselines, we prompted a variety of LLMs using our test set. The models were:

• •

  AceGPT-Instruct 7B Huang et al. (2023): This is one of the first publicly available LLMs that focus on Arabic. The model is based on Llama2, where the model was further pre-trained using Arabic text and instruction tuned using Arabic instruction datasets.

• •

  Jais-chat 13B Sengupta et al. (2023): This is the first open Arabic/English LLM that was trained from scratch based on the GPT-3 decoder architecture.

• •

  Llama3-Instruct 8B & 70B Dubey et al. (2024): These are large foundational models from Meta. Though they are primarily pre-trained on English text, the models have demonstrated reasonable instruction following abilities for other languages.

• •

  Qwen2-Instruct 7B Yang et al. (2024): These are large foundational models that are multilingual by design and have achieved competitive results on standard benchmarks. We later fine tune the base Qwen2 7B model using our instruction data.

We used ROUGE-L as our quality metric. Table 1 shows the baseline results across all the aforementioned models for a variety of tasks. The results show that Llama3 70B achieved the best results followed by Llama3 8B, and AceGPT 7B yielded the worst results.

We fine tuned the base Qwen2 7 billion parameter model using Low-Rank Adaptation (LoRA) Hu et al. (2021). This is a parameter-efficient method that freezes the parameters in the existing layers of a model and introduces trainable rank decomposition matrices for each layer in the model, reducing the parameters that require tuning by orders of magnitude, leading to significantly smaller memory requirements while achieving results that are at par with full fine tuning. We used the peft library from HuggingFace⁵⁵5https://huggingface.co/docs/peft/en/index.

We fine tuned Qwen2 7B using two different sets of instructions that we randomly sampled from our newly created instruction datasets. The size of the first sample was 800K, and the second was 8M. We chose two different sample sizes to quantify the effect of using increased training data. We used the following Arabic system prompt and prompt structure for all our training instruction:

فيما يلي تعليمات باللغة العربية تصف المهمة.> \< اكتب ردًا باللغة العربية يكمل الطلب بشكل مناسب.>

(Translation: “The following are Arabic instructions that describe a task. Provide an answer in Arabic that satisfies the request

### Instruction:

{text of the instruction}

### Response:

{text of the response}

We fine tuned the model for one epoch and the key tuning hyper parameters were as follows:

• •

  Warmup Ratio: We used a warm-up ratio of 0.1, meaning that the learning rate gradually increased from a low value to its initial value over the first 10% of the training steps. This helps stabilize the training process.

• •

  Learning Rate: The learning rate was set to 2e-4 and adjusted using a linear scheduler, which decreases the learning rate steadily during training. This helps the model converge smoothly.

• •

  Optimizer: We used the Adam optimizer, which is effective in handling sparse gradients and changes in the training data. This choice, combined with our learning rate settings, ensures stable and effective updates to the model’s parameters.

• •

  Weight Decay: We also used a weight decay of 0.01 to prevent overfitting and to improve the generalization of the model.

As shown in Table 1, fine tuning Qwen2 7B model using our carefully curated dataset of 800k samples led to statistically significant improvement over the Qwen2-Instruct 7B with respective average ROUGE-L scores of 0.184 and 0.143. We used a paired 2-tailed t-test over per dataset scores with a p-value less than 0.01 to determine statistical significance. In turn, fine tuning using 8M samples statistically significantly outperformed all baseline models, except for Llama 3-70B (ROUGE-L = 0.221; p-value > 0.79), and Qwen2 7B fine tuned on 800k examples with an average ROUGE-L score of 0.224. It performed better than the Arabic-focused models AceGPT 7B and Jais 13B, which scored average ROUGE-L scores of 0.102 and 0.138, respectively. This demonstrates the effectiveness of fine tuning using our dataset with significant improvement across a variety of tasks, where the fine tuned Qwen2 7B 800k and 8M models respectively performed 29% and 57% higher than the Qwen2 7B-Instruct model. We also observe that fine tuning using more examples (8M versus 800k) leads to 22% improvement in results. This suggests that fine tuning using more examples positively impacts the final model’s performance.