Maknuune: A Large Open Palestinian Arabic Lexicon

Like all other DA, and unlike MSA, PAL has no standard orthography rules Jarrar et al. (2016); Habash et al. (2018). In practice, PAL is primarily written in Arabic script, and to a lesser extent in Arabizi style romanization Darwish (2014). Some of the variations in the written form reflect the words’ phonology, morphology, and/or etymological connections to MSA. Orthogonal and detrimental to the orthography challenge, PAL has a high degree of variability within it sub-dialects in phonological terms. We highlight some below, noting that some also exist in other DA.

Like other DA, PAL has a complex morphology employing templatic and concatenative morphemes, and including a rich set of morphological features: gender, number, person, state, aspect, in addition to numerous clitics. We highlight some specific morphological phenomena that we needed to handle.

Previous research on Arabic dialects reveals that the syntactic differences between these dialects are considered to be minor compared to the morphological ones (Brustad, 2000). One particular challenging phenomenon we encountered is a class of nouns used in adjectival constructions, but violating noun-adjective agreement rules, which involve gender, number and rationality Alkuhlani and Habash (2011). For instance, the word \<خِيخَة> xiyxa$\hbar$ /kh ii kh a/ ‘weak/lame’ does not typically agree with the nouns it modifies unlike a normal adjective such \<كْبِير> k.biyr /k b ii r/ ‘old [human]/large [nonhuman]’. So, the words \<سِيَّارَة> siy$\sim$aAra$\hbar$ ‘car [f.s.]’, \<عُرُس> $\varsigma$urus ‘wedding [m.s.]’, and \<نَاس> naAs ‘people [m.p]’ can all be modified by \<خِيخَة> xiyxa$\hbar$; however, they need three different forms of \<كْبِير> k.biyr: \<كْبِيرِة> k.biyri$\hbar$, \<كْبِير> k.biyr, and \<كْبَار> k.baAr, respectively. We mark the POS of such nominals as ADJ/NOUN in our lexicon, as it is a class that deserves further study.

PAL has a rich culture of figures of speech and multiword expressions (compounds, collocations, etc.) that has not been well documented. We highlight some phenomena that we cover in Maknuune.

The current work spans over five years of effort, and a large number of volunteering informants, linguistics students, and citizen linguists (over 130 people). The data was collected from many different sources. First are interviews with (mostly but not entirely) elderly people who live in rural areas such as villages and towns or in refugee camps in the West Bank. The researchers went to the field and met with several people. They attended several social gatherings and participated in different events, e.g. weddings, funerals, field harvests, traditional cooking sessions, sewing, etc. They asked the language users several questions pertaining to the following themes: weddings, funerals, occupations, illnesses, cooking traditional dishes, plants, animals, myths, games, weather terms, tools and utensils, etc. They were particularly interested in documenting terms and expressions that are used mainly by the old generation. Secondly, to achieve the needed balance in the lexicon, the researchers consulted an in-house balanced corpus, that contains $\sim$40,000 words. The corpus comprises data that was transcribed from several recorded conversations that revolve around the same themes as above, written chats and texts, and some internet material (both written and spoken). Common words including verbs, adjectives, adverbs, and function words (e.g., prepositions, conjunctions, particles) were taken from the balanced corpus. At a later stage in the development of Maknuune, we consulted with the Curras Corpus Jarrar et al. (2016) to identify additional missing lemmas, with limited yield. We compare to Curras in terms of coverage in Section 5. All of the above was also supplemented by methodical rounds of well-formedness checking to improve consistency across all fields, i.e., diacritization, transcription, root validity, etc. Finally, in addition to the previous two methods, the researchers employed their linguistic intuition skills, knowledge of Palestinian Arabic (as native speakers) and the knowledge of the language users to provide additional word classes and multiword expressions that are associated with the existing lemmas. It should be noted that whether an MSA lemma cognate of a PAL lemma (with similar or exact pronunciation, or meaning) exists was not considered a factor in including the PAL lemma in the lexicon. We focused on creating a representative sample of PAL including all its sub-dialects.

Each entry in the Maknuune lexicon consists of six required and three optional fields. The six required fields are the Root, Lemma, Form, Transcription, POS & Features, and English Gloss. The optional fields are the MSA Gloss, Example and Notes. Figure 1 presents an example of a number of entries coming from the same root.

In the interest of estimating how well our lexicon would fare with real-world data, we perform an analysis between the Curras and Maknuune lemmas, to see how many of the Curras lemmas Maknuune actually covers. From an initial investigation, we note that there are numerous minor differences that need to be normalized to ensure a more meaningful evaluation. As such, we first pre-process all lemmas (in both lexicons) by stripping the \<سكون> sukun diacritic, stripping all the \<فتحة> diacritics that appear before a \<ا> A, converting the \<همزة وصل> \<ٱ> Ä to \<ا> A, and stripping the \<كسرة> (i) and \<فتحة> (a) diacritics if they appear before \<ة> $\hbar$. We then compare all the annotated lemma:POSType in Curras (56,004 tokens and 8,315 normalized types) to the lemmas in Maknuune. We exclude 12,673 (23%) of the tokens pertaining to punctuation, digits and proper noun POS, none of which were especially targeted by Maknuune. Of the remaining 43,331 entries, 49% have exact match in Maknuune. We sample 10% of the unique entries with no exact match (433 types and 1,965 tokens), and manually annotate them for their mismatch class. We found that 74% of all the sampled types (80% in tokens) are actually present in Maknuune, but with slight differences in orthography mainly in the presence or absence of diacritics but also some spelling conventions. For about 20% of sampled types (17% in tokens), the lemma type is not one that we targeted such as foreign words and proper nouns that are differently labeled in Curras, or MSA words. Finally, 6% of sampled types (3% in tokens) are entries that are admittedly missing in Maknuune and can be added. This suggests that we have very good coverage although the annotation errors and differences make it less obvious to see. A simple projected estimate assuming that our 10% sample is representative would suggest that Maknuune’s coverage of Curras’ lexical terms (other than proper nouns and punctuation) is close to 94% (97% in token space); however a full detailed classification would be needed to confirm this projection.

In this section we conduct an evaluation similar to the one carried out in Section 5.2 but with an MSA lexicon (Calima_MSA), and an Egyptian Arabic lexicon (Calima_EGY).¹⁰¹⁰10For MSA, we compared with the calima-msa-s31_0.4.2.utf8.db version Taji et al. (2018) based on SAMA Graff et al. (2009) and for EGY we only compared to the calima-egy-c044_0.2.0.utf8.db based on Habash et al. (2012). For EGY, only CALIMA analyses entries are selected. The analysis reveals that 44% of Maknuune overlaps with Calima_MSA at the lemma:POSType level (63% if all entries are dediacritized),¹¹¹¹11The shadda ($\sim$) is not included in dediacritization. and that 49% of Maknuune overlaps similarly with Calima_EGY (75% dediacritized). Taking into account that Maknuune spelling follows the CODA* guidelines, the analysis suggests that the 37% of Maknuune lemma:POSTypes, which do not exist in the MSA lexicon we used, are heavily dialectal. The overlap with EGY is predictably higher, and the 25% of Maknuune lemma:POSTypes (dediacritized) not existing in EGY highlights the differences between the two dialects despite their many similarities.

The quantitative analyses we presented above allow us to see the big picture in terms of lexical richness and diversity in Maknuune and its complementarity to existing resources. However, we acknowledge that such an approach misses a lot of details that are collapsed or lost when ignoring subtle differences in semantics, phonology and morphology. We first point at homonyms showing semantic changes and spread, such as \<آوَى> /2 aa w a/ which is ‘thread a needle’ in PAL and ‘shelter sb’ in both MSA and PAL, \<بَطّ> /b a t. t./ which means ‘very small olives that people find hard to pick’ in some villages in Palestine and ‘ducks’ in both MSA and PAL, and \<آخرة> /2 aa kh r e/ which means ‘desserts’ in Nablus and ‘the Day of the Judgment’ in both MSA and PAL, albeit with a different pronunciation. Clearly, additional entries are needed to mark these difference. Furthermore, the majority of the entries in Maknuune are actually pronounced differently from MSA even if spelled the same without diacritics and thus warrant entries of their own, with clear phonological specifications. Finally, if we consider morphology (which is not modeled here per se), many PAL lemmas that have MSA lemma cognates are actually inflected differently, e.g., \<مَدّ> mad$\sim$ ‘extend;stretch’ (in PAL and MSA), has different inflections for some parts of the paradigm: the 2nd person masculine plural is \<مَدَّيتوا> mad$\sim$aytuwA in PAL and \<مَدَدْتُم> madad.tum in MSA. Hence, each lemma in our lexicon heads a morphological paradigm which differs from its MSA counterpart.