Persian-WSD-Corpus: A Sense Annotated Corpus for Persian All-words Word Sense Disambiguation

Word Sense Disambiguation (WSD) is an open problem in Natural Language Processing (NLP) which aims to automatically recognize the correct meaning of ambiguous words in a particular context. For instance, consider the sentence ”$The\ bank\ will\ lend\ us\ money.$”, where we want to disambiguate the word $bank$. Retrieving all possible meanings of $bank$ from a pre-defined sense inventory (WordNet, for instance), a WSD algorithm should be ideally able to associate the word $bank$ with its $financial\ institute$ meaning.
WSD has applications in other NLP tasks such as Machine Translation (Carpuat and Wu, 2007), Information Retrieval and Extraction (Zhong and Ng, 2012), Question Answering (Ramakrishnan et al., 2003), etc. WSD tasks can be distinguished into two generic categories: (1) Lexical Sample WSD and (2) All-words WSD. Developed Lexical Sample WSD systems aim to disambiguate a set of restricted predefined words. Whereas the goal of developing All-words WSD systems is disambiguating all occurring words in a particular context. Generally, All-words WSD approaches are useful for downstream NLP applications (Saeed et al., 2019a). Compared to the Lexical Sample approaches, developing such systems seems to be more challenging. This is mainly because the developed All-words WSD systems should ideally be able to cover a wide range of open-class words in the language of interest. Whereas, the Lexical Sample systems only require disambiguating a limited number of words. In this paper, we focus on All-Words WSD for the Persian language.
WSD approaches can be grouped into two main approaches: (1): knowledge-based and (2): supervised. Knowledge-based WSD approaches exploit information from a lexical resource such as machine-readable dictionaries, thesauri, and ontology to perform WSD. On the other hand, supervised systems apply machine learning techniques on a sense-annotated corpus to train WSD models. Thanks to the training phase, supervised systems generally outperform knowledge-based alternatives. It worth noting that, due to the unavailability of sense-annotated corpora for many languages, performing supervised WSD is not possible. On the other hand, knowledge-based approaches only require lexical resources that are available for a wide range of languages and can be used as an appropriate alternative. To the best of our knowledge, the only available sense-annotated corpus for the Persian language is Persian SemCor (Rouhizadeh et al., 2021), which have been developed automatically. Previous studies on All-words WSD have focused on a variety of languages such as English, Dutch, Italian, etc (Oele and Van Noord, 2017; Popov et al., 2019; Raganato et al., 2017a). However, many low-resource languages such as Persian have not been studied as well. In this paper, we introduce and discuss the creation pipeline of SBU-WSD-Corpus as the first developed test set for the Persian All-words WSD.
Persian (also known as Farsi) is an Indo-European (IE) language that is currently spoken by more than $110$ million people in several countries such as Iran, Afghanistan, and Tajikistan. Persian language uses a modified Arabic script and is written from right to left. Millions of Persian texts are available via online web pages, newspapers, books, etc. As a result, there is no doubt in the necessity of developing computational models for Persian as a low-resource language(Shamsfard, 2011). Similar to other fields of study, standard test sets are required for evaluating WSD approaches. However, none is available for Persian. The main objective of this research is to address the lack of an All-words WSD test set for the Persian language.
SBU-WSD-Corpus contains $5892$ content words of Persian running text. The corpus includes $3371$ instances ($2073$ nouns, $566$ verbs, $610$ adjectives, and $122$ adverbs) which are manually annotated by three annotators. We benchmark SBU-WSD-Corpus with several supervised and knowledge-based WSD models, providing baseline results for future research on All-Words WSD for the Persian language.
The main contributions of this research are as follows :

1. 1.

   Creating a standard All-words WSD data set
   With the goal of developing an standard All-Words WSD data set, we followed all guidelines, suggested by SensEval-2 (Edmonds and Cotton, 2001). To the best of our knowledge, this is the first available test set for Persian All-words WSD task. With the introduction of SBU-WSD-Corpus, we hope to open avenues for future WSD research in Persian. Additionally, we provide details of our corpus creation pipeline, which can be useful for researchers of other low resource languages to develop similar useful resources.

2. 2.

   Presenting benchmarks for future research in Persian All-Words WSD
   To provide baseline for evaluation of Persian All-Words WSD systems, a set of best performing supervised (trained on Persain SemCor) and knowledge-based WSD systems are carried out on SBU-WSD-Corpus. In addition, detailed analysis and comparison between different systems are provided.

3. 3.

   Usefulness of SBU-WSD-Corpus for evaluation of other Persian NLP tasks
   The whole documents of SBU-WSD-Corpus has been manually tokenized, Pos-tagged and lemmatized by an expert linguist. As a result, it can be also used as a test set for evaluating a range of basic Persian preprocessing tools such as PoS-taggers, lemmatizers, tokenizers and sentence segmentations, etc.

4. 4.

   Free access to the developed data set
   To encourage future research on Persian All-words WSD, SBU-WSD-Corpus will be freely available for the research community.

The rest of the paper is structured as follows. Section $2$ surveys a range of related works. Section $3$ describes the different steps of creating the corpus. Section $4$ introduces the WSD experiments applied to the corpus. In section $5$ the results and the analysis about the performance of the evaluated benchmarks are presented. Finally, the conclusions and further possible works are found in section $6$.

Over recent decades, a variety of sense annotated corpora have been developed for both All-words and Lexical-Sample WSD tasks. Generally, sense annotated corpora can be divided into two main groups: (a)WSD Training corpora and (b)WSD Test Set corpora.

• •

  WSD Training corpora, which includes a variety of sense annotated samples in the language of interest. Sense annotated corpora for lexical sample task only include annotated samples for the limited number of predefined words. However, All-words sense annotated corpora should ideally cover multiple instances for a wide range of open-class words.
  Among the developed training WSD datasets, we briefly introduce SemCor (Miller et al., 1994) and its different versions, OMSTI (Taghipour and Ng, 2015), the Italian Syntactic-Semantic Treebank (Montemagni et al., 2003), CLE Urdu Sense Tagged corpus (Urooj et al., 2014) as All-words WSD datasets and DSO corpus (Ng et al., 1999) , Line-hard-Serve corpus (Miller et al., 1993) and the Interest corpus (Bruce and Wiebe, 1994) as Lexical Sample WSD datasets in the following.
  SemCor, is the first and most prominent All-words Sense annotated corpora for English. SemCor contains $352$ manually tagged documents (Taken from Brown corpus (Francis and Kucera, 1979)) and includes $226040$ sense annotations. It was initially tagged with senses from WordNet $2.1$. Sense tags of the current version of SemCor, are mapped to WordNet $3.0$ senses. Different versions of SemCor are also available for some other languages. Jsemcor (Bond et al., 2012), Eusemcor (Agirre et al., 2005), Bsemcor (Koeva et al., 2010) , and Spsemcor (Izquierdo-Beviá et al., 2006) are developed versions of Semcor for Japanese, Basque, Bulgarian and Spanish languages respectively. OMSTI (One Million Sense-Tagged Instances) is another widely used All-words sense annotated corpora for English. It was semi-automatically annotated with senses from WordNet $3.0$ and includes $911134$ sense annotations in $813798$ sentences. An English-Chinese parallel corpus (Eisele and Chen, 2010) is used for the construction of OMSTI. The Italian Syntactic-Semantic Treebank (ISST) is an Italian manually All-words sense-annotated corpus. The corpus consists of $305547$ tokens including $81236$ manually sense tagged words, annotated with Italian WordNet (Roventini et al., 2003). CLE Urdu Digest corpus is the Urdu All-words WSD corpus which contains $17006$ sense annotated nouns, tagged with senses from CLE Urdu WordNet (Urooj et al., 2014).
  The Lexical sample sense annotated corpora surveyed in this section include DSO, Line-hard-Serve and the Interest corpus. DSO corpus is a manually sense-annotated corpus including $192800$ sentences drawn from Brown corpus and Wall Street Journal. $121$ nouns and $70$ verbs have been tagged with senses from WordNet $1.5$. Line-hard-Serve is another predominant English lexical-sample corpus. It includes $12000$ instances from the American Printing House for the Blind, and the San Jose Mercury of the words $line$ (noun), $hard$ (adjective), and $serve$ (verb).

• •

  WSD Test Set corpora, which are not as large as training corpora and as a result, are not appropriate for use as training sets in supervised approaches.
  The major part of developed WSD benchmark corpora for both Lexical Sample and an All-words WSD tasks belongs to SensEval (The International Workshop on Evaluating Word Sense Disambiguation Systems) and SemEval (The International Workshop on Semantic Evaluation) competitions. SemEval (the new name of Senseval) is an ongoing series of evaluations of computational semantics systems in several languages. The main focus of Senseval-1 through Senseval-3 was on both All-words and Lexical Sample WSD tasks. The fourth series of Senseval (renamed Semeval) has been expanded to the evaluation of computational semantic analysis systems not necessarily related to WSD. The outcomes of the competitions are standard WSD frameworks for multiple languages (Edmonds and Cotton, 2001; Navigli et al., 2013; Moro and Navigli, 2015). For instance, the main benchmark for English All-words WSD (presented by Raganato et al. (2017b)) is the unified version of different Senseval and Semeval English All-words WSD tasks (Edmonds and Cotton, 2001; Snyder and Palmer, 2004; Pradhan et al., 2007; Navigli et al., 2013; Moro and Navigli, 2015). It contains $7253$ sense-annotated instances ($4200$ nouns, $1652$ verbs, $955$ adjectives and $346$ adverbs) annotated with senses from WordNet $3.0$ sense inventory. A variety of European languages such as English, French, Dutch, Italian and Spanish are covered by different series of theses competitions.
  Recently, Saeed et al. (2019b) and Saeed et al. (2019a) developed a Lexical Sample and All-words test sets for the Urdu language. The Lexical Sample corpus includes sense annotated samples for $50$ target words ($30$ nouns, $11$ adjectives, and $9$ adverbs) and the All-words corpus contains $5042$ words of Urdu running text and $466$ sense annotated words. Ambiguous words within both corpora were manually tagged with senses from the Urdu Lughat dictionary.

Although several WSD training and test corpora have been developed for a variety of languages, no WSD corpus is available for Persian. To address the lack of a standard WSD benchmark for Persian, we put forward SBU-WSD Corpus as the first Persian All-words WSD test set. We also carried out a set of best performing WSD systems on SBU-WSD-Corpus as baseline for future researches in Persian All-words WSD.

To create a standard All-words WSD test set, we followed the suggestions made by the Senseval-2 (Edmonds and Cotton, 2001) competition. For the All-words task, the Senseval-2 guidelines suggest that (1) a standard test set should contain at least $5000$ words of the running text, and (2) all context words should be tagged. The creation of SBU-WSD-Corpus can be thought of as a pipeline of four steps(i.e, Data Collection , Choosing sense inventory, Annotation process and Corpus Format), described in the following sections (Sections 3.1 to 3.4).The statistics of SBU-WSD-Corpus are presented in Section 3.5.

In this section, we present several supervised and knowledge-based systems as baselines of Persian All-words WSD task. The systems are introduced in section 4.1, the evaluation measures are explained in section 4.3 and the results and analysis about the performance of the systems are shown in section 5.

Table 2 shows the F-Measure performance of all comparison systems on the SBU-WSD-Corpus dataset. We additionally, report the performance of each system, divided by PoS tags. As it can be seen, supervised systems, trained on Persian SemCor consistently outperform knowledge-based systems across the dataset. It clearly shows the high ability of Persian SemCor on training WSD models for Persian. It is also interesting to note the performance of the MFS approach, which is considered as the baseline of supervised systems, can achieve competitive results with the best performing knowledge-based systems. One of the main conclusions that can be taken from the evaluation is the positive effect of word embedding models in disambiguating Persian words. As discussed in section4.2, all the supervised models and also Basile14 utilize word embedding models in their disambiguation pipelines. We provide a detailed analysis of the performance of Basile14, as the best performing knowledge-based model, to clearly show the effect of the word embedding model in its default pipeline.
As mentioned in section 4.1, the disambiguation pipeline of Extended Lesk and Basile14 systems are highly similar. A comparison between the results obtained by these systems indicates that the use of word embedding can have a significant impact on the performance of the system. As it can be seen in table 3, the performance of Basile14 improved by a large margin (12 percent), compared to the Extended Lesk. As discussed in section 4.1, the pipeline of Basile14 includes two key components: (1) Word Embedding model and (2) gloss definitions of the sense inventory, both are available for Persian. Existing mutual words in the gloss of different senses which result in similarity in their semantic vectors can be mentioned as the most important bottleneck of the system. To deal with this, the system expands the gloss of each sense by including the glosses of the semantically related concepts (i.e. the concepts which have a direct relation to the synset) (see more details on section 4.1).
We also reported the performance of the systems, divided by PoS tags. As it can be seen from table 3 the performance of most systems on disambiguating nouns is lower than other PoS tags. It can be explained by the ambiguity level of different PoS tags, shown in table 2. As shown in table 2, the average ambiguity of the present nouns in SBU-WSD-Corpus is 4.0 which is greater than all the other PoS tags. Additionally, in Figure 2, we showed that more than 25 percent of nouns have more than 6 senses, indicating the difficulty of noun disambiguation in the developed data set. On the other hand, adjectives and adverbs seem easier to disambiguate, as their ambiguity level is 1.6 and 1.2 respectively

In this paper, we presented a standard evaluation corpus for Persian All-words WSD. The corpus contains $19$ Persian documents, manually tokenized, lemmatized, PoS-tagged, and senses-tagged. It contains $5892$ words of running text and covers different domains including Economical, Sports, etc.
Additionally, we applied several supervised and knowledge-based WSD systems on the corpus. The results show that the supervised systems can outperform the knowledge-based alternatives. We evaluated several benchmark All-words WSD models on SBU-WSD-Corpus, providing baselines for future improvements on the Persian All-words WSD task. In addition, to encourage future research on Persian All-words WSD, we have made SBU-WSD-Corpus freely available. A possible extension to this work will include applying other knowledge-based WSD methods which are applicable to low-resource languages.