The Impact of Data Corruption on Named Entity Recognition for Low-resourced Languages

Natural Language Processing (NLP) is a rapidly growing field that has been applied to a wide range of tasks and domains (Vaswani et al., 2017; Conneau et al., 2020). However, much of the focus in NLP has been on high-resource languages such as English (Vaswani et al., 2017; Radford et al., 2018, 2019). While this has led to notable advancements for these languages, low-resourced languages have not received as much attention, resulting in a significant performance gap between high- and low-resourced languages. This has prompted an increasing number of studies focused exclusively on low-resourced languages, resulting in the development of models (Ogueji et al., 2021; Alabi et al., 2022) and the introduction of datasets (Adelani et al., 2021, 2022a, 2022b).

Despite this impressive progress, data remains a limiting factor for low-resourced NLP (Adelani et al., 2022a, b). In particular, the two main problems are the availability and quality of data. First, the datasets available for low-resourced languages tend to be smaller than those for high-resourced languages, and for many languages, no data exists at all (Martinus and Abbott, 2019; Adelani et al., 2021). Secondly, the available datasets are often of questionable quality, containing invalid text or incorrect annotations (Kreutzer et al., 2022), which has detrimental effects on the models trained on these datasets (Abdul-Rauf et al., 2012; Alabi et al., 2019).

This means that many existing datasets in low-resourced NLP are either small or of low quality. This observation has led to research that investigates the tradeoff between the amount and quality of data (Gascó et al., 2012; Alabi et al., 2019; de Gibert Bonet et al., 2022). This line of work has provided valuable insights that allow NLP practitioners to make informed decisions when faced with a choice of which dataset should be used to train a model. However, many of these works focus on comparing different datasets, often from different domains, without clearly quantifiable tradeoffs between data quantity and quality (Alabi et al., 2019). While these approaches can be useful, a more comprehensive and precise approach is needed to fully understand the tradeoff between data quantity and quality in low-resourced NLP.

To address this, we take a different perspective and focus on systematically and quantifiably reducing the quality of datasets and examining the effects of this on the performance of NLP models. Additionally, by altering the amount of data used to train our models, we can compare the tradeoffs between quality and quantity. We do this by devising various controllable corruption strategies, and training models on different levels of corrupted data. Our focus is on a named-entity recognition task due to its prevalence in many NLP systems and the availability of a few high-quality datasets in low-resourced languages. We fine-tune existing pre-trained language models, as this is a common and high-performing approach, especially for low-resourced languages (Ogueji et al., 2021; Adelani et al., 2021; Alabi et al., 2022).

We provide systematic evidence to support prior findings that the quality of data, in general, is strongly preferred over quantity. Furthermore, our findings are consistent across eleven different languages and four pre-trained models, suggesting that our conclusions hold true in a general sense.

Our aim is to analyse and quantify the impact of data corruption on the performance of pre-trained language models. This understanding would enable us to make more informed decisions about the relative importance of data quality and quantity, ultimately leading to improved data creation processes and selection of NLP training data for practitioners.

While we can corrupt NER datasets in various ways, we choose corruptions that simulate a mislabelling scenario during the annotation process, e.g. mislabelling a person in a sentence as an organisation. There are two main reasons for this choice. Firstly, many NER datasets are formed by taking an existing text source, which is usually of high quality, such as news data (Adelani et al., 2021) and annotating each word; thus, errors are more likely to appear during the annotation process. Secondly, it is challenging to corrupt the base sentences in a reasonable, quantifiable and incremental way, as sentences encompass meaning which is often hard to change atomically.

Thus, we focus on corrupting only the labels, using different strategies detailed in Section 3.1. For each corruption strategy, we uniformly vary the amount of corruption and train our models using the new, corrupted dataset. This process allows us to evaluate how each corruption strategy affects the model as we adjust the degree of corruption. As an additional experiment, we vary the size of the data available to the model by using only a subset of the sentences without corrupting any labels, allowing us to determine the effect of varying the amount of data on performance.

We discuss the data used in this study in Section 3.2.

Having described our corruption strategies, we now perform our experiments and showcase our results. We consider the three corruption strategies described above, and use four different pre-trained language models, described in Section 4.1. Each run consists of fine-tuning a single pre-trained model on a single language’s dataset, either the original one or a corrupted version. We fine-tune models for 25 epochs, as the results were similar to 50 epochs (which Adelani et al. (2021) used) and trained much faster. We use a learning rate of $5e-5$, a batch size of $64$, and a sequence length of $200$. We run all experiments over three seeds and average the results. The compute nodes used to run our experiments are equipped with NVIDIA RTX 3090 GPUs.

We specifically investigate the effect of progressively corrupting data on the performance of each model, measured by the overall F1 score. This simulates the effect of having low data quality (for instance, due to incorrect annotations), but allows us to study this in a controlled setting. We do not modify the test datasets at all.

Then, to normalise results across models and languages, we divide each F1 score by the value obtained when training the same model on the full, uncorrupted dataset. This effectively measures what fraction of performance is lost when corrupting data and allows us to transform all of the metrics to fall between 0 and 1, resulting in the metrics being comparable across languages and models.

In the cap sentences strategy, where we train models on a subset of data, we specifically remove a certain percentage of the data and train the model on the remaining sentences. Since the specific fraction we keep may have an effect, we run this experiment three times, each time with different random selections of data. We average over these permutations and find that the results are very similar across them.

Our work follows a recent trend of questioning whether more data is always better in NLP, even at the cost of quality (Alabi et al., 2019; Abdulmumin et al., 2022). In contrast to many of these works, we systematically and quantifiably investigate the effects of reducing the quantity and quality of data. Firstly, our findings demonstrate that the quantity of data used for training does not greatly impact final performance, as we were able to achieve around 80% performance with just 10-20% of the original dataset’s sentences. However, removing entity labels, which simulates a reduction in annotation quality, had a significant impact on performance. This supports prior findings that we do not need a large amount of data to perform well when leveraging pre-trained language models (Adelani et al., 2022a). We do note, however, that when the number of sentences falls below a certain threshold (roughly between 200 and 400 sentences), performance drops significantly, indicating that we do need a minimum amount of data to perform well. Furthermore, while quality has anecdotally been shown to be more important than quantity of data (Alabi et al., 2019; Abdulmumin et al., 2022), here we quantify this effect in NER. Our results imply that when we have the budget to label $N$ entities, using fewer fully-labelled sentences is better than using more sentences that are only partially or incorrectly labelled. Our results also suggest that even modest data-collection and annotation efforts should be able to result in datasets that are large enough to obtain decent performance. Quality, however, is of great importance and should be prioritised in the data creation process.

The second overarching observation we can make is that, in most cases, all models exhibit roughly equal behaviour, in terms of the dropoff in performance, as the level of corruption increases. This ranges from the African language-centric models (AfriBERTa and Afro-XLM-R) to the predominantly high-resourced models (XLM-R and mBERT). This suggests that the behaviour we see here is quite general, as opposed to being specific to just a particular model. The variation across the eleven languages is also remarkably low, again highlighting the consistency of our results. This is notable as we considered languages that were included in the pre-trained models’ training data as well as some that were not.

There are numerous avenues for future work. One option would be to expand our work into other NLP tasks such as machine translation or question answering, to determine whether the same trend holds. Developing new corruption strategies that are applicable to other tasks and cover other aspects of quality would also be necessary to achieve this. Investigating the effect of the corruptions on data characteristics, such as the prevalence of rare tokens, would also be valuable. Furthermore, while we do not directly consider methods to address the problems introduced by corrupted data, our work motivates research into developing better methods for dealing with corrupted datasets, mitigating some of the negative effects of training on low-quality data. One promising option would be to use active learning, to choose which sentences should be labelled, or verified by a human annotator (Gal et al., 2017).

Finally, we randomly remove sentences and find that this can still result in high performance. Recent work, however, has demonstrated that carefully choosing the data points to train on can result in significantly more data efficiency, requiring vastly less data while achieving comparable performance (Mindermann et al., 2022; Sorscher et al., 2022). This line of work is promising and has great potential in NLP for low-resourced languages.

In this paper, we present a systematic analysis of the impact of data quality and quantity on the performance of pre-trained models in a named entity recognition task for low-resourced languages. By designing multiple corruption strategies and fine-tuning models on datasets with varying degrees of corruption, we are able to provide useful insights into the relationship between data quality and model performance.

Our results, which are consistent across pre-trained models and languages, demonstrate that pre-trained models can perform effectively with minimal data and that missing or incorrect annotations have a much greater negative impact than having fewer fully-labelled sentences. The findings of this study have the potential to inform future NER dataset creation efforts and aid NLP practitioners in selecting appropriate datasets for fine-tuning.