”You should probably read this”: Hedge Detection in Text

Imagine a situation where a doctor says to a patient ”I think you need surgery immediately!” Many will take this statement seriously without any additional considerations. However, the phrase ”I think” signals uncertainty in the diagnosis. Identifying these signs of uncertain claims, known as hedges, can prevent cases of malpractice and save lives. For this reason, uncertainty detection is an important problem in medicine, finance, engineering, and other high-risk fields. In this work, we explore CoNLL-2010 shared task data [1] to improve current methods of hedge detection.

The degree of certainty in language was first introduced by G. Lakoff [2]. His work introduces the concept of hedges, which are linguistic devices that are used in conversations to indicate the degree of belief. Hedge phrases can be expressed through modal verbs (”could”, ”might”, etc.), peacock expressions (”very likely”, ”everyone”, ”I think”, etc.), and weasel words (”some believe”, ”clearly”, etc.). These expressions are context-dependent and the presence of these in a sentence doesn’t indicate uncertainty. Hence, this problem is difficult and advanced methods are required to identify uncertain claims.

In this work, we focus on detection sentences that contain hedges. We explore different neural network architectures and present a joint model problem formulation to include part-of-speech tags to improve current results on the CoNLL-2010 Wikipedia dataset. While the data used is based on Wikipedia, this work can be applied to speech as a downstream task and other similar problems in the NLP domain. The main contributions of this work are 1) an analysis of various neural network architectures and their performance, 2) a model formulation for including part-of-speech information in the input, and 3) a new top score on the CoNLL-2010 Wikipedia dataset.

Hedge detection has attracted much attention in recent years. The early work relies on shallow linguistic features such as n-grams, manually crafted hedge-word lists, and word-scoring functions [3]. M. Georgescul [4] showed the highest result during the CoNLL-2010 competition, achieving the F1 score of 60.2 using an SVM model [5] with bag-of-words (BoW) features. The general early approach to this problem is a combination of word and context analysis with BoW and part-of-speech (POS) tags represented as various size n-grams [6].

With the popularity of neural networks (NN), Patel et al. [7] applied a convolutional neural network (CNN) [8] to the problem of ambiguity detection in legal literature and showed that this model outperforms SVM models. The addition of attention [9] to the CNN model achieved an F1 score of 67.52 on the CoNLL-2010 Wikipedia data [10]. This work produced the current best result on the Wikipedia data¹¹1Han et al. [11] published new results while this paper was under review. . The work proposes attention formulations that take advantage of attention weights learned from the sentences and provided cue phrases. Their work and the F1 score of 67.52 is used as the baseline in our paper.

In recent years, the popularity of transformer architecture [12] and transformer-based language models [13, 14] led to improvements in NLP tasks, including uncertainty detection. Sinha et al. [15] showed that the fine-tuned BERT model outperformed Tree-LSTM and CNN based model on two out of three corpora, including the BioScope corpus from the CoNLL-2010 challenge. However, the authors did not report the score on the Wikipedia dataset.

Motivated by the early works and POS tags’ use, this paper makes an effort to leverage a joint word and POS model formulation to achieve a new high score on the CoNLL-2010 Wikipedia dataset. The performance improvements gained by applying transformer-based language models to the BioScope corpus further motivate our exploration of more complex models on the Wikipedia corpus.

The CoNLL-2010 Wikipedia dataset was released by [1] as a part of the CoNLL-2010 challenge. The task provides data that is derived from two datasets: biological scientific articles and articles from Wikipedia. Both datasets were manually annotated by two independent linguists who were directed to identify cues, which are expressed through the use of auxiliaries, hedge verbs with speculations, adjectives, adverbs, conjunctions, and weasel words (for the Wikipedia data). A third linguist was invited to evaluate the differences. Each sentence is labeled as certain or uncertain, and phrases that identify uncertainty are annotated as cues. The majority (77.26%) of the sentences are certain; as a result, the data is unbalanced. Hence, the evaluation results are reported as the F1-score.

While the CoNLL-2010 shared task provides data from both BioScope and Wikipedia, this work focuses solely on the Wikipedia data for the following reasons: 1) the language on Wikipedia is more general and not limited to any domains. Hence, the methods applied here are expected to be generalized and applicable to other similar problems. 2) the Wikipedia data contains annotations for weasel words, which are statements of unsupported claims. These weasel words make the uncertainty detection problem more difficult, which can be noted from the current best scores: F1 67.52 for Wikipedia and F1 86.22 for the BioScope data. For these reasons, all the experiments in this paper are focused on Wikipedia data.

The CoNLL-2010 Wikipedia dataset contains 11,110 training and 9,634 evaluation sentences. 10% of the training data is randomly selected as the development data. The models are tested on the evaluation data only used after being optimized on the development dataset. The corpus is pre-processed by tokenizing the words and stripping any punctuation that is web noise or doesn’t contribute to the sentence semantics (such as ”¡¿/-*”). The dataset is POS annotated with spaCy [16], resulting in two parallel strings containing words and POS tags, respectively. Given that the mean sentence length is 22.5 words, the sentence length is limited to 64 words.

In this work, we conduct experiments to improve current results on the CoNLL-2010 Wikipedia corpus. The training data contains only 11,110 sentences, and 10% (1,111) of that data is used as a development set for model selection. The training dataset with 9,999 data points is very small for today’s standards, presenting several challenges. We survey various word embedding models and neural network architectures to find the best model for this task. Furthermore, we look at POS tags for additional information on sentence structure. This POS tag information is used in a joint model approach to improve performance on CoNLL-2010 Wikipedia data. Finally, we analyze the models and present the final results on the evaluation dataset.

We narrow down the choice of models to three architectures to be used on the evaluation dataset. The architectures are joint input GRU and GRU+att, and joint latent space model of word GRU+att and POS LSTM+att. The main question is whether the addition of POS tags can help get better results on the evaluation dataset over the previous top F1 score of 67.52.

First, we start by testing our hypothesis by training an LSTM model on POS tags. The result of this model on the evaluation is equal to a 55.67 F1 score. We find this score high considering that the model makes predictions only on POS tags and promising for our joint model formulation. Next, the joint models are evaluated on the test dataset.

From table 4 we can see that the joint model of word GRU+att and POS LSTM+att produces top scores across the two language models. Both scores outperform the current top result on this dataset and show that leveraging POS tag information in a joint model formulation can improve the results. The previous top score on this dataset [10] introduces a novel concept of external attention where a scoring function measures each word’s similarity with respect to uncertainty-indicating phrases. These phrases would need to be derived by a human from the training dataset. Our paper shows that POS tags’ addition can achieve higher performance without a need for additional human annotations.

Furthermore, we test the performance of all models mentioned in the earlier sections on the evaluation dataset. To our surprise, a simple GRU model with custom-trained word embeddings (Wiki 1g) achieves the top score of 70.24. We hypothesize that this could be because the domain-specific language model can deliver better word representations and outperform general language models. Besides, a simpler GRU-based model architecture plays a role of regularization further improving the model performance on this small dataset. However, the results are dependent on custom word embeddings, which introduces complicity and variance.

In this work, we show that POS tags introduce additional information that can be used to improve results on hedge detection. We formulate the problem as a joint model of POS tags and words trained together. This joint model formulation achieves a high 69.74 F1 score and 70.24 F1 score with domain-specific word embeddings on the CoNLL-2010 Wikipedia dataset. This work introduces a joint POS and word approach that works well on hedge detection. In future work, we would like to continue experiments in this domain to answer questions about whether this joint model formulation works on different datasets as well as other problems in this domain. In addition, we would like to test if domain-specific word embedding performs well on other datasets. Lastly, transformers improve the performance of many NLP tasks, and further research in hedge detection applications is needed.