Sarcasm Detection in a Disaster Context

HurricaneSARC is a dataset of English tweets from the disaster domain, annotated for sarcasm. We randomly sampled $15,000$ tweets from the large-scale repository of tweets introduced by Ray Chowdhury et al. (2019), that were streamed during the Hurricanes Irma, Harvey, and Maria. Next, we labeled these tweets using the Amazon Mechanical Turk crowdsourcing platform. The Turk workers were provided definitions of sarcasm from dictionary.com and Wikipedia and were asked to annotate tweets for the expression of sarcasm, i.e., to assign the true label for tweets perceived as containing sarcasm (or irony), and false otherwise. The annotators were paid fairly. Each tweet was annotated by five different workers, and the final label for a tweet was computed using majority vote. To ensure qualitative results, we employed strict annotator requirements for the task, such as high acceptance rate (>$95$%), and a large amount of completed tasks ($500+$).

The annotation process produced $820$ sarcastic tweets, which amount for $5.5\%$ of the total number of sampled tweets. These tweets represent positive samples (i.e., tweets labeled as expressing sarcasm). To create the negative samples, we sampled an equal amount from the remaining pool (i.e., the tweets labeled as not expressing sarcasm). We also experimented with all negative samples as opposed to downsampling an equal amount. However, this significantly hurt model performance. Thus, consistent with Khodak et al. (2017), we construct a balanced version of the dataset on which we report the modeling results in the following sections. However, we make both dataset versions publicly available to enable further progress on sarcasm detection and understanding of sarcasm in a focused context. In our final balanced version of the dataset, we use ~$25\%$ of the data for testing, ~$18\%$ for validation, and the rest for training. Table 2 shows the number of examples in each split.

We perform a word level lexical analysis to gain insights into the vocabulary used in HurricaneSARC (compared with a general domain) and explore potential challenges and particularities of our dataset. First, we investigate the occurence of emotion and sentiment intensive words in sarcastic and non-sarcastic tweets.

To this end, we use EmoLex Mohammad and Turney (2013), a lexicon containing words and the associated emotion and sentiment evoked by these words. We study this co-occurence on HurricaneSarc by contrast with the iSarcasm dataset introduced by Oprea and Magdy (2020) from a general domain and report the co-occurence ratios in Table 3. Note that both datasets are compiled from Twitter. First, we observe that only $18\%$ of the sarcastic tweets in HurricaneSarc contain words expressing joy, whereas the ratio of non-sarcastic tweets containing joy words is significantly larger, up to $34\%$. In contrast, we observe a different pattern in the iSarcasm dataset, where both the sarcastic and non-sarcastic tweets contain a similar ratio of words expressing joy. On the same note, the HurricaneSARC dataset has a considerable amount of negatively polarized words. In fact, as many as $70\%$ of the tweets in HurricaneSARC contain at least a negative word, whereas in iSarcasm, this ratio is halved at $35\%$.

These findings suggest that the disaster-related tweets tend to be more emotion and sentiment-intensive than tweets from the general Twitter domain, since there are higher stakes involved in disaster scenarios. To better understand the particularities of our HurricaneSARC dataset, we compare a few representative examples from our dataset with some representative examples from the iSarcasm dataset in Table 4. We observe that HurricaneSARC is focused on the event and sarcasm is conveyed towards the ongoing disaster, while iSarcasm is much broader, containing tweets where the sarcasm is expressed towards various targets such as self-deprecation or healthcare.

We also note differences in the average number of words per tweet in our disaster scenario compared to the general Twitter domain. We show our findings in Table 5, where we compare our disaster setting with the general Twitter domain from the iSarcasm dataset Oprea and Magdy (2020). Interestingly, our tweets are longer in length than previous datasets, with the sarcastic tweets containing six additional words on average compared to iSarcasm.

All models are implemented using the Huggingface Transformers Wolf et al. (2020) library. For training, we use a single Nvidia V$100$ GPU. We detail the best hyperparameters used and the computational resources needed to train our models in Appendix A. To ensure statistically significant results, we run the experiments $5$ times, with different model weight initialization, and report the average of the results obtained from the five independent runs.

We show the results using the above baselines in Table 6. First, we observe that CNN performs slightly better than BiLSTM. Second, we notice that the BERTweet model pre-trained on a Twitter corpus outperforms the vanilla BERT model trained on Wikipedia and Bookcorpus by $1.7\%$ in F1, and the other baselines by as much as $9\%$ in F1. These results show that pre-trained language models are successful in learning useful language representations that are beneficial for the detection of sarcasm, and platform-specific pre-training (BERTweet pre-training) further improves the performance (compared with vanilla BERT).

Overall, despite that BERTweet performs better than the other baselines, its performance (in-domain) is still low, with the best method achieving only $0.702$ F1 (Table 6). Next, we investigate various ways to introduce inductive biases into our models by performing a thorough investigation into intermediate task transfer learning.

We explore five intermediate tasks (described below) for the supervised scenario: 1) EmoNet is a dataset Abdul-Mageed and Ungar (2017) for fine-grained emotion detection in the general Twitter domain. It contains $1.6$ million tweets labeled with the Plutchik-$8$ basic emotion categories Plutchik (1980). Here, we use a smaller version of the dataset containing $50,000$ examples provided by the authors; 2) EmoNetSent uses the same data as EmoNet. However, we aggregate the emotion labels based on the positive, negative, and neutral polarities; 3) GoEmotions Demszky et al. (2020) is a dataset of $58,000$ Reddit comments annotated with $27$ types of emotions; 4) CARER Saravia et al. (2018) contains 664,462 English tweets from the general Twitter domain annotated with 8 emotion categories. To ensure a fair comparison with the previous datasets, we downsample the data to $50,000$ examples and use only these in our intermediate pre-training step; 5) SARC Khodak et al. (2017) and 6) iSarcasm Oprea and Magdy (2020) are the datasets presented earlier in the paper.

For unsupervised intermediate tasks, we pre-train on dynamic masked language modeling (MLM) on three corpora: 1) Hurricane Ray Chowdhury et al. (2019) is a large-scale repository of $15$M tweets streamed during the Hurricanes Irma, Harvey, and Maria; 2) EmoNet Abdul-Mageed and Ungar (2017) is the emotion corpus from Twitter presented earlier. However, we remove the labels and train only on the tweets; 3) SARC is the sarcasm Reddit corpus presented earlier. Similar to EmoNet, we remove the labels and train only on the comments. The above datasets are summarized in Table 7.

Our intermediate task pre-training experiment results, shown in Table 13, reveal interesting details: First, supervised pre-training on the EmoNet emotion detection dataset is able to improve the performance over the simple BERTweet model by as much as $3\%$ in F1. Meanwhile, we also notice a $1\%$ improvement when using the iSarcasm dataset, as well as a $1\%$ improvement using the Carer task. All these tasks contain data from the Twitter domain, which indicates that positive transfers of information tend to occur when transferring information between two related domains. In contrast, we observe negative transfers of information between SARC or GoEmotions and HurricaneSARC. We attribute the performance decrease to the disparity between the domains and platforms, i.e., both SARC and GoEmotions are collected from the Reddit platform, whereas our target domain is Twitter. In the unsupervised settings, performing MLM on Hurricane yields the best performance improvement of $2$% in F1. Interestingly, unlike the supervised scenario, pre-training on SARC is able to improve the performance over the vanilla BERTweet model, while EmoNet decreases the F1 score. However, both Hurricane and SARC underwent considerably more pre-training than EmoNet, due to the significantly smaller size of EmoNet.

We summarize our results as follows: 1) Both emotion detection and sarcasm detection make good supervised intermediate tasks for sarcasm detection, as long as the data used originates from the same social platform; 2) Additional unsupervised pretraining using masked language modeling helps the performance even when the intermediate data differs from the target data, provided that the tasks are closely related (e.g., sarcasm domain) and the training data has a considerable size. We also provide results of the experiments using the vanilla BERT model in Appendix B.

To investigate the effect of intermediate task fine-tuning, we perform an error analysis of our models. Specifically, we pass the examples from the test set through two models. First, we get the predictions of the vanilla BERTweet model, which underwent no additional pre-training. Second, we compute the predictions of the best performing emotion-aware BERTweet model, which underwent additional supervised pre-training on the EmoNet emotion dataset. Finally, we analyze examples where additional pre-training is able to change incorrect predictions into correct predictions, or correct predictions into incorrect ones. Our EmoNet BERTweet misclassifies $4$ examples that are correctly classified by the vanilla BERTweet model. On the other hand, there are $28$ examples misclassified by the simple BERTweet model, which are correctly classified by the EmoNet BERTweet model. We show a few of these examples in Table 9. Interestingly, these examples convey strong emotional messages. For example, in People are dying in life support in Puerto Rico and the world is still in a frenzy about football., sadness and anger are strongly expressed along sarcasm. We argue that the supervised pre-training on EmoNet is able to induce important information about the emotions in a tweet, which correlate with the expressed sarcasm.

To answer the first question, we stage the following setup: We train our best BERTweet model on out-of-domain sarcasm detection datasets, then directly test the model on HurricaneSARC. We experiment with iSarcasm Oprea and Magdy (2020) and SARC Khodak et al. (2017) as our out-of-domain datasets, which both cover a general domain (Twitter and Reddit, respectively), whereas HurricaneSARC covers a specialized domain (Twitter). Since SARC is much larger in size compared with the other datasets, we downsample SARC to $5,000$ examples (which is roughly the size of iSarcasm) to discount for the impact of dataset size. We show the results of these experiments in Table 10 and make a few observations. First, training on SARC or iSarcasm and testing on HurricaneSARC hurts the performance considerably, decreasing the F1 by $18.9\%$ and $15\%$, respectively, compared with training and testing on HurricaneSARC. It is interesting to see that the drop in performance is much higher when training on Reddit compared with Twitter, which suggests that differences exist in the way sarcasm is expressed across platforms. On the other hand, training on iSarcasm and testing on SARC shows a decrease of only $9\%$ in F1, whereas we note a decrease of $21\%$ in F1 when training on HurricaneSARC and testing on SARC. These results reinforce that in-domain labeled data is vital in providing good model performance and understanding of the ongoing disaster. However, given the rapid unfolding of a disaster and considering that data annotation is a time-intensive process, only a small amount of annotations can be obtained as the disaster starts to unfold.

Therefore, we now turn to our second question, and propose semi-supervised learning approaches as the solution.

Our proposed methods can use large amounts of unlabeled data generated during natural disasters to considerably reduce the annotation costs as well as reduce the time needed to acquire the labeled data. We explore two state-of-the-art semi-supervised techniques, and train these methods on small subsets of our training set. Using a quarter of the entire training set, these methods perform similarly with techniques leveraging the whole set of labeled examples. We consider the following methods:

1) Noisy student training Xie et al. (2020) is a approach leveraging knowledge distillation and self-training, which iteratively jointly trains two models in a teacher-student framework. Noisy student uses a larger model size and noised inputs, exposing the student to more difficult learning environments, which usually leads to an increased performance compared to the teacher. To add noise to our input examples, we use two approaches: a) Synonym replacement: We replace between one and three words in a tweet with its synonym using the WordNet English lexical database Fellbaum (2012); b) Back-translation: We use back-translation, and experiment with different levels of noise corresponding to different translation chain lengths (e.g., English-French-Spanish-English). Smaller chain lengths lead to less noise, while increasing the length of the chain produces examples with significantly more noise. 2) Uncertainty aware Self-Training Mukherjee and Awadallah (2020) incorporates uncertainty estimates into the standard teacher-student self-training framework by adding a few highly effective changes to the typical teacher-student self-training framework, such as acquisition functions using Monte Carlo dropout and a new learning mechanism leveraging teacher model confidence.

We investigate how the above semi-supervised learning approaches can help reduce the human effort needed to annotate the data. To this end, we train the models on subsets of our dataset with only $100$ and with only $200$ training examples per class. These $100$- and $200$-size subsets are created randomly. We run each of these methods with $10$ different randomly chosen subsets, and report the mean and standard deviation of the obtained F1 for each model.

We show the results of the mentioned approaches in Table 14. We make the following observations. Our experiments involving $200$ (UST-$200$) training examples per class show the feasibility of using semi-supervised approaches for sarcasm detection. Using as little as $25\%$ of the available data, semi-supervised approaches perform similar to models trained on the whole dataset. Therefore, the semi-supervised approaches can deliver the same performance but use four times less annotated data. Since disasters are time-critical events, these results show that SSL methods can significantly contribute to a faster and more reliable understanding of an unfolding disaster. Moreover, the F1 variance of different runs of all the UST (UST-100, UST-200, and UST-ALL) methods is significantly improved compared to the vanilla BERTweet, which shows that semi-supervised models are considerably more stable.