ELF22: A Context-based Counter Trolling Dataset to Combat Internet Trolls

A troll teases people to make them angry, offends people, wants to dominate any single discussion, or tries to manipulate people’s opinions [Mihaylov and Nakov, 2016]. Trolling shows a specific type of aggressive online behavior involving antagonism for the sake of amusement such as a deliberate, deceptive and mischievous attempt to provoke a reaction from other online users [Herring, 2002, Shin, 2008, Binns, 2012, Hardaker, 2013, Golf-Papez and Veer, 2017]. In particular, Hardaker [Hardaker, 2013] introduced the notion of overt and covert trolls with a detailed scale of trolling bahaviour in six categories: aggress, shock, endanger, antipathize, hypocriticize, and digress. Since we focus on counter responses to combat trolls, we used two super-categories, overt and covert strategies, to classify trolls by their detailed tendencies (see Table 1).

The function of countering trolls is to be a “virtually capable guardian”, who can neutralize the impact of trolling on targets. Note that when devising a counter response, it is not always possible to identify who triggered the action of trolling through comments [Hardaker, 2013]. Therefore, we followed the taxonomy of using seven response strategies for counter responses, introduced by Hardaker [Hardaker, 2015], as follows.

Engage strategy indicates acting in accordance with the troll’s intentions. This strategy involves accepting the troll’s opinions positively and responding sincerely to the troll. From the third party’s point of view, however, the users employing this strategy could be considered to have fallen prey to the troll.

Ignore strategy has the goal of not giving what the troll wants. Users attempt to warn others about doubtful behaviors of the troll. The strategy is to deter trolls from their malicious intention so that the trolls leave the discussion. For this strategy to work, it is necessary to be proactive in discovering the troll’s intentions.

Expose strategy takes a stance either going against the troll’s opinion or doubting the authenticity of the troll’s information. This strategy has the risk of being trolled by responding to the message of trolls; however, it could keep the discussion consistent and healthy.

Challenge strategy provides direct opposition to the troll with an aggressive response. Users often use emotional language to express their hostility. This strategy has a similar purpose to expose in terms of confronting trolls, but challenge tries go more strongly against what trolls have said.

Critique strategy goes beyond discovering the true intention of the troll, and overtly judges the troll’s behavior as degrading and uncreative. This strategy weakens the troll’s effectiveness by evaluating its actions rather than the content of the troll comment.

Mock strategy involves making the troll scorned and ridiculed. With mocking, users attempt to isolate the troll while strengthening the cohesion of discussion participants.

Reciprocate strategy attempts to attack the troll by pretending to be another troll. This strategy takes a more radical stance than the challenge strategy in that the purpose is to attack the troll rather than to self-defend against the troll.

For the data collection, we crawled the posts that include troll comments from Reddit using Pushshift API [Baumgartner et al., 2020]. We describe how we limited our conditions as well as some of the significant parts of our data collection process. First, we extracted our data from diverse subreddit communities, taking into account the fact that trolls are “light speakers”, who are not, or do not have the capacity to be, deeply involved in a conversation, as opposed to “heavy speakers”, who present logical arguments to discuss with others [Choi et al., 2015]. Second, we focused on the importance of the fast interactions and interchange of messages amongst online community users, which aggravates the propagation of troll comments, since virality, responsiveness and volume are the three important factors in terms of conversation characteristics of Reddit [Choi et al., 2015]. Third, we considered the title and body text of the post as context. In order to exclude the possibility of disconnection in the context history, we only extracted root comments if they were described to be by trolls.

To collect unified data types, we limited our range of extracting data of online trolls. Concretely, we set the range of thumbs down of troll comments from -2 to -15. The upper limit of a downvoted comment was set to -15, as it excludes comments heavily biased against community tendencies. Next, we extracted the highest-scored comment of thumbs up among the following comments from the root comment as a counter response reference.

We discarded posts if they contained URLs, included pictures, had more than 512 characters, or used language other than English. We then selected the posts whose number of characters is at least 12. Samples of the collected data are shown in Table 2.

For this annotation task, we recruited 12 annotators in total who are familiar with the Reddit website. Native English speakers or those whose education took place in a country where English is used as an official language participated in this experiment since our collected dataset is in English.

The annotation phase of the dataset is organized in two sessions, each of which took 7 days to complete. We asked all the annotators to follow the guidelines which include the definition of trolls, the importance of combating them, and an explanation of each strategy with examples (see Section 3). Since these sessions were conducted online, we devised a strictly controlled environment in order to obtain a qualified dataset. Each annotator had his or her own personalized Google sheet link to annotate a scheduled amount of the dataset every day. Each day, annotators submitted their labeled dataset in order to receive a confirmation from our team. In addition to this, we regularly opened online QA sessions to help annotators, at the same time ensuring that they were faithfully engaged and fully familiar with the task.

For the first annotation session, annotators were given Reddit posts with subreddit names, titles, and body text, including two target sentences to annotate: a ‘troll comment’ and its ‘response’. Annotators were asked to read the troll comment and label it according to whether they considered it to be a ‘covert’ or ‘overt’ type of troll. They were then asked to read the response, which was considered to be a combating sentence against the troll, before giving it a label according to the 7 strategies from the guidelines. We gave the annotators an option to propose their own choice of a counter response strategy against the troll if they believed that the given ‘response’ was considered ineffective. Since the entries of the dataset were systematically distributed so that three annotators could work on the same entries of the dataset, we found disagreements among the annotators after the first session. Therefore, for the second annotation session, if all the three annotators gave a different label to the text, we asked them to re-consider their choices by discussing among them and re-labeling the data. We chose the majority labels otherwise.

Table 2 shows examples of the five elements in each entry: subreddit (omitted in Table 2), title, post, troll (troll comment), and response (counter response). The first entry is a post about vaccination and asking the opinion of others regarding the decision of refusing to be vaccinated. The comment is identified as a troll by the subreddit users, which we interpret as a troll that indirectly aggresses other locutors but does not directly name any reddit user.

First, we collected from Reddit 5,700 posts that include 2,198 unique subreddits. The top 10 subreddits of the collected posts were: unpopularopinion, wallstreetbets, teenagers, nba, relationship_advice, DestinyTheGame, NoStupidQuestions, Genshin_Impact, fo76, and NonewNormal.

We removed posts with incomplete composition during the annotation process. If a sentence was labeled differently such as expose, challenge, and reciprocate by three annotators even after the second annotation session, we have considered it as outliers and discarded it. We only extracted the labeled posts that obtained the majority agreement among three annotators. After filtering, we finalized our dataset with a total 5,535 pairs of troll comments and counter responses, together with 1,151 ‘your response’ sentences that were made by the annotators.

Table 3 shows information about the average and maximum numbers of words of our dataset, where the average numbers of words of contextual texts, troll comments, and responses are 69.2, 38.4, and 25.6, respectively.

Figure 1 shows the distribution of response strategy types in our dataset. We see that Reddit users often used the engage strategy, followed by the exposure and challenge strategies. On the other hand, the ignore and reciprocate strategies occupied only a small portion of the dataset.

Table 4 presents the statistics of our dataset with respect to each label. The labeled posts are randomly split into training (80%), validation (10%), and test (10%). We note that each post has a unique post number in accordance with trolling comments. This means that we assigned the same post number for the comments of countering trolls and ‘your response’ sentences. The consequent distribution of entries of the ELF22 dataset was training (80%), validation (8.6%), and test (11.4%).

To check the validity and consistency of the collected dataset, we performed the Inter-Annotator Agreement (IAA). The Fleiss’s Kappa ($\kappa$) value of the troll type label of our dataset was 0.539, and the value of the counter strategy label was 0.465, achieving a ‘moderate’ agreement between the two labels [McHugh, 2012].

Our annotation task was approved by the Institutional Review Board (IRB)⁵⁵5Approval number: KH2021-126. All participants of annotation tasks indicated their understanding of the procedure for the annotation and acknowledged their agreement to participate. We report that all data are anonymized, and we plan to ensure that future researchers follow ethical guidelines, with the necessary information collected as the occasion demands.

The goal of our work is to categorize responses against trolls in online conversations and support the development of generation bots for countering trolls according to seven strategies proposed in this paper. As shown in Tables 2, 8, and 9, our dataset may contain sarcastic and aggressive language. We tried to observe how they communicate as-is, even though it could include socially biased content or hate speech. We expect that our dataset will be helpful for future work to study effective methodologies for responding to troll language online.

When deploying language models for response generation on our dataset, the models could generate unintended results, such as social and racial biases. There could be solutions including list-based token filtering, prompt conditioning [Keskar et al., 2019], domain-adaptive pre-training [Gururangan et al., 2020], or any detoxification techniques [Gehman et al., 2020] to avoid unintended response generation in accordance with the research purpose.

In this section, we describe three tasks that are used to exploit our dataset: troll strategy classification as binary classification, response strategy classification as multi-label classification, and generating counter responses as text generation.

Task A. troll strategy classification This task involves classifying troll comments with titles and body texts and stating whether the comments are overt trolls or covert trolls. We implemented support vector machine (SVM) and random forest (RF) as dictionary-based classifiers, and BERT [Devlin et al., 2019] and RoBERTa [Liu et al., 2019] as pre-trained transformer-based classifiers. We utilized the pre-trained transformer models from the Huggingface library [Wolf et al., 2019]. The hyperparameter setting for each model is as follows:

SVM We used a linear kennel with a hyperparameter $c$ value of 0.01 out of [0.01, 0.1, 1, 10, 100, 1000, 10000] and a gamma of ‘auto’.

RF We used a hyperparameter $d$ value of 100 out of [0.01, 0.1, 1, 10, 100, 500, 1000].

BERT We used ‘bert-base-cased’ pre-trained checkpoint and fine-tuned the model on our train dataset with 10 epochs. The size of the batch was 32, the learning rate was $2e$-$05$, the weight decay was $1e$-$2$, the warm-up steps were 100, the drop-out rate was set to 0.1, and the max sequence length was 512, which are the same as those that Devlin et al. [Devlin et al., 2019] used to fine-tune models.

RoBERTa We used a ‘roberta-base’ pre-trained checkpoint with a similar number of model parameters to BERT. We followed hyperparameters of the same values as BERT.

For dictionary-based classifiers, the words with the top-512 frequencies from the title, body text, and troll comment were included in a bag of words as input. We prepared a list of candidate hyperparameters for a grid search and selected the one with the best weighted F1 score in the validation dataset. Training times of BERT and RoBERTa were about 10 minutes on one NVIDIA A100-PCIE-40GB. We reported the average results under 5 fine-tuning runs from our test dataset.

Task B. response strategy classification This task involves classifying responses with titles, body texts, and troll comments, stating whether the responses are used in one out of 7 counter strategies. We experimented with the same baselines used in Task A. Aside from 0.1 being selected for SVM $c$, we used the same hyperparameter settings for each model as in Task A.

Task C. counter response generation This task involves generating responses with titles, body texts, troll comments, and labeled strategies. We implemented the following competitive baseline models:

BART [Lewis et al., 2020] is a transformer encoder-decoder model that is trained on Common Crawl News, Wikipedia, book corpus, and stories. We employed a ‘bart-base’ model that contained 6 encoder layers and 6 decoder layers. The size of the batch was 32.

GPT-2 [Radford et al., 2019] is a transformer decoder model that is trained on Common Crawl Webtext. We employed a ‘gpt2’ model that contained 12 decoder layers. The size of the batch was 16.

DialoGPT [Budzianowski and Vulić, 2019] is an additionally pre-trained GPT-2 model on a dialog corpus. Fine-tuning configuration is the same as that of the GPT-2 model.

We fine-tuned the models over 20 epochs with a learning rate of $2e$-$04$, a weight decay of $1e$-$2$, a drop-out rate of 0.1, a max input sequence length of 768, and a max output response length of 128.

In the decoding phase, we employed a beam search with the number of beams set to 5 and with blocking trigram repeats, which is widely used to avoid low-quality generation results [Klein et al., 2017, Paulus et al., 2017]. Based on the GPT2 tokenizer, the average token lengths of train, validation, and test input were 124.2, 122.9, 121, respectively. The average token lengths of train, validation, and test ground-truth responses were 28.1, 29.1, 25.5, respectively. Training times of GPT2, DialoGPT, and BART were each about 30 minutes on one NVIDIA A100-PCIE-40GB.

Tasks A and B. For the classification tasks, we evaluated the weighted Precision (P), Recall (R), F1 score (wF1), and macro F1 score (MF1).

Task C. For the generation task, we performed automatic evaluation and human evaluation. On Task C., we used ROUGE-L F1 score [Lin, 2004], BLEU-1 [Papineni et al., 2002], METEOR [Banerjee and Lavie, 2005], and BERTScore [Zhang et al., 2020] for the automatic evaluation. ROUGE-L F1, BLEU-1, and METEOR are based on n-gram overlaps that evaluate the similarity between the ground-truth response and the predicted one. BERTscore is a machine learning based automatic metric that captures semantic similarities by comparing their contextual embeddings.

Table 5 shows the performance of the models on Tasks A and B. The models are fine-tuned on our train dataset and evaluated against the ground-truth labels for 762 test samples. The pre-trained transformer models BERT and RoBERTa outperformed the machine learning-based models SVM and RF. It implies that labeled strategies in our dataset are independent of simple lexical features, while sufficiently considering contextual text. For Task A, the wF1 scores of SVM and RF were close to 0.5, slightly higher than the random selection performance of binary classification. For Task B, we observed that the transformer-based models had about 50% higher wF1 performance than the machine learning-based models. We speculate that Task B needs to consider semantic features to improve the performance. We also see that SVM and RF show low F1 performance on reciprocate and ignore strategies that contain a small number of train samples, resulting in poor MF1 performance. On the other hand, BERT and RoBERTa show robust performance with the unbalanced dataset.

Table 6 shows the performance of the generation models on Task C. To confirm the effectiveness of the proposed dataset, we experimented with two pre-trained models and three fine-tuned models. Three fine-tuned models showed higher BLEU scores than pre-trained models. We find that BART-ELF22 shows the highest ROUGE and BLEU scores, indicating that words in generated response have the most overlap with ground-truth references. BART showed the highest METEOR performance, which means that it achieved high n-gram precision and recall with the ground-truth references. Aside from BLEU, BART and BART-ELF22 showed high performance on ROUGE, METEOR, and BERTScore. The high BERTScores of BART and BART-ELF22 imply that the generated sentences of the two models derive outputs semantically similar to the references.

To verify that the proposed dataset is enough to train a generation model which produces counter sentences against trolls with a specified response strategy, we conducted human evaluation on counter response generation models. Automatic evaluation metrics are related to the quality of generated texts, but they underestimate texts with a smaller vocabulary and heavily rely on ground-truth references. In particular, this phenomenon is frequently observed due to the characteristics of the Reddit community, which has a lot of free discussions and dialogues. To demonstrate the effectiveness of our dataset while embracing this phenomenon, we evaluated the counter response texts based on two criteria: relevance and compatibility.

First, we experimented with how well our dataset could be used to train a generation model to generate a counter response of a certain strategy when there is a corresponding golden response strategy and text. We evaluated generated texts from the ‘relevance’ point of view [Zhu and Bhat, 2021]. Relevance includes an assessment of the following two factors: 1) how coherent or contextually matched the generated counter response is with the troll text, and 2) how well the generated text counters troll text. We ranked differently generated responses to assess relevance. The higher rank refers to more contextually coherent and effective responses to troll comments. On the other hand, the lower rank refers to responses that may be out of context and endanger the troll victims.

Second, we assessed how effectively our dataset could help train a generation model to create counter responses that correspond to strategies not in the dataset from the ‘compatibility’ perspective. The dataset consists of pairs of a troll comment and its response, with only one golden response strategy and response text to one troll comment. In this regard, we checked if it was necessary to evaluate whether the dataset would be sufficiently effective for the model to generate responses corresponding to various but untrained response strategies against a troll text. Compatibility captures how well the model generates responses to given strategies, scored on a Likert scale of 1 to 5. If a counter response is related to the given strategy and close to a human response, the response gets a higher score.

A total of 5 evaluators fluent in English and familiar with Reddit forums participated in this human evaluation task. These five evaluators were equally assigned with 100 randomly selected samples from our dataset. Before the evaluation process, all evaluators were introduced to the definition of trolls and the seven strategies of countering trolls.

All the five sentences in a single post were evaluated by five evaluators: three response sentences for relevance evaluation and two response sentences for compatibility evaluation. The three response sentences for relevance evaluation include: (1) the ground-truth response; (2) the GPT-2 generated sentence; and (3) the BART-ELF22 generated sentence⁶⁶6We also performed a test on BART, but the result is excluded from the evaluation because the output is easily discerned. We employed the BART-ELF22 model that performed better in the automatic evaluation of Task C.. Two response sentences include the generation of GPT-2 and that of BART-ELF22 when a randomly selected strategy was given.

Figure 2 shows the results of human evaluation. We confirm that BART-ELF22 (Ours) performed better than GPT-2 (baseline) in both relevance and compatibility.

For the relevance criterion, we compared three responses: ground-truth, a response by BART-ELF22 which was fine-tuned with our dataset, and GPT-2, which was not fine-tuned with our dataset. As shown in the left bar plot in Figure 2, the mean values of ground-truth, BART-ELF22, and GPT-2 were 1.37, 2.12, and 2.51, respectively. According to Friedman test, there was a significant difference in mean rank between three responses ($\chi_{2}^{2}=10.00$, $p<.01$). For the post hoc comparison test, we used Wilcoxon Signed Ranks test. The result was in the order of ground-truth, BART–ELF22, and GPT-2, with significance established at $p<.05$. We speculate that fine-tuning with our dataset does improve the quality of the model to generate context-relevant sentences.

For the compatibility criterion, we compared two responses generated by BART-ELF22 and GPT-2. As shown in the right bar plot in Figure 2, the mean values of BART-ELF22 and GPT-2 were 3.22 ($sd=0.18$) and 2.10 ($sd=0.18$), respectively. According to one-way repeated measure ANOVA test, there was a significant difference in the mean values between two responses ($F_{(1,4)}=41.8,p<.01$). We see that fine-tuning with our dataset significantly improves the performance of strategy-controlled sentence generation.

Table 7 shows examples of five corresponding sentences generated by the model when given contextual information and a response strategy labeled in our dataset. In Table 7, the three counter responses above are generated by the ground-truth reference, GPT-2, and BART-ELF22 models, respectively. The two counter responses below are generated by BART-ELF22 with the other strategies.

We find that the counter sentences generated by GPT-2 contain frequently repetitive words and composed of fewer informative tokens regardless of strategies. In BART-ELF22, as in GPT-2, there were some cases where the variation was small even if the given strategy was changed, but the frequency was relatively small compared to GPT-2. We observe that BART-ELF22 generates data by varying counter responses according to the specified strategy. This result suggests that the way the BART-ELF22 appends the strategy phrase-wise to the context does work. We see that the proposed dataset supports improvement on controlled response generation according to the response strategy.