Stance Detection in Web and Social Media: A Comparative Study

Stance detection using traditional feature engineering: Various works on stance detection use traditional feature engineering. For instance, Sen et al. [20] proposed a novel set of features with SVM model and a feedforward neural network model. HaCohen-kerner et al. [10] used 18 features including character skip-ngrams and character ngrams. Dilek et al. [15] used unigrams, bigrams, hashtags, external links, emoticons, and named entities as features to a SVM model. Dey et al [5] proposed a two-phase SVM architecture with borrowed and novel feature sets.

Stance detection using neural models: In recent years, there have been many works using neural models for stance detection. Du et al. [7] used an attention based model for stance classification. Mitre et al. [28], the winning team of the SemEval 2016 Task 6A challenge, proposed a transfer learning method with features learned via distant supervision on two large unlabelled datasets. Wei et. al. [27]), the second position holders of SemEval 2016 Task 6A challenge, used Kim’s CNN. Chen et al. [3] applied neural network model to classify stance of social media posts by considering users’ taste, topics’ taste and user comments on posts, whereas Dey et al. [6] used a two phase LSTM model.

Surveys on stance detection models: There have been some relevant surveys as well. Zubiaga et al. [29] discussed various stance detection approaches for rumour detection and resolution. Several stance detection approaches were compared on Spanish and Catalan datasets in the StanceCat task [24]. Wang et al. [25] analysed the shortcomings of different stance detection models. But this survey did not actually compare performances of different methods over specific datasets, and also did not explore the reproducibility of different models, as is done in the present paper.

(1) Convolutional Neural Networks: This method [27], which uses Kim’s 1-D CNN-based sentence classification model [14], performed second-best in the SemEval stance detection task (Task 6A). We used the code that has been made available by the authors.²²2https://github.com/nestle1993/SE16-Task6-Stance-Detection Note that, we applied our pre-processing techniques on the dataset before applying this model, and this step significantly improves the performance (see Section 7).

(2) Target-Specific Attention Neural Network [TAN]: Du et al. [7] proposed a novel bidirectional LSTM-based attention mechanism. We briefly describe the architecture below. A target sequence of length $N$ is represented as $[z_{1},z_{2},\ldots,z_{N}]$ where $z_{n}\epsilon R^{d^{{}^{\prime}}}$ is the $d^{{}^{\prime}}$-dimensional vector of the $n$-th word in the target sequence. The target-augmented embedding of a word $t$ for a specific target $z$ is $e^{z}_{t}=x_{t}\odot z$ where $\odot$ is the vector concatenation operation. The dimension of $e^{z}_{t}$ is $(d+d^{{}^{\prime}})$. An affine transformation maps the $(d+d^{{}^{\prime}}$)-dimensional target-augmented embedding of each word to a scalar value as per Eqn. 1:

$$a^{{}^{\prime}}_{t}=W_{a}e_{t}^{z}+b_{a}$$

(1)

where $W_{a}$ and $b_{a}$ are the parameters of the bypass neural network. The attention vector $[a^{{}^{\prime}}_{1},a^{{}^{\prime}}_{2},\ldots,a^{{}^{\prime}}_{T}]$ undergoes a softmax transformation to get the final attention signal vector (Eqn. 2):

$$a_{t}=softmax(a_{t})=\frac{e^{a^{{}^{\prime}}_{t}}}{\sum_{i=1}^{T}e^{a^{{}^{\prime}}_{i}}}$$

(2)

Challenges in Reproducibility: Du et al. [7] mentioned that they trained embeddings on a manually scraped corpus, but they neither released the corpus nor the embeddings. We have used the pre-trained Glove 6B 300d embeddings for this purpose.³³3https://nlp.stanford.edu/projects/glove/ Additionally, whether dropout is used and what activation function is used in the middle layers were not mentioned in [7]. We have used dropout and ReLU activation function.

An observation about the TAN model: Du. et al [7] claim that using the target-augmented embeddings enable the model to make “full use of the target information in stance detection” (quoted from [7]). However, we believe that this architecture does not take advantage of the target information at all. We give a simple proof for our claim:

Proof: From Eqn. (1), we have: $a^{{}^{\prime}}_{t}=W_{a}e_{t}^{z}+b_{a}\implies a^{{}^{\prime}}_{t}=W_{a}(x_{t}\odot z)+b_{a}$

$$\because W_{a}=W_{ax}\odot W_{az}\>(\mbox{where}\>W_{ax}\epsilon R^{d}\>\mbox{and}\>W_{az}\epsilon R^{d^{{}^{\prime}}}):$$

$$\therefore a^{{}^{\prime}}_{t}=W_{ax}\cdot x_{t}+W_{az}\cdot z+b_{a}$$

Now, from Eqn. (2), we have:

$$a_{t}=\frac{e^{a^{{}^{\prime}}_{t}}}{\sum_{i=1}^{T}e^{a^{{}^{\prime}}_{i}}}=\frac{e^{W_{ax}\cdot x_{t}+W_{az}\cdot z+b_{a}}}{\sum_{i=1}^{T}e^{W_{ax}\cdot x_{i}+W_{az}\cdot z+b_{a}}}=\frac{e^{W_{ax}\cdot x_{t}}}{\sum_{i=1}^{T}e^{W_{ax}\cdot x_{i}}}$$

$$\therefore\frac{da_{t}}{d{z}}=0$$

We also back our claim with an empirical experiment, wherein we do not augment the target embeddings to the word-embeddings in the bypass neural network, i.e., we use $e^{z}_{t}=x_{t}$ (instead of $e^{z}_{t}=x_{t}\odot z$ in the original model). We call this architecture the TAN-. We show later in the paper that results obtained by both the TAN and TAN- architectures are very similar.

(3) Recurrent Neural Network with Long Short Term Memory(LSTM): In the earlier TAN paper [7], one of the baselines was LSTM without target-specific embedding and target-specific attention. In this work we have also reproduced this LSTM-based method.

Challenges in Reproducibility: The challenges faced were same as TAN model and we took the same steps as for the TAN model [7].

(4) SVM-based SEN Model: Sen et al. [20] proposed a SVM based stance detection model using five sets of features – stance vector, textual entailment, sentiment, medical knowledge based feature and a standard context based BoW feature. The stance vector is created on a sentence level based on an assumption [1] that the main information present in a sentence revolves around some particular parts-of-speech like the Nouns, Adjectives, Verbs, Adverbs. Thus these parts-of-speech are the main building blocks of the stance expressed by a sentence towards a particular claim. To identify the sentiment feature (positive or negative or neutral), we used a standard sentiment analyzer given in Stanford CoreNLP Toolkit to obtain the sentiment for a sentence.

Challenges in Reproducibility: In the BoW feature, we used word-unigrams, since the exact value of $n$ for n-grams was not mentioned in the original paper [20].

(5) Two-step SVM: Dey et al. [5] proposed a two-step stance detection approach. In the first step, they find whether a tweet is relevant to the given claim, and in the next step they detect the stance (if the tweet is relevant). The first step uses features such as Weighted MPQA Subjectivity-Polarity Classification and Wordnet Based Potential Adjective Recognition, whereas the second phase comprises of Sentiwordnet and MPQA Based Sentiment Classification, Frame Semantics, Target Detection, Word n-Grams and Character n-Grams.

Challenges in Reproducibility: According to Dey et al. [5], the two most important features are (i) Wordnet Based Potential Adjective Recognition and (ii) Frame Semantics. Especially, in the second phase, frame semantics is the most decisive feature. This feature attempts to estimate the relative importance of multiple clauses present in a tweet, where the clauses are considered to be separated by ‘connector words’. However, there is lack of clarity about this feature. It is written that “We assign more weightage to the more important clause, in case connector words are present in the sentence.” (quoted from [5]). But it is not clarified how exactly the relative weights of the clauses are decided. Due to this lack of clarity, we could not implement the frame semantics feature of this model. We have implemented all other features except frame semantics.

A general challenge in reproducibility across all models: Almost none of the prior works described in this section stated the exact values of the hyperparameters in the models. Hence, we adopted the same approach as Du et al. [7] – hyperparameters were tuned using 5-fold cross-validation on the training set (of each dataset).

Hyper-parameter Tuning: As stated in Section 4, almost none of the prior works specify all the hyperparameter values. We tuned all hyperparamters (that are not stated in the respective papers) using 5-fold cross-validation on the training set. In case a hyperparameter value is specified in a paper, the said value is used. Model-wise hyperparameter values that were tuned by us are mentioned in Table 3 (using same notations as in our codes (for those not mentioned in the respective papers)).

Vote Scheme: We use the vote scheme proposed in [27] for prediction on the test set. For each model, we run ten parallel epochs, whose validation sets are randomly selected from the training set and are non-overlapping. According to [27], in each epoch, some iterations are deliberately chosen to predict the test set. Then, when this epoch ends, for every sentence in the test set, the label which appears most frequently in these predictions as the result of this epoch is appointed. Finally, when ten epochs end, voting happens within results of these ten epochs by the same method described above to determine the final labels. Performing multiple times independently and voting twice provides a robust mechanism for predicting. Note that the voting scheme is used for TAN, TAN-, CNN and LSTM models only.

Performance Metric: To evaluate the performance of all models, we use the same metric as used by the official SemEval 2016 Task A [18] – the macro-average of the F1-score for ‘favor’ and ‘against’ classes.

Effect of Preprocessing on existing methods: In this work, we applied some tweet-specific preprocessing (tweet normalization and hashtag preprocessing) on the SemEval datasets (as stated in Section 3). Table 4 reports the performance of some of the models, as reported in the original paper (that proposed a model) and after this tweet-specific preprocessing. We see that the performance of the existing methods improves significantly due to this preprocessing.

Comparative analysis: Table 5 and Table 6 describe the performances of all models on SemEval dataset and MPCHI dataset respectively. Since we could not reproduce the Frame Semantics feature of the two-step SVM model [5], we have reported both the performances of our implementation and that reported in the original paper [5] for the SemEval datasets (the original paper worked only on the SemEval datasets, not the MPCHI datasets).

Where all models fail: We have considered the labels predicted by all the seven models, and checked those tweets/texts where all the models fail (i.e., no model was able to give the correct label). In total, there are 72 tweets (across all SemEval datasets) and 42 posts (across all MPCHI datasets) where all models classified wrongly; Table 7 shows some examples of such tweets/text. We manually observed these misclassified tweets and text, and made the following observations.

• •

  In case of tweets (the SemEval datasets) the errors were mostly on tweets that contain (i) sarcastic comments [23], and (ii) questions.

• •

  In case of the MPCHI dataset, there were some posts containing health-related facts, which actually express no stance w.r.t the target. All the models were unable to capture this notion. It is possible that the stance can be understood by a human having a lot of contextual background knowledge; however, it is difficult to understand the stance just from what is mentioned in the tweet/text.

Acknowledgement: The work is partially supported by a project titled “Building Healthcare Informatics Systems Utilising Web Data” funded by Department of Science & Technology, Government of India.