Check_square at CheckThat! 2020:
Claim Detection in Social Media via Fusion of Transformer and Syntactic Features

We use two publicly available pre-processing tools for English and Arabic tweets. We use Baziotis et. al.’s [3] tool for English to apply the following normalization steps: tokenization, lower-casing, removal of punctuation, spell correction, normalize hashtags, all-caps, censored, elongated and repeated words, and terms like URL, email, phone, user mentions. We use Stanford Stanza [39] toolkit to pre-process Arabic tweets by applying the following normalization steps: tokenization, multi-word token expansion and lemmatization.

In the case of extracting word embeddings from a transformer network, we use the raw text as the networks have their own tokenization process.

We use the following syntactic features for English and Arabic tasks: Parts-of-Speech (POS) tags, named entities (NE) and dependency parse tree relations. We use the pre-processed text and run off-the-shelf tools to extract syntactic information of tweets and then convert each group of information to feature sets. For English we used spaCy[24] and Stanford Stanza [39] for Arabic tweets to extract the following syntactic features. In all the features, we experiment with keeping and removing stop-words to evaluate their affect.

Part-of-Speech: For both English and Arabic, we extract 16 POS tags in total and through our empirical evaluation we find that the following eight tags to be the most useful when used as features: NOUN, VERB, PROPN, ADJ, ADV, NUM, ADP, PRON. For Arabic, the additional four tags are useful features: DET, INTJ, AUX, PART. We used the chosen set of POS tags for respective language to encode the syntactic information of tweets.

Named Entities: We identified the following named entity types to be the most important features through our evaluation: (GPE, PERSON, ORG, NORP, LOC, DATE, CARDINAL, TIME, ORDINAL, FAC, MONEY) for English and (LOC, PER, ORG, MISC) for Arabic. We also found that while developing feature combinations named entities do not add much value to overall accuracy, and hence our primary and contrastive submissions do not include them.

Syntactic Dependencies: these features are constructed using dependency relation between tokens in a given tweet. We use the dependency relation between two nodes in the parsed tree if the the child and parent nodes’ POS tags are one of the following ADJ, ADV, NOUN, PROPN, VERB or NUM. All dependency relations that match the defined constraint are converted into the triplet relation such as (child node-POS, dependency-relation, parent-POS) and pairs such as (child node-POS, dependency-relation) where the relation is not part of a feature representation. This process is shown in Figure 1. We found that the features based on pairs of child and parent node perform better than the triplet feature. The dimension of the feature vector for English and Arabic is 133 and 134 respectively.

For encoding a feature, we get a histogram vector which contains the number of type of tag, named entity or syntactic relation pair. The process of feature encoding is shown in Figure 1. Finally, we normalize each type of feature with maximum value in the vector.

One simple way to get a contextual representation of a sentence is to average the word embeddings of each token in a given sentence. For this purpose, we experiment with three types of word embeddings pre-trained on three different sources for English: GloVe embeddings [37] trained on Twitter and Wikipedia, word2vec embeddings [31] trained on Google News, and FastText [30] embeddings trained on multiple sources. In addition, we also experiment with removing stop-words from the average word representation, as stop-words can dominate in the average and result in less meaningful sentence representation. For Arabic, we use word2vec embeddings that are trained on Arabic tweets and Arabic Wikipedia [49].

Another way to extract contextual features is to use BERT [9] embeddings that are trained using the context of the word in a sentence. BERT is usually trained on a very large text corpus which makes them very useful for off-the-shelf feature extraction and fine-tuning for downstream tasks in NLP. To get one embedding per tweet, we follow the observations made in [9] that, different layers of BERT capture different kinds of information, so an appropriate pooling strategy should be applied depending on the task. The paper also suggests that the last four hidden layers of the network are good for transfer learning tasks and thus we experiment with 4 different combinations, i.e. concatenate last 4 hidden layers, average of last 4 hidden layers, last hidden layer and $2^{nd}$ last hidden layer. We normalize the final embedding so that $l2$ norm of the vector is 1. We also experimented with BERT’s pooled sentence embedding that is encoded in the CLS (class) tag, which performed significantly poorer than the pooling strategies we employed. For Arabic, we only experimented with a sentence-transformer [40] that is trained on multilingual training corpus and outputs a sentence embedding for each tweet/sentence.

Sentence Representation: To get the overall representation of the tweet, we concatenate all the syntactic features together with either average word embedding or BERT-based transformer features and then apply PCA for dimensionality reduction. SVM classifier is trained on the feature vectors of tweets to output a binary decision (check worthy or not). The overall process is shown in Figure 2.

As a backbone network to extract sentence embeddings and fine-tuning with triplet loss, we use the recently proposed Sentence-BERT [9] that learns the embeddings in a Siamese (pairs) and triplet network settings. We experiment with the pre-trained Siamese Network models trained on SNLI (Stanford Natural Language Inference) [5] and STSb (Semantic Textual Similarity benchmark) [7] datasets that have been shown to perform very well for semantic textual similarity.

English dataset consists of training, development (dev) and test splits with 672, 150 and 140 tweets respectively on the topic of COVID-19. We perform grid search using development set to find the best parameters. Arabic dataset consists of training and test splits with 1500 tweets on 3 topics and 6000 tweets on 12 topics respectively with 500 tweets on each topic. For validation purpose, we keep 10% (150 samples) from the training data as development set. The official ranking of submitted system for this task is based on Mean Average Precision (MAP) and Precision@30 (P@30) for English and Arabic datasets, respectively.

To train the SVM models for both English and Arabic, we perform grid search over PCA energy (%) conservation, regularization parameter C and RBF kernel’s gamma. Parameters range for PCA varies from 100% (original features) to 95% with decrements of 1, and both C and gamma vary between -3 to 3 on a log-scale with 30 steps. For faster training on a large grid search, we use ThunderSVM [55] which takes advantage of a GPU or a multi-core system to speed up SVM training.

Our submissions used the best models that we obtained from the grid search and are briefly discussed below.

English: We made 3 submissions in total. Our primary (Run-1) and $2^{nd}$ contrastive (Run-3) submission uses sentence embeddings computed from BERT-large word embeddings as discussed in the proposed work section. In addition, both submissions use POS tag and dependency relation features. Interestingly, we found that the best performing sentence embeddings did not include stop-words. The primary submission (Run-1) uses an ensemble of predictions from three models trained on concatenated last 4 hidden layers, average of last 4 hidden layers and $2{nd}$ last hidden layer. The $2^{nd}$ contrastive submission (Run-3) uses predictions from the model trained on the best performing sentence embedding computed from concatenating last 4 hidden layers. Our $1^{st}$ contrastive submission (Run-2) uses an ensemble of predictions from three models trained with GloVe[37] on Twitter with 25, 50 and 100-dimensional embeddings but with the same POS tag and dependency relation features. We use majority voting to get the final prediction and mean of decision values to get the final decision value. We found that removing the stop-words to compute average of word embeddings actually degraded the performance and hence included them in the average.

We also add some additional results to see the effect of stop-words, POS tags, named entities, dependency relations and ensemble predictions in Table 3. The effect of stop-words can be clearly seen in alternative runs of Run-1 and Run-3, where the MAP clearly drops by 1-2 points. Similarly, the negative effect of removing POS tag and dependency relation features can be seen in rest of the alternative runs. Lastly, adding named entity features to the original submissions also decreases the precision by 1-2 points. This might be because the tweets have very few named entities and are not useful to distinguish between check-worthy and not check-worthy claims. For comparison with other teams in the challenge, we show top 3 results at the bottom of the table for reference. Team Accenture [56] fine-tuned a RoBERTa model with an extra mean pooling and a dropout layer to prevent overfitting. Team Alex [34] experimented with different tweet pre-processing techniques and various transformer models together with logistic regression and SVM. Their main submission used logistic regression trained on 5-fold predictions from RoBERTa concatenated with tweet metadata. Team QMUL-SDS [1] fine-tuned a BERT model pre-trained specifically on COVID twitter data.

Arabic There are a total of four submissions that we made in this task. Our best performing submission (Run-1) uses 100-dimensional word2vec Arabic embeddings trained on a Twitter corpus [49] in combination with POS tag features. Our second and third submissions are redundant in terms of feature use, so we only mention the second one (Run-2) here. In addition features used in first submission, it uses dependency relation features and 300-dimensional Twitter embeddings instead of 100-dimensional. Our last submission (Run-3) uses only pre-trained multilingual sentence-transformer⁵⁵5https://github.com/UKPLab/sentence-transformers [41] that is trained on 10 languages including Arabic. In the first three submissions, we removed the stop-words from all the features as keeping them resulted in a poorer performance. Precision@K and Average Precision (AP) results on the test set are shown in the same order in Table 4. Official metric for ranking is P@30. For comparison with other teams in the challenge,we show top 3 results at the bottom of the table for reference. Team Accenture [56] experimented with and fine-tuned three different pre-trained Arabic BERT models and used external data to increase the positive instances. Team TOBB-ETU [26] used logistic regression and experimented with Arabic BERT and word embeddings together to classify tweets. Team UB_ET [18] used a multilingual BERT for ranking tweets by check-worthiness.

The dataset in this task has 1,003 tweets for training and 200 tweets for testing. These tweets are to be matched against a set 10,373 verified claims. From the training set, 197 tweets are kept for validation. To fine-tune the sentence-transformer network with the triplet loss, we use a batch size of eight and train the network for two epochs. The official ranking of this is based on Mean Average Precision@5 (MAP@5). All tweets and verified claims are in English.

Our primary (Run-1) and $2^{nd}$ contrastive (Run-3) submission uses BERT-base and BERT-large pre-trained on SNLI dataset with sentence embedding pooled from the CLS and MAX tokens respectively. We fine-tune these two networks with the triplet loss. On the contrary, our $1^{st}$ contrastive submission (Run-2) uses multilingual DistilBERT model [41] trained on 10 languages including English. This model is directly used to test the pre-trained embeddings.

Interestingly, pre-trained embeddings extracted from multilingual DistilBERT without any fine-tuning turn out to be better for semantic similarity than fine-tuned monolingual BERT models. Having said that, the fine-tuned monolingual BERT models do perform better than extracted pre-trained embeddings and the difference can be seen in Run-1-2 and Run-3-2 in Table 5. We also try to fine-tune the multilingual model which drastically decreases the retrieval performance. The decrease can be attributed to the pre-training process [41] in which the model was trained in a teacher-student knowledge distillation learning framework and on multiple languages. As stated in the proposed work section, we conduct a second evaluation to retrieve the claims with highest similarity without KD-Search and the results are significantly better as shown in Table 5. For comparison with other teams in the challenge, we have shown top 3 primary submissions at the bottom of the table for reference. Team Buster.AI [4] investigated sentence similarity using transformer models, and experimented with multimodality and data augmentation. Team UNIPI-NLE [36] fine-tuned a sentence-BERT in two steps, first to predict the cosine similarity of positive and negative pairs, followed by a binary classification of whether a tweet-claim pair is a correct match or not. Team UB_ET [53] experimented with three different models to rank the verified claims and their main submission used a DPH Divergence from Randomness (DFR) term weighting model.