Towards Propagation Uncertainty: Edge-enhanced Bayesian Graph Convolutional Networks for Rumor Detection

Traditional methods on rumor detection adopted machine learning classifiers based on handcrafted features, such as sentiments Castillo et al. (2011), bag of words Enayet and El-Beltagy (2017) and time patterns Ma et al. (2015). Based on salient features of rumors spreading, Wu et al. (2015); Ma et al. (2017) modeled propagation trees and then used SVM with different kernels to detect rumors.

Recent works have been devoted to deep learning methods. Ma et al. (2016) employed Recurrent Neural Networks (RNN) to sequentially process each timestep in the rumor propagation sequence. To improve it, many researchers captured more long-range dependency via attention mechanisms Chen et al. (2018), convolutional neural networks Yu et al. (2017); Chen et al. (2019), and Transformer Khoo et al. (2020). However, most of them focused on learning temporal features alone, ignoring the internal topology structure.

To capture topological-structural features, Ma et al. (2018) presented two recursive neural network (RvNN) based on bottom-up and top-down propagation trees. Yuan et al. (2019); Lu and Li (2020); Nguyen et al. (2020) formulated the propagation structure as graphs. Inspired by Graph Convolutional Network (GCN) Kipf and Welling (2017), Bian et al. (2020) first applied two GCNs based on the propagation and dispersion graphs. Wei et al. (2019) jointly modeled the structural property by GCN and the temporal evolution by RNN.

However, most of them treat the edge as the reliable topology connection for message-passing. Ignoring the uncertainty caused by unreliable relations could lead to lacking robustness and make it risky for rumor detection. Inspired by valuable research Zhang et al. (2019a) that modeled uncertainty caused by finite available textual contents, this paper makes the first attempt to consider the uncertainty caused by unreliable relations in the propagation structure for rumor detection.

Graph Neural Networks (GNNs) Kipf and Welling (2017); Schlichtkrull et al. (2018); Velickovic et al. (2018) have demonstrated remarkable performance in modeling structured data in a wide variety of fields, e.g., text classifcation Yao et al. (2019), recommendation system Wu et al. (2019) and emotion recognition Ghosal et al. (2019). Although promising, they have limited capability to handle uncertainty in the graph structure. While the graphs employed in real-world applications are themselves derived from noisy data or modeling assumptions. To alleviate this issue, some valuable works Luo et al. (2020); Zhang et al. (2019b) provide an approach for incorporating uncertain graph information by exploiting a Bayesian framework Maddox et al. (2019). Inspired by them, this paper explores the uncertainty in the propagation structure from a probability perspective, to obtain more robust rumor detection results.

The overall architecture of EBGCN is shown in Figure 2. Given the input sample including text contents and its propagation structure, we first formulate the propagation structure as directed graphs with two opposite directions, i.e., a top-down propagation graph and a bottom-up dispersion graph. Text contents are embedded by the text embedding layer. After that, we iteratively capture rich structural characteristics via two main components, node update module, and edge inference module. Then, we aggregate node embeddings to generate graph embedding and output the label of the claim.

For training, we incorporate unsupervised consistency training on the Bayes by Backprop loss of unlabeled latent relations. Accordingly, we optimize the model by minimizing the weighted sum of the unsupervised loss and supervised loss.

For the supervised learning loss $\mathcal{L}_{c}$, we compute the cross-entropy of the predictions and ground truth distributions $C=\{c_{1},c_{2},...,c_{m}\}$, i.e.,

where $\textbf{y}^{i}$ is a vector representing distribution of ground truth label for the $i$-th claim sample.

For the unsupervised learning loss $\mathcal{L}_{e}$, we amortize the posterior distribution of the classification weight $p(\varphi)$ as $q(\varphi)$ to enable quick prediction at the test stage and learn parameters by minimizing the average expected loss over latent relations, i.e., $\varphi^{*}=\text{arg}\min_{\varphi}\mathcal{L}_{e}$, where

where $\hat{\textbf{r}}$ is the prediction distribution of latent relations. To ensure likelihood tractably, we model the prior distribution of each latent relation $r_{t},t\in[1,T]$ independently. For each relation, we define a factorized Gaussian distribution for each latent relation $q_{\varphi}(\varphi|\textbf{H}^{(l-1)},G;\Theta)$ with means $\mu_{t}$ and variances $\delta^{2}_{t}$ set by the transformation layer,

where $f_{\mu}(\cdot;\theta_{\mu})$ and $f_{\delta}(\cdot;\theta_{\mu})$ refer to compute the mean and variance of input vectors, parameterized by $\theta_{\mu}$ and $\theta_{\delta}$, respectively. Such that amounts to set the weight of each latent relation.

Besides, we also consider the likelihood of latent relations when parameterizing the posterior distribution of prototype vectors. The likelihood of latent relations from the $l$-th layer based on node embeddings can be adaptively computed by,

In this way, the weight of edges can be adaptively adjusted based on the observed graph, which can thus be used to effectively pass messages and learn more discriminative features for rumor detection.

To sum up, in training, we optimize our model EBGCN by minimizing the cross-entropy loss of labeled claims $\mathcal{L}_{c}$ and Bayes by Backprop loss of unlabeled latent relations $\mathcal{L}_{e}$, i.e.,

where $\gamma$ is the trade-off coefficient.

We evaluate the model on three real-world benchmark datasets: Twitter15 Ma et al. (2017), Twitter16 Ma et al. (2017), and PHEME Zubiaga et al. (2016). The statistics are shown in Table 1. Twitter15 and Twitter16²²2https://www.dropbox.com/s/7ewzdrbelpmrnxu/rumdetect2017.zip?dl=0 contain 1,490 and 818 claims, respectively. Each claim is labeled as Non-rumor (NR), False Rumor (F), True Rumor (T), or Unverified Rumor (U). Following Ma et al. (2018); Bian et al. (2020), we randomly split the dataset into five parts and conduct 5-fold cross-validation to obtain robust results. PHEME dataset³³3https://figshare.com/articles/dataset/PHEME_dataset_for_Rumour_Detection_and_Veracity_Classification/6392078 provides 2,402 claims covering nine events and contains three labels, False Rumor (F), True Rumor (T), and Unverified Rumor (U). Following the previous work Wei et al. (2019), we conduct leave-one-event-out cross-validation, i.e., in each fold, one event’s samples are used for testing, and all the rest are used for training.

For Twitter15 and Twitter16, we compare our proposed model with the following methods. DTC Castillo et al. (2011) adopted a decision tree classifier based on information credibility. SVM-TS Ma et al. (2015) leveraged time series to model the chronological variation of social context features via a linear SVM classifier. SVM-TK Ma et al. (2017) applied an SVM classifier with a propagation tree kernel to model the propagation structure of rumors. GRU-RNN Ma et al. (2016) employed RNNs to model the sequential structural features. RvNN Ma et al. (2018) adopted two recursive neural models based on a bottom-up and a top-down propagation tree. StA-PLAN Khoo et al. (2020) employed transformer networks to incorporate long-distance interactions among tweets with propagation tree structure. BiGCN Bian et al. (2020) utilized bi-directional GCNs to model bottom-up propagation and top-down dispersion.

For PHEME, we compare with several representative state-of-the-art baselines. NileTMRG Enayet and El-Beltagy (2017) used linear support vector classification based on bag of words. BranchLSTM Kochkina et al. (2018) decomposed the propagation tree into multiple branches and adopted a shared LSTM to capture structural features. RvNN Ma et al. (2018) consisted of two recursive neural networks to model propagation trees. Hierarchical GCN-RNN Wei et al. (2019) modeled structural property based on GCN and RNN. BiGCN Bian et al. (2020) consisted of propagation and dispersion GCNs to learn structural features from propagation graph.

For Twitter15 and Twitter16, we follow Ma et al. (2018); Bian et al. (2020); Khoo et al. (2020) and evaluate the accuracy (Acc.) over four categories and F1 score ($F_{1}$) on each class. For PHEME, following Enayet and El-Beltagy (2017); Kochkina et al. (2018); Wei et al. (2019), we apply the accuracy (Acc.), macro-averaged F1 (m$F_{1}$) as evaluation metrics. Also, we report the weighted-averaged F1 (w$F_{1}$) because of the imbalanced class problem.

Following comparison baselines, the dimension of hidden vectors in the GCL is set to 64. The number of latent relations $T$ and the coefficient weight $\gamma$ are set to $[1,5]$ and $[0.0,1.0]$, respectively. we train the model via backpropagation and a wildly used stochastic gradient descent named Adam Kingma and Ba (2015). The learning rate is set to $\{0.0002,0.0005,0.02\}$ for Twitter15, Twitter16, and PHEME, respectively. The training process is iterated upon 200 epochs and early stopping Yuan et al. (2007) is applied when the validation loss stops decreasing by $10$ epochs. The optimal set of hyperparameters are determined by testing the performance on the fold-$0$ set of Twitter15 and Twitter16, and the class-balanced charlie hebdo event set of PHEME.

Besides, on PHEME, following Wei et al. (2019), we replace TF-IDF features with word embeddings by skip-gram with negative sampling Mikolov et al. (2013) and set the dimension of textual features to $200$. We implement this variant of BiGCN and EBGCN, denoted as BiGCN(SKP) and EBGCN(SKP), respectively.

For results of baselines, we implement BiGCN according to their public project⁴⁴4https://github.com/TianBian95/BiGCN under the same environment. Other results of baselines are referenced from original papers Khoo et al. (2020); Wei et al. (2019); Ma et al. (2018).

Table 2 shows results of rumor detection on Twitter15, Twitter16, and PHEME datasets. Our proposed model EBGCN obtains the best performance among baselines. Specifically, for Twitter15, EBGCN outperforms state-of-the-art models 2.4% accuracy and 3.7% F1 score of unverified rumor. For Twitter16, our model obtains 3.4% and 6.0% improvements on accuracy and F1 score of non-rumor, respectively. For PHEME, EBGCN significantly outperforms previous work by 40.2% accuracy, 34.7% $\text{m}F_{1}$ , and 18.0% $\text{w}F_{1}$.

Deep learning-based (RvNN, StA-PLAN, BiGCN and EBGCN) outperform conventional methods using hand-crafted features (DTC, SVM-TS), which reveals the superiority of learning high-level representations for detecting rumors.

Moreover, compared with sequence-based models GRU-RNN, and StA-PLAN, EBGCN outperform them. It can attribute that they capture temporal features alone but ignore internal topology structures, which limit the learning of structural features. EBGCN can aggregate neighbor features in the graph to learn rich structural features.

Furthermore, compared with state-of-the-art graph-based BiGCN, EBGCN also obtains better performance. We discuss the fact for two main reasons. First, BiGCN treats relations among tweet nodes as reliable edges, which may introduce inaccurate or irrelevant features. Thereby their performance lacks robustness. EBGCN considers the inherent uncertainty in the propagation structure. In the model, the unreliable relations can be refined in a probability manner, which boosts the bias of express uncertainty. Accordingly, the robustness of detection is enhanced. Second, the edge-wise consistency training framework ensures the consistency between uncertain edges and the current nodes, which is also beneficial to learn more effective structural features for rumor detection.

Besides, EBGCN(SKP) and BiGCN(SKP) outperforms EBGCN and BiGCN that use TF-IDF features in terms of Acc. and $\text{w}F_{1}$. It shows the superiority of word embedding to capture textual features. Our model consistently obtains better performance in different text embedding. It reveals the stability of EBGCN.

In this part, we further evaluate the effects of key components in the proposed model.

Early rumor detection is to detect a rumor at its early stage before it wide-spreads on social media so that one can take appropriate actions earlier. It is especially critical for a real-time rumor detection system. To evaluate the performance on rumor early detection, we follow Ma et al. (2018) and control the detection deadline or tweet count since the source tweet was posted. The earlier the detection deadline or the less the tweet count, the less propagation information can be available.

Figure 4 shows the performance of early rumor detection. First, all models climb as the detection deadline elapses or tweet count increases. Particularly, at each deadline or tweet count, our model EBGCN reaches a relatively high accuracy score than other comparable models.

Second, compared with RvNN that captures temporal features alone and STM-TK based on handcrafted features, the superior performance of EBGCN and BiGCN that explored rich structural features reveals that structural features are more beneficial to the early detection of rumors.

Third, EBGCN obtains better early detection results than BiGCN. It demonstrates that EBGCN can learn more conducive structural features to identify rumors by modeling uncertainty and enhance the robustness for early rumor detection.

Overall, our model not only performs better long-term rumor detection but also boosts the performance of detecting rumors at an early stage.

In this part, we perform the case study to show the existence of uncertainty in the propagation structure and explain why EBGCN performs well. We randomly sample a false rumor from PHEME, as depicted in Figure 5. The tweets are formulated as nodes and relations are modeled as edges in the graph, where node 1 refers to the source tweet and node $2\text{-}8$ refer to the following retweets.

As shown in the left of Figure 5, we observe that tweet 5 is irrelevant with tweet 1 although replying, which reveals the ubiquity of unreliable relations among tweets in the propagation structure and it is reasonable to consider the uncertainty caused by these unreliable relations.

Right of Figure 5 indicates constructed graphs where the color shade indicates the value of edge weights. The darker the color, the greater the edge weight. The existing graph-based models always generate the representation of node 1 by aggregating the information of its all neighbors (node 2, 5, and 6) according to seemingly reliable edges. However, edge between node 1 and 5 would bring noise features and limit the learning of useful features for rumor detection. Our model EBGCN successfully weakens the negative effect of this edge by both the edge inference layer under the ingenious edge-wise consistency training framework. Accordingly, the model is capable of learning more conducive characteristics and enhances the robustness of results.