Recurrent Graph Neural Networks for Rumor Detection in Online Forums

As online social media becomes increasingly present in peoples’ daily lives, greater proportions of users get news and journalistic content directly from their ”feed” on accounts like Facebook, Twitter, YouTube, and Reddit. Pew research reported that social media outpaced print news as a news source in the United States in 2018 (Shearer, 2018). Some social media platforms are also news-centric – for instance, a 2016 Pew study found that seven out of ten Reddit users use the platform to get their news (Barthel et al., 2016). Following these findings, the seminal work (Vosoughi et al., 2018) showed that user interaction signals – differential patterns of liking, re-sharing, and commenting – distinguish between posts that link to certain categories of online content, in particular content later identified as ”rumors” versus other content.

Due to these phenomena, a recent sub-field of applied machine-learning research has grown, focusing on graph-based artificial intelligence models for classifying links and content shared on social media, particularly for rumor detection (Bondielli and Marcelloni, 2019). Most rumor detection models recently introduced have been tuned for and evaluated on data from online social networks like Twitter or Facebook (e.g. Wu and Liu, 2018; Rosenfeld et al., 2020). These platforms have a natural social graph created by users, which provides an inherent graph on which a Graph Neural Network can propagate rumor information. However, relatively less attention in rumor detection research has been given to online forums like Reddit. Our work addresses two nuances specific to forums. First, most forums do not have a natural who-follows-who social graph. Second, most forums do not feature a ”repost” option on their platform, preventing usual inter-user cascades seen in social networks (Vosoughi et al., 2018). Instead, each article in a forum is posted a limited number of times, each time independently by users across the platform. This means that each forum post (unlike social network posts) is the start of its own discussion cascade consisting of a long comment-tree graph, as opposed to repost/share behavior found on social networks.

To address these nuances, we provide a two-fold contribution to this space. First, we illustrate the construction of emergent social networks from online forum data, which we use both for feature learning and the downstream neural network computational graph via a GNN. Second, we introduce a Recurrent-GNN model which can well-handle the independent, sequential nature of article posting on forums. Our approach combines an RNN, to capture time-order of posts, with a Graph Neural Network (GNN), to capture the post comment relations of users. We evaluate our approach on the publicly-available Reddit corpus, testing two classification tasks: topic classification and rumor detection. We address preliminaries in Section 2, detail our methods in Section 3, describe our evaluation experiments in Section 4, and conclude with a brief discussion in Section 5.

News and opinion pieces are being produced with record-breaking volume, and links to these pieces propagate swiftly on social sites like Twitter, Facebook, and Reddit. Formally, a link $m$ is propagated by a sequence of posts $\mathbf{p}=\{p_{1},p_{2},\ldots\}$ having corresponding authors $\mathbf{a}=\{a_{1},a_{2},\ldots\}$. On forum sites like Reddit, for which our main contributions are intended, each post also has a sequence of commenters  $\mathbf{c}=\{c_{1},c_{2},\ldots\}$. Note that $\mathbf{a},\mathbf{c}\subseteq\mathbbm{U}$ where $\mathbb{U}$ is the total user set of the forum. For each task, we pair each message $m$ with a categorical label $y$, and train various models to predict $y$ given feature data associated with any $\mathbf{p}$, $\mathbf{a}$, and $\mathbf{c}$.

Classification tasks having the above general setting, in which a label $y$ is inferred from a some social media information piece $m$, have attracted great attention from researchers in recent times. For instance, (Ott et al., 2013) uses sentiment analysis to distinguish spam comments and trustworthy comments, and (Shu et al., 2017) detect fake news via assessment on credibility of article headlines. However, content-based classification in this domain can be challenging, especially in the context of rumor detection, as pieces with differing labels can nonetheless feature similar topics and writing styles. (Zhou et al., 2019). Because of this, other approaches like Traceminer (Wu and Liu, 2018) and CSI (Ruchansky et al., 2017) have been proposed which learn from user interaction signals and information propagation paths. Similarly, graph embedding methods such as Node2Vec (Grover and Leskovec, 2016), SDNE (Wang et al., 2016) and graph convolutional networks – GCNs – have been widely used for network analysis and graph feature extraction in social network studies. In this work, we propose a framework which combines GCNs (Graph Convolutional Networks (e.g. Kipf and Welling, 2017; Hamilton et al., 2017; Veličković et al., 2017)) and RNNs (Recurrent Neural Networks) to model the information diffusion process of article links.

We note that the pipeline and model we introduce for our Reddit case-study, as with many of the approaches listed above, can be used for any supervised classification tasks featuring labelled posts or article links. When applied to a task like rumor detection, it is crucial to note that a machine-learning model cannot be used to establish or predict an ultimate verdict on whether a piece of content is true, false, or of high/low quality. As detailed in Section 4, we derive a rumor label for an article by its presence on a fact-check site, regardless of verdict. In this case, our hypothesis is simply that user interaction signals in social media data can be correlated, via deep learning, to articles’ potentials to be controversial in the specific sense that they are noticed by a fact-check site. Since we train our model (and baseline models) on fact-check site data, our results are subject to any biases or errors in that data. Nevertheless, as seen in our experimental results, behavioral signals are useful in this regard, and thus our illustration of methods in this space may interest other researchers studying these phenomena.

In this work we introduce a Recurrent Graph Neural Network (R-GNN), illustrated in Figure 1, which models two information diffusion processes on forums. First, a graph convolutional network (GCN) encodes features from commenters under each post. Second, a recurrent neural network (RNN) learns a link representation from the sequence of post encodings. We detail these components and feature construction in the next sections.

In this section we describe the evaluation of our R-GNN model against five baselines on two tasks: article categorization and rumor detection. Four of our baselines are standard (non-neural) machine learning methods applied to simplified features. Our fifth baseline is an established RNN-based method called TraceMiner, which has been previously evaluated on similar tasks using Twitter data (Wu and Liu, 2018). We evaluate two versions of our R-GNN against these baselines. First, we remove the GCN component from our model, concatenating all authors $\mathbf{a}$ and commenter sequences $\{\mathbf{c}_{1},\mathbf{c}_{2},\ldots\}$ associated with a post into a single sequence, which we feed to an RNN. We refer to this version as “R-GNN(-replygraph)”, as it removes the influence of the comment graph signal from the learning process. This provides an ablation study of R-GNN to better evaluate the combination of GCN and RNN components in our proposed approach. Second, we evaluate the full R-GNN as described in Section 3. Finally, we note that all of our experiments implicitly test our hypothesis that a “proximal” friendship graph can be derived from user interactions as a useful signal in these tasks, as described in Section 3. All-told, we design our experiments to answer three main research questions as follows:

• RQ1

  Can signals derived purely from user interactions (absent a natural social graph) be successful in classifying links that are shared in online forums?

• RQ2

  Can diffusion process modeling with deep neural networks outperform standard ML models, applied to online forums?

• RQ3

  Can our RNN+GCN hybrid model outperform simpler RNN-only baselines, especially for rumor detection?

In this work we introduced an approach for rumor detection and (more generally) link classification on forum websites, and evaluated this approach on Reddit data. Our model, a Recurrent Graph Neural Network (RGNN), is able to capture both the diffusion process of each link through post comment-graphs via a GCN, and simultaneously the sequential nature of link-posting on forums via an RNN. When applied to a link topic categorization task, our approach had superior performance to other RNN-based methods, but had comparable performance to non-neural baselines. When applied to a rumor detection task, our approach had superior performance to all baselines. To our knowledge, this is the first appearance of an RGNN in this space, and among the first demonstrations of deep learning on online user interactions without a natural social graph.

Automated rumor detection via artificial intelligence (and more generally, online content categorization) in social networks is a growing area of research, featuring a rich landscape of model architectures and classification tasks. In this short paper we have examined a narrow subset of potential tasks, model architectures, and available features in this space. For instance, to better understand the effect of various interaction-based graph signals – which let the model learn from diffusion processes on the directed graph of user actions – we have disregarded the many content-based signals, e.g. text or images, available for the tasks in our paper and others. However, in doing so, we have shed light on the capacity of state-of-the-art GNNs to model article sharing in forums with interaction-based features alone. This exposes headroom to improve GNN architectures and interaction-based feature construction for classification tasks.