Understanding Political Polarization via Jointly Modeling Users, Connections and Multimodal Contents on Heterogeneous Graphs

In the graph $G=(V,E,X)$ of our study, $G$ denotes the graph, $V$ denotes the set of nodes, $E$ denotes the set of edges, and $X$ represents the attributes of a node. There are two types of nodes including users $V_{user}\subset V$ and tweets $V_{tweet}\subset V\>(V_{user}\cup V_{tweet}=V,V_{user}\cap V_{tweet}=\varnothing)$ as shown in Figure 1. An edge $e\in E$ between a user node $v_{user}\in V_{user}$ and a tweet node $v_{tweet}\in V_{tweet}$ is created if this user interacts with this tweet. In our study, the interactions between a user and a tweet include posting, retweeting and quoting. Since we do not consider the user-user following relations, there is no edge between two user nodes. The retweet and quote relations are represented as a star topology where the tweet that is retweeted/quoted by multiple users serves as the hub of the topology and is connected to these users. No edge is drawn between tweet nodes as well. As a result, the general heterogeneous network is effectively reduced to a bipartite heterogeneous graph. The attributes $X_{user}\subset X$ of the user nodes are user characteristics including the number of followers, friends, listed memberships, statuses, favorites, verification status, as well as the profile description. These publicly available user characteristics are crawled during the data collection. The attributes $X_{tweet}\subset X$ of the tweet nodes have two components: (1) multimodal contents $M_{tweet}$ and (2) the user characteristics of the original author $X_{author}$  ($X_{tweet}=\{M_{tweet},X_{user_{author}}\}$). Multimodal contents such as text including hashtags, and image (if any) are used. The user characteristics of the original author of each tweet node are also used to embed the tweet node. In this way, the user-tweet relations are modeled explicitly as each tweet is associated with a vector indicating the information of the original author.

Textual and visual feature embedding methods are employed to capture the semantic meanings of the user and tweet nodes. We use the state-of-the-art framework - Sentence-BERT (Reimers and Gurevych 2019), to represent the user profile description and tweet text content. Similar to word embeddings (Mikolov et al. 2013), it converts sentences into a vector space where the vectors of two semantically similar sentences will be closer. The output vectors have 384 dimensions. Inspired by Zhang et al. (2018), an image is encoded into a feature vector space using ResNet-50 (He et al. 2016), which is pre-trained on the ImageNet dataset (Krizhevsky, Sutskever, and Hinton 2012). The 2048-dimensional image features from the last pooling layer are extracted. In addition, if a tweet is associated with videos, they can be mapped into the feature space via different feature embedding methods. The framework is the same. In our study, we focus on text and image.

To capture the heterogeneous attributes or contents associated with each node, we adopt the Heterogeneous Graph Neural Network (HetGNN) (Zhang et al. 2019) which first uses random walks with restart to generate neighbors for nodes and capture the structural information, and then leverages bi-directional LSTM (Bi-LSTM) (Hochreiter and Schmidhuber 1997) and the attention mechanism (Veličković et al. 2018) to aggregate node features within each type and among types. Figure 2 illustrates our framework.

We first fuse heterogeneous attributes $X_{v}$ of node $v\in V$ via a neural network $f_{1}$. In our study, the attributes $X_{user_{u}}$ of the user node $v_{user_{u}}\in V_{user}$ include the number of followers and profile description, etc. The attributes $X_{tweet_{t}}$ of the tweet node $v_{tweet_{t}}\in V_{tweet}$ include both the multimodal contents $M_{tweet_{t}}$ and the user characteristics $X_{user_{author}}$ of the author who originally posts this tweet ($X_{tweet_{t}}=M_{tweet_{t}}\cup X_{user_{author}}$). Let $\boldsymbol{x_{i}}\in\mathbb{R}^{d_{f}\times 1}$ denote the i-th attribute in $X_{v}$ where $d_{f}$ is the feature dimension. The multimodal information can be obtained using different feature embedding methods as aforementioned. Therefore, the attribute feature embeddings $f_{1}(v)\in\mathbb{R}^{d\times 1}$ ($d$: attribute embedding dimension) of $v$ are calculated as follows:

$$f_{1}(v)=\frac{\sum_{i\in X_{v}}[\overrightarrow{LSTM}\{h(\boldsymbol{x_{i}})\}\oplus\overleftarrow{LSTM}\{h(\boldsymbol{x_{i}})\}]}{|X_{v}|}$$

(1)

It is noteworthy that $h$ denotes a feature transformer which can be identity or fully connected neural network, etc.

Next, we aggregate the neighbor information of node $v$ via two steps: (1) using Bi-LSTM to aggregate embeddings of the neighbor nodes of the same type, and (2) aggregating the embeddings of the neighbor nodes of different types through the attention mechanism. By employing the random walk with restart strategy, for each node $v\in V$, we generate the user-type sampled neighbor set $N_{user}(v)$ and the tweet-type sampled neighbor set $N_{tweet}(v)$. The aggregated t-type neighbor embeddings $f_{2}^{t}(v)\in\mathbb{R}^{d\times 1},t\in\{user,tweet\}$ ($d$: aggregated attribute embedding dimension) are then calculated as follows:

$$f_{2}^{t}(v)=\frac{\sum_{v^{\prime}\in N_{t}(v)}[\overrightarrow{LSTM}\{f_{1}(v^{\prime})\}\oplus\overleftarrow{LSTM}\{f_{1}(v^{\prime})\}]}{|N_{t}(v)|}$$

(2)

where $v^{\prime}$ denotes the neighbor nodes in the t-type sampled neighbor set $N_{t}(v)$. The output embeddings $\mathcal{E}_{v}\in\mathbb{R}^{d\times 1}$ of node $v$ are computed as follows:

$$\mathcal{E}_{v}=\alpha_{v,v}f_{1}(v)+\alpha_{v,user}f_{2}^{user}(v)+\alpha_{v,tweet}f_{2}^{tweet}(v)$$

(3)

where $\alpha_{v,\cdot}$ represents the attention coefficients of (1) content embeddings of $v$ or (2) the aggregated neighbor embeddings to node $v$. These two sets of embeddings are grouped and denoted as $F(v,i)\in\mathbb{R}^{d\times 1}$ which is defined as follows:

$$F(v,i)=\begin{cases}f_{1}(v)&i=v\\ f_{2}^{t}(v)&i=t\text{, where }t\in\{user,tweet\}\end{cases}$$

(4)

$$i\in\boldsymbol{P}\text{, where }\boldsymbol{P}=\{v\}\cup\{user,tweet\}$$

(5)

Therefore, Eq. (3) is re-formulated as:

$$\mathcal{E}_{v}=\sum_{i\in\boldsymbol{P}}\alpha_{v,i}F(v,i)$$

(6)

The attention coefficients $\alpha_{v,i}$ are built by a single-layer feedforward neural network parametrized by a weight vector $\boldsymbol{u}\in\mathbb{R}^{2d\times 1}$, with the LeakyReLU nonlinearity. To make the coefficients comparable, we normalize them across all choices of $i$ using the softmax function. Thus, the coefficients are expressed as:

$$\alpha_{v,i}=\frac{\exp(\text{LeakyReLU}(\boldsymbol{u^{T}}[F(v,v)\oplus F(v,i)]))}{\sum_{j\in\boldsymbol{P}}\exp(\text{LeakyReLU}(\boldsymbol{u^{T}}[F(v,v)\oplus F(v,j)]))}$$

(7)

where $\boldsymbol{\cdot^{T}}$ represents transposition and $\oplus$ is concatenation.

We do not require any ideology-related “supervision” (i.e., ideology labels) during the training process. The goal of the representation learning of our framework is to maximize the similarity between the embeddings of two nodes if they have similar attribute features and/or are geometrically closer to each other in the graph, otherwise, minimize the similarity. Therefore, we apply the negative sampling technique (Mikolov et al. 2013). The goal can be interpreted as, for each $v\in V$, to minimize the cross entropy loss as follows:

$$log\sigma(\mathcal{E}_{v_{pos}}\cdot\mathcal{E}_{v})+log\sigma(-\mathcal{E}_{v_{neg}}\cdot\mathcal{E}_{v})$$

(8)

where $\sigma$ represents the Sigmoid function, $v_{pos}$ and $v_{neg}$ are the positive and negative sample nodes. Zhang et al. (2019) defines the positive sample nodes of the node $v$ as the nodes that can be reached by node $v$ via a random walk, and the negative sample nodes of the node $v$ as any nodes in the graph. Inspired by the findings of Ying et al. (2018) that importance-based neighborhoods can help improve the representation learning of GNNs, we propose a social-platform based negative sampling strategy. We call it social-platform based because it is based on our observation of the online social platforms, in this case, Twitter. We attempts to measure the likelihood of a user interacting with a tweet. If the likelihood is high, but the user never interacts with the tweet, then this is considered as an important negative pair. The social-platform based negative sampling strategy is designed to capture this kind of negative pairs.

More specifically, we sample the nodes based on the social capital features. This component is only considered when the node type of the negative sample is $user$. If the node type of the negative sample is $tweet$, we apply random sampling. The higher the normalized number of statuses a user has posted since the creation of the Twitter account, the more chance the user will be sampled. The motivation is that the number of statuses indicates the probability of a user’s posting/retweeting/quoting behavior. In other words, it indicates the probability of a user interacting with a tweet.

We acknowledge that this strategy has limitations. For example, we aggregate the number of statuses of each user since the creation of the Twitter account. This implicitly assumes that the user behavior does not change significantly over time. Although it would be better to measure the user behavior in a more fine-grained and more dynamic fashion, it is beyond the scope of this study and left to future work.

Similarly to Zhang et al. (2019), the dimension of the final feature space is set to be 128. For random walk, the walk length is 30 and window size is 5.

While there are other feature embedding methods and heterogeneous graph representation learning methods that could potentially improve the expressive power of the overall representation (Lv et al. 2021), our ultimate goal is not to perfectly characterize the users, tweet contents and relations. Instead, we aim to show that jointly modeling the information in a bipartite heterogeneous graph can benefit the ideology detection task and help understand online political polarization.

To understand the differences among the users of Left, Middle, and Right, we visualize the user descriptions and the multimodal post contents. Figures 4a and  5a show the word clouds of the user descriptions of the users in each group of the inflation and vaccine datasets. Interestingly, “love” is observed in the descriptions of all three groups across two datasets. The words of Left and Right are associated with the political affiliation. For instance, there are “blm”, “resist”, and “democrat” in Left, while “maga”, “conservative”, “trump”, and “chirstian” in Right. Left-leaning users are found to be more engaged in online civil right movements such as “#BlackLivesMatter” (Panda, Siddarth, and Pal 2020) and “#StopAsianHate” (Lyu et al. 2021). “resist” represents the American liberal political movement that protested the presidency of Trump.³³3https://en.wikipedia.org/wiki/The˙Resistance˙(American˙political˙movement) [Accessed 2022-01-09] “Make America Great Again” or “MAGA” is a campaign slogan popularized by Trump.⁴⁴4https://en.wikipedia.org/wiki/Make˙America˙Great˙Again [Accessed 2022-01-09] With respect to “christian”, Pew Research Center (2015) found there are proportionally more Christians among conservatives than among Democrats and Democratic leaners.

Figures 4b and 5b show the word clouds of the most popular topics.⁵⁵5We describe how we determine the most popular topics in Appendix. We observe that users of different clusters focus on different topics. Take the inflation dataset as an example, we find that “gas” and “crisis” in Right are related to the tweets that express negative opinions against President Joe Biden⁶⁶6The tweet has been paraphrased to protect the user privacy.:

“These are what #BidenDelivers: Soaring gas prices, inflation, millions of unfilled jobs, Crisis in the Middle East, Border issue, Woke Military, Increasing crime, Supply chain issue, Critical Race Theory, Taxes, Refugee resettlement, IRS expansion.”

Note that given the limited number of unique images in the vaccine dataset, we only conduct analysis on the images of the inflation dataset. The images that are related to Left, Middle and Right are in general different. Figure 4c shows the representative figures.⁷⁷7We discuss how the representative figures are selected in Appendix. Almost all groups post images of political figures (e.g., Joe Biden, Donald Trump). In Left (lower left) and Middle (upper right), we observe more portraits, while in Right (upper left) there are more sarcasm images of Joe Biden. Interestingly, users of Right share more eye-catching images. We find users in Left share screenshots of long paragraphs and charts that explain the reasons for inflation (upper left and right in Left). Users in Middle use images of items that can reflect inflation (lower right in Middle) more often. However, users in Right use images that intend to depict inflation more vividly (lower left in Right).

We calculate the number of users per unique tweet in each group. This value indicates the number of retweet/quote activities. The higher this value is, the more users retweet/quote the same tweet, suggesting more retweet/quote activities. There are more retweet/quote activities in the Left cluster (4.58/1.73 users per unique tweet in inflation/vaccine) and Right cluster (2.90/1.90 users per unique tweet in inflation/vaccine) than in the Middle cluster (1.03/1.02 users per unique tweet in inflation/vaccine). Fewer retweet/quote activities indicate a sparser network on which GNN normally performs poorly due to the limited knowledge gain from less representative neighbors (Jia et al. 2020).

For the inflation dataset, we also observe two sub-clusters composed of left-leaning users (highlighted in Areas 4 and 5 in Figure 3a). By analyzing the user characteristics and the multimodal contents, we find that it is mainly because of the difference in user descriptions. Unlike previous studies (Waller and Anderson 2021; Conover et al. 2011b) that focus on the political polarization in terms of two groups (left and right), by combining information of users, contents, and relations, we are able to conduct a more fine-grained analysis and provide insights into political polarization guided by the clusters found by our ideology-agnostic representation learning framework (i.e., no political labels during training). In our case, the three clusters are different regarding user descriptions, multimodal contents and retweet/quote activities.