CommunityLM: Probing Partisan Worldviews from Language Models

Political polarization is a prominent component of politics in the United States Poole and Rosenthal (1984); McCarty et al. (2016); Heltzel and Laurin (2020). Previous studies have shown growing polarization in social media Bail et al. (2018); Demszky et al. (2019); Darwish (2019) and substantial partisan and ideological differences in media diet Bozell (2004); Gil de Zúñiga et al. (2012); Hyun and Moon (2016). Li et al. (2017) show that partisanship makes reliable predictions about an individual’s word understanding. R. KhudaBukhsh et al. (2021) used modern machine-translation techniques to demonstrate that the left and right communities use English words differently. Milbauer et al. (2021) extended the method to uncover worldview and ideological differences between 32 Reddit communities. These studies are word-level analyses based on Word2vec word embeddings, and none of them use pre-trained language models.

Prompting is a standard technique to make pre-trained language models generate texts conditioned on prompts. Recent work has shown that, through prompt engineering, pre-trained language models can achieve good zero-shot performance on NLP tasks from sentiment classification to reading comprehension Radford et al. (2019); Brown et al. (2020) and mine factual or commonsense knowledge Petroni et al. (2019); Davison et al. (2019); Jiang et al. (2021); Talmor et al. (2020). Through prompting, Palakodety et al. (2020) used a fine-tuned BERT Devlin et al. (2019) model with fill-in-the-blank cloze statements to mine insights and compare prediction differences between Indian regional and national YouTube news channels. Feldman et al. (2021) fine-tuned GPT-2 on COVID-19 tweet corpora to mine user opinions.

However, none of these studies fine-tune GPT-style language models on community data to probe community worldviews. In this work, we focus on Republican and Democratic Twitter communities and conduct a feasibility study using fine-tuned GPT-2 partisan language models to generate community responses and to predict community stance. As exemplified in Table 1, we observe clear partisan differences. In this sociopolitically fragmented society, our motivation is to provide a simple and flexible interface for people to probe each other’s worldviews on topics of interest and to encourage constructive dialogue. We demonstrate through our experiments and analyses that the proposed method is a reliable tool to probe community opinions. The contribution of the work is as follows:

• •

  We present a simple CommunityLM framework based on GPT-2 language models to mine community insights by fine-tuning or training the model on community data. This study focuses on Democrat and Republican communities on Twitter but can be easily extended to probe insights from any community based on their public discourse or media diet¹¹1The source code of our paper is available at: https://github.com/hjian42/CommunityLM.

• •

  We use ANES questions as prompts and find that GPT-generated opinions are predictive of community stance towards public figures and groups. We experiment with 4 types of prompts and find that the fine-tuned CommunityLM with an “X is the” prompt outperforms all the baselines (including pre-trained GPT-3 Curie) in predicting community stance.

• •

  We analyze the errors made by community language models and demonstrate the capability of the models to probe community preferences towards public figures by ranking.

We construct a Twitter dataset containing 4.7M tweets (100M word tokens) by Republican and Democrat communities respectively. We first sampled 1M active U.S. Twitter users before and after the 2020 presidential election. We adapt the standard method Volkova et al. (2014); Demszky et al. (2019) to estimate their political affiliation from the political accounts they follow and collect tweets of Republican and Democratic Twitter users from 2019-01-01 to 2020-04-10. We pick this period because the ANES 2020 survey was collected between 2020-04-10 to 2020-04-18 and we want to ensure the Twitter training data does not leak information beyond 2020-04-10. We subsample 4.7M tweets from each side to achieve a balanced set and use Nguyen et al. (2020)’s tweet tokenizer for data processing. Details are described as follows.

U.S. Twitter User Sampling. We first sample a subset of active Twitter users from the “decahose”, Twitter’s 10% sample of tweets. We define active U.S. users as those who posted at least 10 original tweets before and after the 2020 presidential election period (2020-07-01 to 2021-06-31). We then use Litecoder²²2https://github.com/social-machines/litecoder to extract user locations from their profile location strings and filter out users not based in the U.S. We construct the follow graph of the resulting set of 1,074,650 Twitter users.

Partisan Assignment. We follow previous studies Volkova et al. (2014); Demszky et al. (2019) to estimate the party affiliation of Twitter users from the political accounts they follow. Specifically, we update the list of Twitter handles of US politicians from Demszky et al. (2019) by adding current federal officeholders and governors from Ballotpedia³³3We update American politicians from https://ballotpedia.org/List_of_current_members_of_the_U.S._Congress and https://ballotpedia.org/Governor_(state_executive_office). The final list has 457 Republican and 473 Democratic politician Twitter handles. To identify committed partisan users, we adopt the following rules: a user is labeled as a Democrat if they followed no fewer than 6 Democratic politicians and no Republican politician from the list in February 2022, whereas a person is labeled as a Republican if they followed no fewer than 2 Republican politicians and no Democratic politicians. We choose these thresholds because there are 69% Democratic users and 26% Republican users on Twitter⁴⁴4https://www.pewresearch.org/politics/2020/10/15/differences-in-how-democrats-and-republicans-behave-on-twitter (2.65:1). This step predicts 182,788 Democratic-leaning and 72,186 Republican-leaning users (2.53:1).

Tweet Pre-processing. We use the tweet tokenizer from Nguyen et al. (2020) to process all the data. This tokenizer converts user mentions and web/url links into special tokens @USER and HTTPURL. We delete HTTPURL from the tweets because it does not contain useful community information. We do not lower the case but filter out tweets with less than 10 tokens, producing 7,554,409 Democratic and 4,759,441 Republican tweets. We randomly sample from Democratic tweets to ensure both partisan communities have the same number of 4,759,441 tweets for training language models to ensure a fair community model comparison.

We present a simple CommunityLM framework which adapts GPT-style language models to mine community insights. This framework consists of four steps: (1) fine-tune or train GPT language models on community data, (2) design prompts based on survey questions, (3) generate community responses with language models, (4) aggregate community stance based on responses.

Task Formulation. The ANES survey has self-reported party affiliation from participants. We use responses from Republican and Democratic participants and calculate their average ratings towards each of 30 items (persons and groups). These average ratings are provided in Appendix B. If the average rating of Republican participants is higher than that of Democratic participants toward one item (e.g., Joe Biden), it is labeled as “R”. Otherwise, the item is labeled as “D”. 70% items are labeled “D” and 30% “R”. The 9 items with “R” label are Donald Trump, Mike Pence, Marco Rubio, Nikki Haley, Clarence Thomas, whites, capitalists, big business, and the Republican Party. The task asks a model to predict which community is more favorable towards an item. To address the data imbalance, we prefer weighted F1 to accuracy as a measure of model performance.

Baselines. We evaluate the performance of trained and fine-tuned CommunityLM (GPT-2) against 4 baselines. The first baseline is Frequency Model which counts the frequency of an item’s name in each community’s data and classifies the community with higher word frequency to be the label. The second baseline is Keyword Retrieval which uses keywords to retrieve tweets containing the keywords from each community’s data, computes the average community stance, and selects the community with a higher stance score. Keyword Retrieval (full) means using the full names as keywords and Keyword Retrieval (surname) means using the surname of people. The third and fourth baselines use pre-trained GPT-2 and pre-trained GPT-3 Curie respectively. “[CONTEXT]” is a preceding context “As a Democrat/Republican, I think”, which is concatenated with the prompts to generate partisan responses on each item. We compute the average community stance on 1000 synthetic responses and pick the community with a higher average stance score. It is noted that we also fine-tune or train GPT-2 on the aggregate partisan tweets and show their results in Appendix D.

Overall Performance. First, we observe that fine-tuned CommunityLM with “X is the” prompt achieves the best performance in both accuracy (97.33%) and weighted F1-score (97.29%) on the task. The same model’s performance is sensitive to the prompt design and the longest prompt out of the four seems to work the best. “X” alone is bad, because it will result in many responses like “X @USER”, “X???”, “X.”, which are common Twitter posts and are too short to interpret their attitudes. Second, fine-tuned CommunityLM outperforms trained CommunityLM from scratch. It indicates that pre-training GPT-2 is helpful, probably because pre-training injects the general knowledge about the named entities into GPT-2. Third, we find that Keyword Retrieval (surname) is a strong baseline in both accuracy (93.33%) and F1 (93.33%), but its performance is also sensitive to the selection of keywords. As we see, the weighted F1 performance of Keyword Retrieval (full), which uses a strict full name matching (e.g., “Joe Biden”), drops to 87.00%. In contrast, language models are able to learn the associations between different names for the same person and generalize without worrying about name forms. Last, fine-tuned CommunityLM outperforms pre-trained GPT-2 and GPT-3 baselines. It is worth noting that the performance of pre-trained GPT-3 Curie is consistently better than pre-trained GPT-2. GPT-3 with the “X is/are” prompt achieves the same score as the Keyword Retrieval (surname) baseline.

Error Analysis. The rule-based Keyword Retrieval (surname) baseline misses “illegal immigrants” and “big business”. The fine-tuned CommunityLM with “X is/are the” misses “White people”. The pre-trained GPT-3 with “X is/are” prompt misses “Dr. Anthony Fauci” and “Asian people’. It is interesting because the top 5 items with the closest average rating gap between ANES partisan participants are Asian people (5.5%), White people (5.9%), Hispanic people (7.7%), Dr. Anthony Fauci (8.4%), and Black people (9.7%).

Ranking Public Figures. We use the average community stance scores computed on the generated tweets from the fine-tuned CommunityLM model to rank 16 public figures for each community, hoping to understand how they perceive these people. In Figure 1, we observe that Republican politicians are rated poorly by the Democratic model and vice versa. Overall, the ratings from the Republican model are more negative than the Democratic model. Interestingly, we find that Andrew Yang is rated quite highly by both models, likely because of the sampling bias of Twitter. It is noted that “Andrew Yang” is also ranked 1st by the Democrat community and 3rd by the Republican community with the retrieval approach.

In this paper, we present a simple CommunityLM framework to evaluate the viability of fine-tuned GPT-2 community language models in mining community insights in the context of political polarization between Republicans and Democrats. We adopt ANES survey questions and experiment with four types of prompts to generate community responses through GPT-2, showing that generated opinions are predictive about which community is more favorable towards selected public figures and groups. Our results show that fine-tuned CommunityLM (GPT-2) outperforms the baseline methods. We analyze the model errors and run qualitative analyses to demonstrate that GPT-2 community language models can be used to rank public figures and probe word choices.

There are a few limitations in the current approach. First, language models can synthesize unreliable responses. Structured knowledge Wang et al. (2021); Yasunaga et al. (2021) can be used to reduce nonsensical or unfaithful generation. Therefore, it is important that we use statistical patterns rather than individual synthesized tweets to draw conclusions Feldman et al. (2021). Second, language models are shown to be sensitive to prompt design in our experiments and are also vulnerable to negation and misprimed probes Kassner and Schütze (2020). In the future, we plan to develop a systematic approach to design effective prompts and evaluate the robustness of CommunityLM. Third, we focus on the classic red and blue polarization and do not consider a more fine-grained segmentation of U.S. politics. We hope to extend this work to study multiple sociopolitical communities in America and surface their unheard voices.

We propose a general framework to probe community insights and observe differences between the Democratic and Republican communities on Twitter. While we do not discuss how to react to these findings, the intention of our research is encourage people to escape from their echo chambers, hear voices from other communities, and engage in constructive communication. One reasonable ethical concern is that by using a language model to predict community opinions, instead of asking individuals from the community directly, don’t we risk erasing individual voices? To that concern we would like to emphasize that our model is no substitute for deeper engagement with a community; as discussed in the limitation paragraph, the language model is just an entry point for understanding a community’s perspective. It serves to synthesize the points expressed by the speakers in the training data more effectively than we know how to do by hand. Any automated or semi-automated prediction system risks misinterpreting or “erasing” an expressed opinion, and we show in our work that the simpler methods of doing so are more error-prone, and hence measurably more unfair than the proposed approach in the paper.

We would like to thank anonymous reviewers for their helpful comments on our paper. We also want to thank Belén Carolina Saldías Fuentes, Will Brannon, Suyash Fulay, Wonjune Kang, Hope Schroeder, and Shayne Longpre for their discussion and feedback in the early stages of the project.