Quantifying How Hateful Communities Radicalize Online Users

Social media has recently been implicated in a host of societal dysfunctions, such as extremism, polarization, anti-science rhetoric and dissemination of hate. By allowing individuals with particular interests to connect irrespective of physical distances, the web allowed for the creation of countless niches, some of which can be problematic. Reddit (www.reddit.com) is a social media platform that has become one of the most popular meeting places for such interest-based niche groups, where people create thematic forums called “subreddits.”

One negative externality caused by the ease of congregation in the digital world is that users with racist, misogynist, and similarly uncivil views are able to find others just like them and form communities dedicated to those harmful stances. The formation of such groups can then serve as breeding ground for further deplorable opinions, which, through an echo chamber between community members, may become even more extreme and alienated from reality and good-citizenry, potentially including offline hateful behavior.

This study aims to answer the following research question: Does becoming active in a hateful subreddit increase a person’s usage of hate speech outside of that subreddit? Answering that questions will help us understand the impact of hateful communities as originators of hate speech in the platform as a whole.

We create a case-controlled quasi-experimental causal study through Interrupted Time Series (ITS) design and control-treatment user matching. We generate high-precision hate speech lexicons for multiple communities and domains, which are utilized to obtain the measures of hate speech over time. We then find an immediate increase in a user’s hate speech as soon as the user becomes active in the hateful subreddit. The causal nature of ITS models implies that becoming active in a hateful community is the factor leading to an increase in hate speech. We observe a repeated pattern of pre-join ramping up in hate speech, and post-join ramping down in hate speech. These findings help us understand whether halting access to the most extreme communities can break the pathway in which users, despite being hateful enough to join a hate community, get even further radicalized after becoming active in such communities.

Hate Speech in Social Media. While it is difficult to devise a uniform definition of hate speech that is both implementable as a computer program while also satisfying critics [34], it has been shown that the main targets for hate speech are groups based on race, behavior (i.e. insecure people, sensitive people) and physical appearance [29, 34]. Analysis of both directed and generalized hate speech on Twitter has found that directed hate speech is angrier than generalized hate speech, which in turn is angrier than general tweets [12]. One measurement of hate speech in politically fringe digital communities estimates that one-quarter of all posts contain some form of hate speech, with 13.7% of posts containing explicit hate speech and 15.5% containing implicit hate speech [33]. There is a positive association between time spent on Gab (a far-right social media) and hate speech [13]. Further exploration of extremist subreddits reveals how those groups combine the the up-vote / down-vote system, Reddit’s feed algorithm, governance structure, and anonymity, to surface the extremist content within them and promote extreme discourse against opposing groups, while self-validating their views [14, 26, 16].

In-Group and Out-Group Behavior on Social Media. Subreddits are distinctive enough that research has shown it is possible to identify subreddits’ posts based on both their style (86.5% accuracy) and topic (71.1% accuracy) [36]. Moreover, ”the presence of directed political incivility in interactions between dissimilar users tends to discourage further cross-ideological engagement. Conversely, interactions in which ideologically like-minded users engage in derogation of the out-group are less likely to be sustained in the short term” [25]. Contrasting the aforementioned findings, in Facebook and Twitter posts about the political out-group were shared more than posts about the in-group, and the number of terms about the out-group increases the odds of a post being shared [32]. We note group dynamics are not unique to social media, as a singular analog example, research has shown in-group favoritism and out-group derogation among Europeans and Maoris, but for different reasons: ethnic identity for Maoris and social dominance orientation for Europeans [19].

Hate Speech Identification and Prediction. Another direction of research seeks to leverage machine learning to classify and predict hate speech at scale. Successful approaches have leveraged transfer learning [1] and moderation-minded work [17]. Multiple researches have built datasets for hate speech identification [27, 31, 28, 10], in addition to building a dataset, De Gilbert et al. [10] used Pointwise Mutual Information (PMI) comparison between hateful and non-hateful sentences, finding the most hateful words are derogatory or refer to targeted groups of hate speech.

Moderation of Hate Speech. Various moderation approaches could help reduce the amount of hate speech, including the banning of high-profile communities on Reddit [7], as well as employing somewhat softer alternatives to bans, such as quarantines [6, 9]. Quarantines also led a portion of the communities’ members to leave Reddit for more lightly regulated platforms, which merely forwards the issue of handling hate speech to another platform [9]. Those findings corroborate with previous study showing that subreddits influence one another, and thus intervention in one subreddit can be expected to have additional effects on other subreddits, especially those with a large shared user-base [39]. After a subreddit’s ban, top users become less active while most users reduced their community-derived language without a reduction in activity. Moreover explicitly racist communities had a stronger reduction in activity from bans than dark humor communities [37]. Previous research also shows that hateful users differ from regular users, e.g., in their vocabulary as well as engagement [8], hence to the extend top users of hateful subreddits are the most hateful users, this can at least partly explain the diverging fates of the top users in relation to regular members.

The initial analyses show that the users who joined hateful subreddits will increase their usage of hateful speech outside of that subreddit by an average of $\approx 30\%$ immediately after joining. This estimate was made by using the individual coefficients obtained from the ITS model, and the pre/post gap measurement explained in the Model Analysis section under Methods. In more detail, users who joined r/CoonTown show the smallest increase in hate speech outside of r/CoonTown with just $3\%$ average increase, while the users who joined other three subreddits display substantially larger increase in hate speech usage with $38\%$, $40\%$, and $30\%$ for r/fatpeoplehate, r/GreatApes and r/Incels respectively. The relative increase in hate speech immediately after users join the hateful subreddit is illustrated in Figure 1. Values are measured as the percentage increase of the users’ hate speech immediately post-join compared to their immediately pre-join hate speech levels.

In addition to the immediate increase measured, there are further insights to be observed from plotting the results of the ITS model over time, as shown in Figure 2. For each studied subreddit, combining the dummy variables (see Methods) through all possible interaction terms allows a single ITS model to generate four regression best-fit estimates of hate speech, both for treatment and control users, on both pre- and post-treatment periods. It is not surprising that the users in the treatment group (those who joined hateful subreddits) already had a relatively high rate of hate speech before joining the hateful subreddits compared to the control users. Still, the treatment groups show significant increase in rates of hate speech after joining the hateful subreddits, while the control users show no change. Such difference in trends suggests that joining the hateful subreddit has a major effect on the rate of hate speech expressed by the users who joined.

From the plots we observe that, with varying intensities, treatment groups were already ramping up their hate speech before they joined the hateful subreddit. ITS design still allows us to conclude that the joining of that hate subreddit increases the user’s hate speech immediately, but this trend nevertheless raises questions regarding what drives users to become members of hateful communities at the first place. From the ITS model, we observe a post-join downwards trend, indicating that the immediate post-join hate speech spike wanes over time, although at the end of the analysis period all subreddits were still above the hate speech values observed at the beginning of the analysis period. Fixing the hate speech baseline to the start of the analysis period, the end-of-period hate speech values saw increases of 189%, 10%, 25% and 320% for r/GreatApes, r/CoonTown, r/Incels, and r/fatpeoplehate respectively.

To ensure the ITS model shows robust results regardless of the time bandwidth, we additionally performed a Sensitivity Analysis (see Methods). Here, we vary the bandwidth and observe the change of various model coefficients. Across all bandwidths we observe a high p-value ($>$0.05) in the three parameters that measure external effects occurring on all users: $time$, $inter$, and $time\times inter$. Those high p-values suggest that the we cannot reject the null hypothesis that the true values of aforementioned variables are zero. Since the three coefficients obtained by ITS model are statistically indistinguishable from zero, we assume that we do not have evidence for any significant global (external) effects acting on all users, as those effects would be captured by the three aforementioned variables. Hence we conclude the control users express no change in their hate speech over time. This implies all measured effects are resultants of treatment users joining the hateful subreddit. The action of joining, denoted as the $expos$ variable and its interaction terms, is the only difference between treatments and controls. In summary, thanks to the sensitivity analysis we show that the observed results are replicable across bandwidths, and not merely a by-product of an overfit optimal bandwidth.

We can observe a downwards trend in the treatment group post-joining. Given Reddit’s active role in moderation, this downwards trend can, at least partially, result from Reddit’s banning of individual users, which likely targets the most egregious offenders - those that most elevate the group’s hate speech. Considering this common practice of banning users who express uncivil behavior, the average hate speech for the group is expected to decrease over time as only samples for the non-banned less-hateful members are still available.

To test this hypothesis we analyze the link between hate speech level and account lifespan – the time an account is active on Reddit. We measure the average post-join hate speech for users who quit Reddit in less than one year after joining the studied subreddit and compare them to the users who stay longer. We observe that accounts with a longer lifespan have a lower average hate speech, as shown in Table II.

Complementarily, we leverage our data to explore hate speech trends over time for an expanded set of contexts in which treatment users can be found. We analyze the hate speech in the following scenarios: in all subreddits, inside the treatment subreddit, outside the treatment subreddit, inside banned subreddits and inside non-banned subreddits. The plots illustrating the rate of hate speech of treatment users over time are shown in Figure 3.

While effect sizes vary based on each subreddit and its unique lexicon, we generally observe that treatment users already had higher hate speech levels in the “banned subreddits“ category, and that after joining the studied hateful subreddit, their hate speech displays a visually noticeable increase in the banned subreddits category. It also increases in the “all“ category, which is to be expected since this includes the hateful subreddit itself. Hate speech trends in the non-banned set closely tracks that of the outside set, which is to be expected as those are highly overlapping. Yet it is noteworthy that the time-series of both sets are still visually above that of the control set.

All analysis were done based on the custom lexicons developed during this study. Figure 4 illustrates the lexicons’ most frequently written hate words in the in-group and out-group settings, considering posts after the users became active members of the hateful subreddit. We compare both distributions via a Spearman Rank Correlation test, which measures the monotonicity of the relationship between two datasets, here represented as the in-group and out-group word distributions. For all four subreddits studied the test obtained a p-value below 0.05, meaning we can reject the hypothesis that both word distributions are generated by the same underlying system. This confirms there is a difference in the distributions with which hate words are chosen when communicating to the in-group versus out-group.

When looking at racism lexicons, the usage of the n-word nearly doubles when members are communicating with the in-group. Similarly significant increases happen with other highly offensive words such as “obeast“ in the fat-shaming context. We also observe a reduction in group slang such as “sheboon“ and “roastie“ when speaking outside the hateful community. This effect happens across all categories of hate speech studied. Conversely, the out-group data shows an increase in derogatory-yet-not-as-societally-shunned terms such as “fatlogic“ and “fag“. In summary, discoursing in the out-group displays a relative toning-down of the most offensive words and of insider slangs, counter-weighted by a relative increase in milder hate words.

Through a robust ITS analysis we were able to identify a significant increase in hate speech resulting from joining subreddits known for their toxicity. This increase was observed outside of the community which incited it, thus implying a spread of hate arising from joining hateful communities.

When analyzing the ITS models in Figure 2, we noted that there is an upwards trend in hate speech that predates the moment when treatment users become active members of the hate community. It seems probable that that users were already becoming attracted to that specific thematic, perhaps already being passive members of the studied community, known in Reddit lingo as ”lurkers”, or users who read a subreddit’s forum but do not engage in discussion. Due to dataset limitations, we measure the treatment start as the date in which a user made a first post in the studied subreddit. There is a looming possibility that treatment users were already lurkers before becoming active, and thus their external behavior had already began reflecting their increasing extremism.

Additionally, we observe a downward trends in the treatment group post-joining. We find an evidence for a link between the decreased hate speech and the accounts’ lifespan. The lifespan is affected by either a voluntary abandonment of accounts created for usage on the hateful community, or banning by Reddit, although our data does not permit to differentiate amongst those possibilities. Besides the effects of Reddit’s banning polices, that were briefly explored in this study, there could be other reasons for such phenomenon. A commonplace explanation could be a simple regression to the mean – a phenomenon in which those performing extremely well or extremely poorly (outliers) tend to move closer to the mean over time. Further research is needed to confirm this hypothesis.

When looking at the percentage of hate speech produced by the members of the studied hateful subreddits in various contexts, it is observable that those users’ hate speech also increases in other subreddits that were also eventually banned at a later date as illustrated in Figure 3. We can see an expected connection between a subreddit adopting hateful words, and a probability of it being banned. Moreover, since a subreddit cannot adopt hate words after it is already banned, it must be that the adoption of hate words precedes the ban, although this analysis cannot necessarily prove that the adoption causes the ban.

Regarding hate speech vocabulary, in Figure 4 we observe clear differences in hate word choices when subreddit members express themselves inside versus outside the community. Those differences were confirmed by a Spearman Rank Correlation test. We observe that users choose more “insider“ hate words within their community, but less frequently bring these slang to the outside world, which is in line with some previous findings [37]. Such behavior might be one way of evoking group membership, by displaying knowledge of the group’s unique vocabulary, which has been shown to entice greater responsiveness from the community [36]. The converse is also true, where users who employ more obscure in-group language outside of their group might simply find low engagement in other forums, either due to the out-group’s disinterest in a foreign cause, or simply due to others inability to comprehend messaging laden with group-specific language. Both scenarios would discourage the usage of group-specific language in favor of a more commonplace vocabulary and style when communicating outside of the users’ community.

From a qualitative perspective, in both the racist and the fat-shaming communities there appears to be a toning-down on the relative usage of the most egregiously offensive words, such as the n-word and “obeast“, when communicating out-of-community. A possible reason to explain this is that even hate group members are still somewhat affected by society’s pressure towards civility. A secondary factor might once again be moderation, as those who chose the most offensive words in the out-group would be at higher likelihood of having their posting ability blocked, thus halting their spread of such vocabulary.

From observing four different subreddits, covering three categories of hate speech, we have shown a causal relationship between a user becoming active in the community and the user’s hate speech increasing immediately after. Such negative externality that arises from the growth of hateful communities makes a case for stronger moderation of such communities, an area where quarantining subreddits shows some promise. In general these findings corroborate with previous research [7, 37] that suggests positive hate reducing effects can be obtained by restraining such hateful communities.

In this paper we presented a causal inference analysis of the impact that joining a hateful subreddit has on a user’s ensuing hate speech in other subreddits. We looked at Reddit, one of the major social media platforms, and discover that the hate speech increases as an effect of joining hateful subreddits. We find the positive effects in three different categories of hate speech: racism, misogyny and fat-shaming.

This study displays novelty for using causal modeling to understand the influence of social media in hate speech, especially by leveraging ITS design to identify the effects. Additionally, we perform the sensitivity analyses by exploring the optimal time bandwidths and assessing the modulating effects of different time bandwidths on calculated effect sizes.

However, potential limitations warrant consideration. Our findings indicate that despite the matching technique employed, there were still pre-existing differences between the treatment and control groups, which stem at least in part from the need of matching without using the target variable, as commonly expected from such approaches [18]. This causes the treatment group to have a higher baseline hate speech level, measured as const + expos. Nevertheless the ITS model utilized is robust to different baselines by including a control group precisely to allow measurement of such extraneous effects. The approach to lexicon construction or the choice of lexicon usage can also have an impact in the analyses as hate speech is not binary, but rather covers an entire spectrum, from positively talking about the out-group, passing through mild-offenses and ending in a toxic discourse. Certain outlets might be more conducive to different extents of hate speech and thus differences could be found when lexicons differ.

We were also limited in our data availability, as it is only possible to measure a user’s active actions, namely posting and commenting. This constraint makes analysing the effects of passive content consumption challenging. Similarly, we only study four subreddits, and the additional analyses is needed to assess the generality of our conclusions in Reddit and other social media platforms.

Given the inability to obtain data regarding when a user formally chooses to become member of a subreddit, nor data regarding which posts were being read by a user, the current approach could not investigate if those relate to the observed pre-join upwards trend, thus leaving opportunity for future work. The mirror case is also true, exploration of what is driving the treatment users speech downwards after joining the hateful subreddit could unearth additional information to be leveraged in combating hate speech.

Funding for this work is provided through the USC-ISI Exploratory Research Award and through DARPA (Award # HR0011260595 and # HR001121C0169)

The authors declare no conflicts of interest.