Investigating the Validity of Botometer-based Social Bot Studies

Social bot or not? An extensive amount of research has been published in recent years suggesting that social media platform like Twitter are inhabited by vast numbers of social bots. These are supposed to be accounts pretending to be human users but which are operated automatically by malicious actors with the goal of manipulating public opinion. The supposed influence of social bots in political discussions has raised significant concerns, particularly given their alleged potential to adversely impact democratic outcomes.

The idea of social bot armies has been widely and frequently covered by media outlets across the world, with new reports of supposed social bot activity appearing almost on a weekly basis for the last couple of years, in the context of a wide variety of different topics. Discussions that have reportedly been attacked by social bot activity include the Brexit referendum, elections and political unrests in various countries, climate change, immigration, racial unrest, cannabis, vaping, COVID-19, vaccines, and even celebrity gossip. As a consequence, political countermeasures against the supposed dangers of social bot activity have been discussed and legal regulations have been implemented, for example California’s Bot Disclosure Law (2019) or Germany’s ‘Medienstaatsvertrag’ (2020).

Many of these news reports and also most scientific publications about social bots from research groups around the world are based on Botometer (originally called BotOrNot), which has often been referred to as the “state-of-the-art bot detection method”. A Google Scholar search in May 2022 using the query "BotOrNot" OR "Botometer" returns 1,720 results.

A typical definition of a social bot is given in [1]: “Social bots are automated accounts that use artificial intelligence to steer discussions and promote specific ideas or products on social media such as Twitter and Facebook. To typical social media users browsing their feeds, social bots may go unnoticed as they are designed to resemble the appearance of human users (e.g., showing a profile photo and listing a name or location) and behave online in a manner similar to humans (e.g., ’retweeting’ or quoting others’ posts and ’liking’ or endorsing others tweets).”

However, as has been pointed out by Rauchfleisch and Kaiser [15], there is some confusion as to what exactly “social bot” researchers or tools like Botometer are trying to find. Authors in this field often use the terms ‘social bot’, ‘bot’, ‘social media bot’, or ‘automated account’ more or less interchangeably, even though—according to the above definition—automation is a necessary but not a sufficient condition for a social bot. A deeper discussion of these issues can be found in [6]. For the purposes of this paper, however, the exact definition of these terms will not matter. In particular, we will demonstrate that the vast majority of the accounts that are flagged as “bots” by Botometer are real people and do not involve any automation at all. In the rare occasions where we found partly automated accounts, e.g. automated retweets or accounts that automatically cross-posted content from other social media platforms on Twitter, we will point this out explicitly.

This paper is structured as follows: In Section 2, we discuss theoretical and methodological limitations of automatic bot-detection tools like Botometer. We point out a fundamental theoretical flaw of the commonly used approach of estimating the level of social bot activity. In Section 3, we evaluate the performance of Botometer empirically using various samples of Twitter accounts and discuss related research that used a similar approach. When using Botometer on accounts of known humans, the false-positive rate turns out to be significant. In Section 4, we describe our experiments to evaluate the validity of Botometer scores in real-world scenarios. To our knowledge, a systematic evaluation of this type has never been reported before. Our results are devastating for the whole body of Botometer-based research: Nearly all accounts that are labeled as “bots” based on Botometer scores are false positives. Many of these accounts are operated by people with impressive academic and professional credentials. Not a single one of the hundreds of accounts we inspected—each of which had been flagged by Botometer—was a “social bot” according to the above definition. In Section 5, we present our conclusions.

Botometer is an automated tool designed to discriminate social bots from human users. It is built on a supervised machine learning approach. To discriminate between the two classes, a random forest classifier is trained on two samples of user accounts, one labeled “human” and one labeled “bot”. The classification is based on more that 1,000 features which, according to [2], include statistical features of retweet networks, meta-data, such as account creation time, the median number of followers of an accounts social contacts, the tweet rate, and features based on part-of-speech tagging and sentiment analysis.

The training of Botometer is based on a publicly available dataset¹¹1https://botometer.osome.iu.edu/bot-repository/datasets.html where (in 2019) 57,155 accounts were labeled “bot” and 30,853 were labeled “human” [18] The accounts come from a variety of different sources and many of the labels seem at least questionable. The largest subset of “bots” comes from a sample of spammy or promotional accounts from the early days of Twitter (2009 - 2010). The study where these accounts were collected referred to them as “content polluters” and did not claim that these accounts were automated or bots [13]. Many of the accounts from other sources were apparently labeled manually by laypersons with little understanding of the state-of-the-art in human-machine interaction and the difficulty of evading Twitter’s detection of nefarious platform use, and based on a naïve understanding of what constitutes a “bot” (possibly based on questionable clues like a high amount of retweets, a small or large number of followers, missing profile picture, digits in the Twitter handle, or, as empirically validated in [17], opposing political views). Some accounts in the “bot repository” were explicitly labeled as “bots” because they appeared to have participated in “follow trains”, a technique used by human political activists on Twitter to rapidly increase their follower count. Clearly, the lack of reliable ground truth data is the first glaring methodological problem of Botometer. It seems far from obvious that training a classifier on a rather arbitrary selection of account samples which are based on vastly different ideas of what constitutes a “bot” will result in a useful tool.

Botometer is available both over an API and over a web interface. It provides a score between 0 and 1 for individual Twitter accounts which is calculated by calibrating the raw score provided by the random forest classifier. Higher scores are associated with a higher “bot likelihood” on accounts that are labeled “bot” in the bot repository. This “bot likelihood” is linearly rescaled to a scale from 0 to 5 and presented on the website. Additionally, a “complete automation probability” (CAP) is provided since version 3. The CAP is based on a non-linear rescaling of the bot score according to the Bayes rule and is supposed to be interpreted as the posterior probability that an account is a bot. According to [18], the CAP is based on the assumption that the prior probability of observing a bot is 0.15 and provides “generally more conservative” scores than the original bot score. That is, the rescaled “bot likelihood” is based on the assumption that roughly 50% of the accounts encountered by Botometer are bots, whereas the CAP is based on the assumption that 15% of these accounts are bots.²²2 Notably, this 15% estimate was obtained using an earlier version of Botometer as a classifier and without manually verifying those results. [16]

Now consider the typical methodology of most disinformation studies which employ Botometer (or similar tools). Two of many such studies will be discussed in detail in Section 4:

1. 1.

   A large sample of tweets or user accounts is collected related to a certain topic, for example, all followers of certain accounts or all tweets that contain certain hashtags or keywords, e.g. ‘political issue’.

2. 2.

   The list of accounts is fed into Botometer and the resulting “bot scores” are stored in a file.

3. 3.

   The study authors take a look at the histogram of the “bot scores”. On this basis, a suitable threshold is selected in some obscure or arbitrary manner. This step is often skipped and instead, a threshold of 50% (2.5 out of 5) is employed.

4. 4.

   The amount of “bots” and “bot-generated” tweets is calculated based on this threshold, resulting in headlines like “30% of the Twitter users who tweet about ‘political issue’ are bots”, or “half of the tweets about ‘political issue’ are generated by bots”.

Even under the optimistic assumption that there is a significant correlation between the “bot score” and the true nature of the account, this approach is fundamentally flawed. By adjusting the threshold, almost any amount of “bots” between 0% and 100% that is desired or expected by the authors can be produced as a result (see Figure 3). Researchers expecting a large number of social bots will choose a low threshold, while researchers expecting a low number of social bots will choose a high threshold.³³3This resembles a technique of adjusting a bedridden patient’s blood pressure reading by heavily tilting the bed, as described in Samuel Shem’s satirical novel House of God: “You can get any blood pressure you want out of your gomer.” On the other hand, if the threshold is not adjusted, the implicit assumption is that the relative number of bots in the real-world sample is the same as in the training and/or validation data used to optimize Botometer. In either case, using this approach to estimate the prevalence of “bots” is a textbook example of circular reasoning. The prevalence of social bots desired or expected by the researchers directly affects the prevalence that will be “measured”.

The problem can also be pointed out in a probabilistic framework. An optimal classifier for “bots” vs. “humans” will follow the Bayes decision rule which minimizes the number of errors. Therefore, based on a feature vector $\vec{x}$, it will make a decision for the class Bot if and only if

Botometer, like any other classifier, incorporates an estimate of the prior probability $p(Bot)$, explicitly or implicitly. As discussed above, the calibrated “‘bot score” is based on the assumption that $p(Bot)$ is roughly 50%, whereas the CAP is based on the assumption that $p(Bot)$ is 15%. Adjusting the “bot score” threshold is equivalent to modifying the prior probability $p(Bot)$. However, $p(Bot)$ is nothing but the prevalence of social bots that researchers are trying to measure in the first place. This is a circular dependency: In order to classify accounts into “bots” and “humans” with the goal of using the relative frequency of “bots” as an estimate for $p(Bot)$, we already need to know $p(Bot)$ beforehand.

Using a classifier like Botometer to obtain a reliable estimate of $p(Bot)$ by counting unvalidated classification results would require probability density functions $p(\vec{x}|Bot)$ and $p(\vec{x}|Human)$ with virtually no overlap. Only then, the impact of the unknown probability $p(Bot)$ on the classification result could reasonably be neglected. However, looking at actual “bot score” distributions and given the consistent failure of Botometer in our experiments, this is clearly not the case.⁴⁴4A similar problem arises when human labelers are instructed to rate accounts as “bots” or “humans”. The ratings will typically be based on unrealistically high expectations of the bot prevalence $p(Bot)$ (fueled by Botometer-based publications and media coverage), a limited understanding of the state of the art in artificial intelligence combined with misconceptions of what features might be “bot-like” (i.e. a bad estimate of $p(\vec{x}|Bot)$), as well as false and narrow expectations of what a “normal” human behavior on Twitter might be (i.e. a bad estimate of $p(\vec{x}|Human)$ ). As a result, many accounts that are clearly not automated but were rated “bots” by human labelers can be found in the “bot repository” used to train Botometer.

These considerations leave little hope that Botometer or similar tools might be of any value for estimating the prevalence of “bots”.

The problem that Botometer produces enormous amounts of false positives and that it should never be trusted without manual verification has been pointed out for years [10, 5, 11, 12, 15].

Different methods have been used to demonstrate the problem. A simple and effective way is to use Botometer to classify accounts that are without doubt operated by humans. When we tested Botometer in April 2018, nearly half of U.S. Congress members present on Twitter were misclassified as bots (47%), using the most commonly used “bot score” threshold of 50% (or 2.5 on a scale from 0 to 5), see Figure 1. ⁵⁵5Surprisingly, in May 2019, Botometer performed dramatically better on the members of Congress; the false positive rate dropped from 47% to 0.4%. Possibly, these accounts had been added to the Botometer training data as examples of human users in the meantime.

In similar experiments in May 2019 [5, 11, 12], we found that

• •

  10.5% of NASA-related accounts are misclassified as bots.

• •

  12% of Nobel Prize Laureates are misclassified as bots.

• •

  14% of female directors are misclassified as bots.

• •

  17.7% of Reuters journalists are misclassified as bots.

• •

  21.9% of staff members of UN Women are misclassified as bots.

• •

  35.9% of the staff of German news agency “dpa” are misclassified as bots.

Clearly, Botometer’s glaring false positive problem has still not been solved with the current version. In January 2021, even the Twitter account @POTUS of the newly elected president of the United States, Joe Biden, was classified as a “bot”, with a “bot score” of 3.2, see Figure 2. In May 2022, both presidential accounts @POTUS and @JoeBiden received “bot scores” of 3.8.

The lack of reliability goes both ways. When we tested Botometer with real, automated Twitter bots in May 2019, we found that

• •

  36% of known bots by New Scientist are misclassified as humans.

• •

  60.7% of the bots collected by Botwiki are misclassified as humans.

A similar, but more systematic approach to evaluate the ability of Botometer to discriminate between automated accounts and humans was chosen by [15]. The authors performed experiments on five datasets of verified bots and verified humans. Although the datasets had been partly used to train Botometer, the authors find that “the Botometer scores are imprecise when it comes to estimating bots. […] This has immediate consequences for academic research as most studies using the tool will unknowingly count a high number of human users as bots and vice versa.”

Although they clearly demonstrate severe limitations of Botometer, evaluations on manually selected lists of accounts do not allow for an estimate of Botometer’s reliability in a real-world setting. In a real-world scenario, an enormous variety of human user behaviors may occur which might not be included in manually constructed test samples. Also, it seems unlikely that actual malicious “social bots” would behave in the same manner as the bots employed in these experiments. Therefore, the only way to get a realistic impression of Botometer’s performance in real-world scenarios is to take a closer look at the accounts that are classified as bots in a specific setting. Unfortunately, it turns out that authors of studies of this type are extremely reluctant to share their lists of “bots”. In our impression, most of the study authors we contacted were fully aware that the lion’s share or even all of the accounts they counted as “bots” were, in fact, humans.

This reluctance to share data motivated us to replicate one study of this type about alleged social bots among the followers of German political parties [9]. In only one case, we were able to obtain the relevant raw data from the authors of a peer-reviewed Botometer-based study. We are grateful that Dunn et. al. [4] shared their list of “bots” with us to allow us to take a closer look.

In this section, we will first evaluate the performance of Botometer on the basis of two peer-reviewed studies where this tool was employed to estimate the prevalence of social bots. Lastly, we will summarize the findings of two recent studies where Botometer results have been checked manually.

The field of social bot research is fundamentally flawed. While social bot researchers have received an enormous amount of public attention, the vast majority of their findings fully rely on the accuracy of Botometer. Many researchers simply refer to all accounts in their studies as “bots” or even as “‘social bots” as long as they exceed some arbitrarily chosen Botometer threshold. However, this approach is highly questionable from a theoretical point of view. And, as we have demonstrated empirically, Botometer fails miserably and consistently when evaluated under real-world conditions. Studies claiming to investigate the prevalence, properties, or influence of social bots based on Botometer have, in reality, just investigated false positives and artifacts of this approach.

While automated Twitter accounts exist, we have yet to see a single credible example of a malicious social bot, i.e. an account that pretends to be a human user and is operated by some sinister actor to manipulate public opinion.

In our impression, a prevailing culture of intransparency is a major factor that has enabled an entire field of research to rely on deeply flawed methods. Publishing or sharing raw data, as it is common practice in many other fields of science, would have helped to identify and highlight the fundamental methodological problems much earlier.

We conclude that all past and future claims about the prevalence or influence of social bots that are not accompanied by lists of account IDs are highly questionable and should be ignored. In those cases where actual social bot  accounts are named, we recommend an in-depth analysis of these accounts to verify whether these are not simply human beings—made of flesh and blood—who have been misclassified as social bots like hundreds of thousands of Twitter users before them.