Fact-Checking at Scale with DimensionRank

Boolean fact-checking is the assignment of a boolean label true or false, to some given proposition. For example, relative to the statement the American economy is strong in 2020, a boolean system would assign this a binary label of either true or false. A related variant to this problem is to report a single real-valued probability (between $0$ and $1$) for the given proposition. PolitiFact, for example, indicates low probability with the figurative term “pants-on-fire,” referring to an idomatic expression for telling a falsehood.

Relational fact-checking involves the retrieval of related documents, facts, or propositions, for a given proposition $p$, with some notion of agreement polarity. For example, relative to the statement the American economy is strong in 2020, a relational system might find related financial data, and indicate whether they support or contradict the original assertion.

We believe that both formulations are interesting. However, we believe that relational fact-checking is primary. As we will see, it is possible to solve relational fact-checking, without first solving boolean fact-checking. We believe boolean fact-checking can be created on the basis of a relational fact-checking system, using a form of belief propagation.

An article-form fact-check for the purported “fact” (i.e. proposition) $p$ takes the form of an article, which for us is a sequence of sentences. This article, in order to make its case will use some of its sentences to introduce assumptions, and use the rest of the sentences to justify inferences, based on the assumptions introduced.

In symbolic terms, we can characterize the situation as follows. We want to argue for the proposition $p$, and we must introduce some assumptions in order to do this, which we will denote $q_{1},...,q_{N}$. Thus, in order to check one initial proposition $p$, we end up introducing $N$ additional assumptions.

Because we are interested in fact-checking in the first place, we presumably believe, any fact we are depending on should be checked. In such a case, an article-form fact-check raises more questions than it answers. That is, where we once had one proposition $p$ to check, we now have $N$ new propositions to check after the first round. But, these $N$ article-form fact-checks will each introduce $N$ new assumptions each, meaning we will have $N^{2}$ to check after the second round, $N^{3}$ after the third round, and so on. Thus, not only does an article-fact check induce more new facts to check than we started with, but it actually induces exponentially more. Assuming a relatively constant staff, an employee-driven fact-checking organization will generate facts to check exponentially faster than they can check them.

The second problem we have with recursion is that there is no inherent “base case” to a recursive fact-checking operation. The checking of one fact requires the checking of $N$ others. Does the recursive growth in the number of relevant facts grow forever? What happens when a fact-check for a proposition $p$ ends up depending on itself? These philosophical questions admit, we believe, no known answer other than the scientific method itself. This will require us to recast the problem from one of checking “facts” to one of checking theories.

Underlying the application of science we must first have a language in which to write our propositions, and a set of inference rules to draw conclusions based on initial assumptions. The pair of language and inference rules are called a logical calculus. We can use for a logical calculus any descendant of the propositional calculus (Andrews, 1986). For example, we can use the propositional calculus itself, or the first-order calculus, or an intensional calculus. The calculus specifies a language $L$ and a set of inference rules $R$. $L$ can be seen as a denumerably infinite set of propositions $L=\left\{p_{1},p_{2},...\right\}$. A theory $T$ (in $L$) is a set of propositions, each taken from $L$.

In science, propositions can be categorized as either data or explanatory principles. Data record measurements of the “outside world.” Aside from data, we have explanatory principles, which attempt to “explain” the data, by predicting it, as a consequence of a theory. We generally prefer the theory “does the most with the least,” in the sense that it predicts as much data as possible, while also being as small as possible (cf. Occam’s Razor). Now, the data are just measurements, and as such can be incorrect. Sometimes, scientists will try to argue against the accuracy of data, rather than give up on a theory. Nevertheless, incorrect data is still data, and must be distinguished from explanatory principles.

Underlying the scientific method is a calculus, which we underline, is a descendant of the propositional calculus. Aside from introducing a language, the calculus a set of inference rules. The inference rules allow us to extend a theory $T$. An inference rule $R(t_{1},...,t_{m})\rightarrow\alpha$, says that, if $\left\{t_{1},...,t_{m}\right\}\subseteq T$, then we can create a new set $T^{\prime}=T\cup\left\{\alpha\right\}$, where $\alpha$ is inferred from $T$. For example, from assuming all men are mortal and Socrates is a man, we can infer Socrates is mortal, in first-order logic.

Because we have assumed we are working with a propositional calculus we have the concept of negation. The negation of the statement $\alpha$ would be the statement $\alpha$ is not true, written $\neg\alpha$. If a theory contains $\alpha$ and $\neg\alpha$, it is said to be inconsistent. A theory that is inconsistent is also said to have a contradiction. If there are no contradictions, the theory is called consistent. From this perspective, the goal of science is to produce theories consistent with the data, and not inconsistent with it. One can argue against a theory either by contradicting directly its core assumptions, or else by contradicting one of its predictions.

An individual theory must be consistent to be correct. However, the set of all possible theories, if unified together, would not be consistent. That is, at any given time, there groups of active theories that cannot all be true. We can regard the set of all theories, or else all actively considered theories, as a set of theories called the universe of active theories. The union of all propositions in all active theories can be called the universe of all active propositions. Our goal then is to be able to “check” any proposition in the universe of active propositions, and do so without degrading in the (normal) case that the union of all active theories cannot consistently be merged.

There has been much discussion recently in various kinds of press, including business and political, about the notion of a “platform.” We define a platform as a channel that connects many producers to many consumers. That is, each producer can user the channel to some set of consumers. The producer and consumer can be in different organizations, in which case the constitute provider and client. Or the producer and consumer can be in the same organizations, in which case they constitute two steps in the supply chain.

Under this definition, an old-fashioned physical marketplace would be kind of a platform, because it connects many producers to many consumers at the same time. A highway network is a platform, because it transports goods from producer to consumer, as is a railroad network.

In the information-technology field, one platform is built upon another. The personal-computer hardware platform connects software producers to consumers. The operating system sits atop the hardware, and is a platform that connects application producers to consumers. The Internet browser sits atop the operating system, and is a platform connecting web site producers to consumers. Popular web sites like Amazon, Google and Facebook sit atop the Internet browser and the Internet, and connect users to other users.

An information-sharing platform is a platform whose object is the communication, storage and organization of information.¹¹1 By “information” we can either mean arbitrary sequences of bits. Claude Shannon famously studied the quantification of the amount of information in a sequence of bits, but we do not need to worry here about how to quantify information. Information platforms are interesting because they allow us as humans to transcend the normal bounds that exist on how many individuals can work effectively together.

Dunbar’s number suggests that the largest number of meaningful relationships a person can have is $150$ (Dunbar, 1982). The reason hypothesized for a limit is that each person has a limited information-processing capacity. In order to interact with someone meaningfully, we not only need to spend time communicating with them, but also considering what they have said once communication is over. Using information-sharing platforms, like Google Search, Facebook, Twitter, etc., we are able to effectively work with, communicate with, and otherwise share information with far more people than had every been possible before. Information-sharing platforms allow us to create super organizations, that can achieve more than organizations without the same technology.

The ability for people to share information more effectively than ever before has precipitated the crisis of fact-checking, requiring the need for fact-checking at scale. However, we can also turn this crisis to opportunity, by using the power of the platform itself to solve the problem of fact-checking.

Let us break down the component concepts involved in our claim.

Here, then, are our criticisms of the PolitiFact model.

The answer is that we will operationalize the task of fact-checking, as the task of giving the best arguments for-and-gainst a given proposition $p$. This is, in other words, a primarily relative fact-checking formulation.

The crucial new query, that has not been possible with one-dimensional quality ratings is then: list me the best arguments for (or against) the proposition $p$.

Algorithm: To run this query, we first select all comments $c^{p}_{a}$ that refer to $p$ with agreement polarity $a$. That is, we filter these comments according to their agreement polarity. Finally, we sort the remaining comments by hotness. This way, we have got the best arguments, either for or against, the target proposition $p$.

This formulation allows us to present a version of relative fact-checking, which is the task of, for a proposition $p$, bringing up relevant other information, and indicating its relation to the proposition $p$.

In this section we will show how to construct a logical calculus within our network formulation, which is equivalent in power to the theorem-proving system of the propositional calculus. That is, whatever can be proven in propositional calculus can be proven in our platform calculus. In fact, we can show that each node will have $O(1)$ size in the length of the proof overall. This means that ours is not a trivial formulation, in which, e.g., an entire proof is simply written in a single message. And, by construction, whatever cannot be proven in the propositional calculus cannot be proven in our calculus either. The propositional calculus is consistent and complete, meaning that a set of hypotheses $H$ derives a proposition $\alpha$, if and only if the inference is really warranted. Thus, our new calculus is also consistent and complete for the propositional calculus.

The ordinary DimensionRank algorithm describes how to give a personalized search result based on a non-personalized result (Coppola, 2020). That is, to personalize a search result for user $u$, we simply take the non-personalized search result, $R$, and tailor this set to $u$ using their unique user embeddings, to obtain $R^{u}$, personalized to $u$. We can of course apply the same principle here to the special case to give a personalized ranking of arguments for-and-against. In other words, one user might be interested in certain arguments more than another, and each user would be able to get a personalized experience using this algorithm.

Bill Gates has recently suggested that one of the most important questions is how to balance authoritative voices with the activities of the citizen-users of the platform (Gates, 2020).

We can offer a solution that balances the public’s desire to express themselves, with the additional desire to maintain an ability to allow “authoritative voices,” wherever they may come from, to weigh in. Suppose we have authoritative voices, whose voices we want to promote for a given fact. We can simply promote their comments to any target message $m$ to occur as a prefix to the arguments from the rest of the platform. To maintain a balance towards free speech and civic involvement, the rest of the users still have the ability to offer alternative arguments, both for and against, any target message. Finally, both platform users and designated authoritative voices can each recursively fact-check one another. This is a possibility that hasn’t been possible before our platformization of the fact-check.

We have implemented this algorithm in our open-source DimensionRank search and social media software package, available at thinkdifferentagain.art. We have our own network running this algorithm at deeprevelations.com.