Unification-based Reconstruction of Multi-hop Explanations for Science Questions

Answering multiple-choice science questions has become an established benchmark for testing natural language understanding and complex reasoning in Question Answering (QA) Khot et al. (2019); Clark et al. (2018); Mihaylov et al. (2018). In parallel with other NLP research areas, a crucial requirement emerging in recent years is explainability Thayaparan et al. (2020); Miller (2019); Biran and Cotton (2017); Ribeiro et al. (2016). To boost automatic methods of inference, it is necessary not only to measure the performance on answer prediction, but also the ability of a QA system to provide explanations for the underlying reasoning process.

The need for explainability and a quantitative methodology for its evaluation have conducted to the creation of shared tasks on explanation reconstruction Jansen and Ustalov (2019) using corpora of explanations such as Worldtree Jansen et al. (2018, 2016). Given a science question, explanation reconstruction consists in regenerating the gold explanation that supports the correct answer through the combination of a series of atomic facts. While most of the existing benchmarks for multi-hop QA require the composition of only 2 supporting sentences or paragraphs (e.g. QASC Khot et al. (2019), HotpotQA Yang et al. (2018)), the explanation reconstruction task requires the aggregation of an average of 6 facts (and as many as $\approx$20), making it particularly hard for multi-hop reasoning models. Moreover, the structure of the explanations affects the complexity of the reconstruction task. Explanations for science questions are typically composed of two main parts: a grounding part, containing knowledge about concrete concepts in the question, and a core scientific part, including general scientific statements and laws.

Consider the following question and answer pair from Worldtree Jansen et al. (2018):

• •

  $q$: what is an example of a force producing heat?
  $a$: two sticks getting warm when rubbed together.

An explanation that justifies $a$ is composed using the following sentences from the corpus: ($f_{1}$) a stick is a kind of object; ($f_{2}$) to rub together means to move against; ($f_{3}$) friction is a kind of force; ($f_{4}$) friction occurs when two objects’ surfaces move against each other; ($f_{5}$) friction causes the temperature of an object to increase. The explanation contains a set of concrete sentences that are conceptually connected with $q$ and $a$ ($f_{1}$,$f_{2}$ and $f_{3}$), along with a set of abstract facts that require multi-hop inference ($f_{4}$ and $f_{5}$).

Previous work has shown that constructing long explanations is challenging due to semantic drift – i.e. the tendency of composing out-of-context inference chains as the number of hops increases Khashabi et al. (2019); Fried et al. (2015). While existing approaches build explanations considering each question in isolation Khashabi et al. (2018); Khot et al. (2017), we hypothesise that semantic drift can be tackled by leveraging explanatory patterns emerging in clusters of similar questions.

In Science, a given statement is considered explanatory to the extent it performs unification Friedman (1974); Kitcher (1981, 1989), that is showing how a set of initially disconnected phenomena are the expression of the same regularity. An example of unification is Newton’s law of universal gravitation, which unifies the motion of planets and falling bodies on Earth showing that all bodies with mass obey the same law. Since the explanatory power of a given statement depends on the number of unified phenomena, highly explanatory facts tend to create unification patterns – i.e. similar phenomena require similar explanations. Coming back to our example, we hypothesise that the relevance of abstract statements requiring multi-hop inference, such as $f_{4}$ (“friction occurs when two objects’ surfaces move against each other”), can be estimated by taking into account the unification power.

Following these observations, we present a framework that ranks atomic facts through the combination of two scoring functions:

• •

  A Relevance Score (RS) that represents the lexical relevance of a given fact.

• •

  A Unification Score (US) that models the explanatory power of a fact according to its frequency in explanations for similar questions.

An extensive evaluation is performed on the Worldtree corpus Jansen et al. (2018); Jansen and Ustalov (2019), adopting a combination of k-NN clustering and Information Retrieval (IR) techniques. We present the following conclusions:

1. 1.

   Despite its simplicity, the proposed method achieves results competitive with Transformers Das et al. (2019); Chia et al. (2019), yet being orders of magnitude faster, a feature that makes it scalable to large explanatory corpora.

2. 2.

   We empirically demonstrate the key role of the unification-based mechanism in the reconstruction of many hops explanations (6 or more facts) and explanations requiring complex inference (+12.0 Mean Average Precision).

3. 3.

   Crucially, the constructed explanations can support downstream question answering models, improving the accuracy of BERT Devlin et al. (2019) by up to 10% overall.

To the best of our knowledge, we are the first to propose a method that leverages unification patterns for the reconstruction of multi-hop explanations, and empirically demonstrate their impact on semantic drift and downstream question answering.

A multiple-choice science question $Q=\{q,C\}$ is a tuple composed by a question $q$ and a set of candidate answers $C=\{c_{1},c_{2},\ldots,c_{n}\}$. Given an hypothesis $h_{j}$ defined as the concatenation of $q$ with a candidate answer $c_{j}\in C$, the task of explanation reconstruction consists in selecting a set of atomic facts from a knowledge base $E_{j}=\{f_{1},f_{2},\ldots,f_{n}\}$ that support and justify $h_{j}$.

In this paper, we adopt a methodology that relies on the existence of a corpus of explanations. A corpus of explanations is composed of two distinct knowledge sources:

• •

  A primary knowledge base, Facts KB ($F_{kb}$), defined as a collection of sentences $F_{kb}=\{f_{1},f_{2},\ldots,f_{n}\}$ encoding the general world knowledge necessary to answer and explain science questions. A fundamental and desirable characteristic of $F_{kb}$ is reusability – i.e. each of its facts $f_{i}$ can be potentially reused to compose explanations for multiple questions

• •

  A secondary knowledge base, Explanation KB ($E_{kb}$), consisting of a set of tuples $E_{kb}=\{(h_{1},E_{1}),(h_{2},E_{2}),\ldots,(h_{m},E_{m})\}$, each of them connecting a true hypothesis $h_{j}$ to its corresponding explanation $E_{j}=\{f_{1},f_{2},\ldots,f_{k}\}\subseteq F_{kb}$. An explanation $E_{j}\in E_{kb}$ is therefore a composition of facts belonging to $F_{kb}$.

In this setting, the explanation reconstruction task for an unseen hypothesis $h_{j}$ can be modelled as a ranking problem Jansen and Ustalov (2019). Specifically, given an hypothesis $h_{j}$ the algorithm to solve the task is divided into three macro steps:

1. 1.

   Computing an explanatory score $s_{i}=e(h_{j},f_{i})$ for each fact $f_{i}\in F_{kb}$ with respect to $h_{j}$

2. 2.

   Producing an ordered set $Rank(h_{j})=\{f_{1},\ldots,f_{k},f_{k+1},\ldots,f_{n}\text{ }|\text{ }s_{k}\geq s_{k+1}\}\subseteq F_{kb}$

3. 3.

   Selecting the top $k$ elements belonging to $Rank(h_{j})$ and interpreting them as an explanation for $h_{j}$; $E_{j}=topK(Rank(h_{j}))$.

We carried out an empirical evaluation on the Worldtree corpus  Jansen et al. (2018), a subset of the ARC dataset Clark et al. (2018) that includes explanations for science questions. The corpus provides an explanatory knowledge base ($F_{kb}$ and $E_{kb}$) where each explanation in $E_{kb}$ is represented as a set of lexically connected sentences describing how to arrive at the correct answer. The science questions in the Worldtree corpus are split into training-set, dev-set, and test-set. The gold explanations in the training-set are used to form the Explanation KB ($E_{kb}$), while the gold explanations in dev and test set are used for evaluation purpose only. The corpus groups the explanation sentences belonging to $E_{kb}$ into three explanatory roles: grounding, central and lexical glue.

Consider the example in Figure 1. To support $q$ and $c_{j}$ the system has to retrieve the scientific facts describing how friction occurs and produces heat across objects. The corpus classifies these facts ($f_{3},f_{4})$ as central. Grounding explanations like “stick is a kind of object” ($f_{1}$) link question and answer to the central explanations. Lexical glues such as “to rub; to rub together means to mover against” ($f_{2}$) are used to fill lexical gaps between sentences. Additionally, the corpus divides the facts belonging to $F_{kb}$ into three inference categories: retrieval type, inference supporting type, and complex inference type. Taxonomic knowledge and properties such as “stick is a kind of object” ($f_{1}$) and “friction is a kind of force” ($f_{5}$) are classified as retrieval type. Facts describing actions, affordances, and requirements such as “friction occurs when two object’s surfaces move against each other” ($f_{3}$) are grouped under the inference supporting types. Knowledge about causality, description of processes and if-then conditions such as “friction causes the temperature of an object to increase” ($f_{4}$) is classified as complex inference.

We implement Relevance and Unification Score adopting TF-IDF/BM25 vectors and cosine similarity function (i.e. $1-cos(\vec{x},\vec{y})$). Specifically, The RS model uses TF-IDF/BM25 to compute the relevance function for each fact in $F_{kb}$ (i.e. $rs(h_{j},f_{i})$ function in Equation 1) while the US model adopts TF-IDF/BM25 to assign similarity scores to the hypotheses in $E_{kb}$ (i.e. $sim(h_{j},h_{z})$ function in Equation 2). For reproducibility, the code is available at the following url: https://github.com/ai-systems/unification_reconstruction_explanations. Additional details can be found in the supplementary material.

This paper proposed a novel framework for multi-hop explanation reconstruction based on explanatory unification. An extensive evaluation on the Worldtree corpus led to the following conclusions: (1) The approach is competitive with state-of-the-art Transformers, yet being significantly faster and inherently scalable; (2) The unification-based mechanism supports the construction of complex and many hops explanations; (3) The constructed explanations improves the accuracy of BERT for question answering by up to 10% overall. As a future work, we plan to extend the framework adopting neural embeddings for sentence representation.

The authors would like to thank the anonymous reviewers for the constructive feedback. A special thanks to Deborah Ferreira for the helpful discussions, and to the members of the AI Systems lab from the University of Manchester. Additionally, we would like to thank the Computational Shared Facility of the University of Manchester for providing the infrastructure to run our experiments.