Learning Causal Bayesian Networks from Text

Causation is a powerful psychological tool for human to choreograph his surrounding environment into a mental model, and use it for reasoning and decision-making. However, inability to identify causality is known to be one of the drawbacks of current Artificial Intelligence (AI) systems (Lake et al., 2015). Extraction of causal relations from text is necessity in many NLP tasks such as question answering and textual inference, and has attracted a considerable research in recent years Wood-Doughty et al. (2018); Zhao et al. (2018, 2017); Ning et al. (2018); Rojas-Carulla et al. (2017). However, the state-of-the-art methods are limited to the identification of causal relations between low-level individual events (Dunietz et al., 2017; Hidey and McKeown, 2016; Mirza and Tonelli, 2016) and fail to capture such relationships at conceptual level. Furthermore, relying on linguistic features limits the identification of causal relations to those whose cause and effect are located in the same sentence or in consecutive sentences.

In this paper, we propose a method for extracting concepts and their underlying causal relations from written language. Furthermore, to leverage the extracted causal information, we represent the causal knowledge in the form of a Causal Bayesian Network (CBN). Having this tool enables answering complex causal and counter-factual questions, such as: How psychotherapy can affect the patient’s emotion?, or What would happen if instead of medicine X, medicine Y was prescribed?

The contribution of this paper is three-fold. Firstly, we focus on identifying causal relation between concepts (e.g. physical activity and health). Secondly, We propose a novel method to represent the extracted causal knowledge in the form of a Causal Bayesian Network, enabling easy incorporation of this invaluable knowledge into downstream NLP tasks. Thirdly, we release PsyCaus dataset which can be used to evaluate causal relation extraction models in the domain of psychology ¹¹1https://github.com/farhadmfar/psycaus. In addition, our proposed method identifies causality between concepts independent of their locations in text, and is able to identify bi-directional causal relations between concepts, where two concepts have causal effect on each other. By aggregating linguistic variable, we construct a hierarchy where each variable, e.g. delusional disorder, lies under its related concept, e.g. disorder. This hierarchical and inheritance structure allows for the inference of causal relations between concepts that are not directly discussed in the text.

In order to evaluate our proposed method, we gathered a corpus of psychological articles The experimental results shows that the proposed method performs significantly better than the state-of-the-art methods.

Identification of causality in NLP is not trivial as a result of language ambiguity. Hence, most current approaches focus on verb-verb, verb-noun, and noun-noun relations. The explicit relations are often captured with narrow syntactic and semantic constructions (Do et al., 2011; Hendrickx et al., 2009; Mirza and Tonelli, 2016; Hidey and McKeown, 2016) which limits their recall. To go beyond surface form constructions few works have proposed neural models (Martínez-Cámara et al., 2017; Dasgupta et al., 2018; Zhao et al., 2017) covering wider causal constructions. However, most works don’t go beyond extracting causality between adjacent events, and so lack the ability to capture causality in non-adjacent concept level, e.g. genetics and hallucination. Therefore, in this paper we propose a model for identifying causality between concepts, independent of their location, and represent the causal knowledge in form of a Causal Bayesian Network.

Given the input, in form of human written language, we aim to extract the causal relation between concepts and represent the output in form of a Causal Bayesian Network. Hence, we split this task into three sub-tasks: extracting linguistic variables and values, identifying causality between extracted variables, and creating conditional probability table for each variable. In the following sub-sections each of these sub-tasks are explained.

In this section, we evaluate the effectiveness of our method and compare it with that of the state-of-the-art methods on a collection of Wikipedia articles related to Psychology. Each article within this collection is selected based on the terms in APA dictionary of Psychology (VandenBos, 2007). This collection contains a total number of 3,763,859 sentences. Among all possible relation between concepts, we studied 300 relationships between concepts, which were annotated by 5 graduate student from school of Psychology. Each tuple of relationship in form of $(A,B)$, were labelled as $-1,0$, or $1$, where $1$ indicates $A\xrightarrow{cause}B$, $-1$ shows that $B\xrightarrow{cause}A$, and $0$ implies no causal relations. With the overlap tuples (25%) we measured the degree of agreement between annotators using Fleiss’ Kappa measure Fleiss and Cohen (1973), which was around $0.67$. This indicates the reliability of our test setup.

We compare our model to the feature-based and distribution-based methods proposed by Rojas-Carulla et al. (2017), with different proxy projection functions, including { w2vii,  w2vio,  w2voi,  counts,  prec-counts,  pmi,  prec-pmi}. Furthermore, we compare our model to heuristic models, consisting of frequency, precedence, PMI, PMI (precedence), where in each of the models two parameters are calculated, $S_{V_{i}\rightarrow V_{j}}$ and $S_{V_{i}\rightarrow V_{j}}$, indicating $V_{i}\xrightarrow{cause}V_{j}$ if $S_{V_{i}\rightarrow V_{j}}>S_{V_{i}\rightarrow V_{j}}$ and $V_{i}\xleftarrow{cause}V_{j}$ if $S_{V_{i}\rightarrow V_{j}}<S_{V_{i}\rightarrow V_{j}}$.

Figure 1 shows the accuracy of different methods for identifying causal relationships. We observe that our method (the red bar) outperforms all other approaches with an accuracy of $0.694$. This indicates an improvement of 13% over the state-of-the-art feature-based methods (the blue bars), 17% over the distribution-based approaches (the orange bars), and 20% over the baseline methods (green bars). The baseline methods represented the worst performance, however, the accuracy achieved by precedence suggests that most of our corpus is written in form of active rather than passive voice, resulting in consequential connection between concepts.

To analyse the sensitivity of our method to the threshold $\mu$ in Equation 1, we trained the model on PsyCaus’s training set($D_{\text{tr}}$), and analysed the development set performance in terms of macro-averaged F1 with a range of values $[0,1)$ for $\mu$. As shown in Figure 2, F1 score reaches the maximum of 0.66 with $\mu=0.05$, well above a random classifier.

During the annotation process, we noticed that some concepts, e.g. eating disorder and emotion, may have bi-directional causal relations, depending on the context ($\text{eating disorder}\xleftrightarrow{\text{cause}}\text{emotion}$). We ran our model against these examples and found out that our approach is interestingly capable of identifying these relations as well. In Equation 1, approximation of absolute values of both operands in the negation to one indicates bi-directional causality. While a bi-directional causal relation cannot be presented in a CBN, as it is a directed graphical model, a decision tree can contain these types of information. In addition, some concepts, e.g. delusional disorder and displeasure, that have not been connected with any type of causality connectives were also accurately identified as causal relations. This is due to the hierarchical design of variable-values in our model.

In this paper we have presented a novel approach for identifying causal relationship between concepts. This approach enables machines to extract causality even between non-adjacent concepts. Hence, a significant improvement was delivered comparing to naive baselines. Furthermore, we represented the causal knowledge extracted from human-written language in form of Causal Bayesian Network. To the best of our knowledge this representation is novel. Having a Causal Bayesian Network can empower many downstream applications, including question-answering and reasoning. Among all applications, causal and counter-factual reasoning, which can be build on top of the outcome of this paper, may address some current hallmarks of Artificial Intelligence systems.