Counterfactual Explanations of Black-box Machine Learning Models using Causal Discovery with Applications to Credit Rating
^{* ††}thanks:

Structural causal model (SCM) [12] is a mathematical framework for handling causal relationships. SCM comprises a set of endogenous variables $\mathbf{V}=\{V_{1},V_{2},...,V_{p}\}$, a set of exogenous variables $\mathbf{U}=\{U_{1},U_{2},...,U_{q}\}$, which are variables not determined by the endogenous variables, and a set of functions $\mathbf{F}=\{f_{1},f_{2},...,f_{p}\}$, which determines the values of endogenous variables from those of other endogenous and exogenous variables. In addition, we define the parent set of endogenous variables as $\mathrm{pa}(V_{i})$ and we assume variables in $\mathbf{U}$ are independent. Consequently, if SCM is specified by

and the system is assumed to be autonomous, Equation (1) is called a structural causal model, wherein the quantitative causal relations of variables are expressed by a directed graph called a causal graph. Here, autonomy implies that changing any function or distribution of variables does not change the distribution of other functions or distributions.

LEWIS [10] is an XAI method that provides counterfactual explanations for machine learning predictions based on structural causal models. Consider the following binary classification problem. The explanatory variable is $X\in\mathbf{V}$ and the value of the explanatory variable is $\{x,x^{\prime}\}\in X$ $(x>x^{\prime})$, wherein all the values of the explanatory variables are discretized. Let $O\in\{o,o^{\prime}\}$ be the predicted value corresponding to the explanatory variable, where $o$ is the positive predicted value and $o^{\prime}$ is the negative predicted value. The counterfactual in any causal model $M=\langle V,U,F\rangle$ can be defined for a certain individual $u\in U$ as [12]

where $Y_{X=x}$ is the counterfactual of the value of $Y$ if the value of $X$ were $x$, with $Y_{M_{x}}$ being that of $Y$ if the value of $X$ were $x$ in the causal model $M$. Further, the counterfactual probability $P(Y_{X=x})$ can be expressed as $P(Y|\mathit{do}(X=x))$ using the intervention symbol $\it{do}$. It represents the conditional probability of $Y$ if the value of $X$ were changed to $x$ and can be computed based on a causal graph.

In this setting, LEWIS defines the following three explanatory scores as the probability of a counterfactual, necessity score (Nec)

sufficiency score (Suf)

and necessity and sufficiency score (Nesuf)

The necessity score and sufficiency score (called reversal probability) calculate the probability of the degree of change in the predicted value if the explanatory variable had a different value, given the value and predicted value of a given explanatory variable. In particular, the necessity score represents the probability that the predicted value would change if the value of the explanatory variable decreased, whereas the sufficiency score represents the probability that the predicted value would change if the value increased. The necessity and sufficiency score is a score that balances the necessity score and sufficiency score and can be considered as a feature importance in machine learning [10].

In [10], LEWIS was used to examine a binary classification model that determines whether a loan was approved or rejected. For example, $\text{Nec}=0.1$ for a feature implies that if a person whose loan is predicted to be approved decreased the value of a feature, there would be at most a 10% chance of a prediction for the loan to be rejected. In addition, $\text{Suf}=0.8$ implies that if a person whose loan is predicted to be rejected increased the value of a certain feature, there is a maximum probability of 80% that the loan would be predicted to be approved.

These three scores are expressed as

if the causal graph $G$ corresponding to the causal model $M$ is known and the monotonicity $(x>x^{\prime}\Rightarrow O_{x}>O_{x^{\prime}})$ is satisfied. In this case, the global explanation score of LEWIS, $\mathrm{maxNesuf}(X)$, is given by the maximum value of $\mathrm{Nesuf}(x,x’)$ for all pairs of the value of the explanatory variable, $(x,x’)$. Finally, if no causal graph is provided to LEWIS, it assumes that $P(O|\mathit{do}(X))=P(O|X)$ under the assumption that there are no confounding factors, which is likely to be violated.

In Section III-A, we observed that the score differed greatly depending on whether the explanatory variable could yield the predictive variable. In particular, all explanatory scores will be 0 for the explanatory variables that are independent of the target variable. In addition, in case of a directed edge, or reverse causation, from the target variable to the explanatory variable, the target variable and explanatory variable are dependent. However, even if the value of the explanatory variables were changed, the value of the target variable would not be changed; thus, so the value of Nesuf becomes 0, and Suf and Nec provide no information. Therefore, we proposed the following prior information on the causal structure to compute the explanation scores:

1. (a)

   Target variable has the direct parent-child relationship with all explanatory variables, that is, there is a direct causal path from the explanatory variables to the target variable.

2. (b)

   Target variable is the sink variable, where a sink variable is a variable that does not cause any variable.

With prior information (a) (Figure 3a), the target variable is dependent on all explanatory variables. In this prior information, the estimated score is adjusted by variables that satisfy the backdoor criteria; if they don’t satisfy it, the intervention query is directly estimated, as in prior work. Hence if the explanatory variables and target variable are not independent, the estimated score will be at least the same as when no causal graph is used. This facilitates the performing of causal discovery using only explanatory variables. In prior information (b) (Figure 3b), we considered the influence of variables independent of the target variable and reverse causation, which could not be identified in prior information (a). In this prior information helps eliminate reverse causality from the target variable to the explanatory variables when performing causal discovery including the target variable.

In the numerical experiment, we generated artificial data from the 8-variable causal graph shown in Figure 4, performed causal discovery based on prior information on the causal structure proposed in the previous section, and estimated the Nesuf for the predicted values. We performed evaluations considering the order of estimated the Nesuf and that of true Nesuf and the error of the estimated Nesuf value. We compared these results with the value of Nesuf estimated assuming that $P(Y|\mathit{do}(X))=P(Y|X)$, similar to that in case of [10]. The artificial data to be generated were continuous and mixed data. Continuous variable values were generated from linear or non-linear function systems. For mixed data, only a linear system was used, the probability distribution of the error variables $X_{1}$ and $X_{7}$ was a Bernoulli distribution, and the probability distribution of the other error variables were a uniform or Gaussian distribution. The model, sample size, discretization method, and the number of discretization bins are presented in Table I.

For the evaluation, we calculated the mean absolute error (MAE) with respect to the true Nesuf estimate. We used $\overline{\mathrm{MAE}}$, which is the average of the MAE

for seven variables, $X_{1},\,\ldots,\,X_{7}$, where $N$ is the number of experimental trials, $\mathrm{maxNesuf}_{X_{i},\mathrm{true}}$ is the true value of $\mathrm{maxNesuf}(X_{i})$, and $\mathrm{maxNesuf}_{x_{i},\mathrm{est}}$ is the estimated value of $\mathrm{maxNesuf}(X_{i})$. The standard error of $\mathrm{maxNesuf}(X_{i})$ was also evaluated.

Here, we assume that $\mathbf{a}=\{a_{1},\ldots,a_{n}\}$ and $\mathbf{b}=\{b_{1},\ldots,b_{n}\}$ convert the value of the true Nesuf and the estimated Nesuf into a rank for $n$ variables, respectively. Defining $\mathbf{d}=\{d_{1},\ldots,d_{n}\}$ by $d_{i}=a_{i}-b_{i}$, we can calculate Spearman’s rank correlation coefficient [13] with

where $\mathrm{SPR}$ attains the value $-1\leqq\mathrm{SPR}\leqq 1$. The closer it is to 1, the more the order of the estimated Nesuf matches the true order of feature importance. Moreover, the closer it is to $-1$, the more the order relationship is reversed. In this experiment, the evaluation was based on the average MAE, the standard error and the average $\overline{\mathrm{SPR}}$ of 100 trials.

Next, for causal discovery, we used six methods tailored to specific functional forms. We implemented the causal discovery algorithm in Python. We used the causal-learn [14] method for PC, linear non-Gaussian acyclic model (LiNGAM) package [15] for DirectLiNGAM, RESIT, and linear mixed (LiM) and causalnex[16] method for NOTEARS and NOTEARS-MLP.

• •

  PC [17]: The PC algorithm is a causal discovery algorithm based on conditional independence. The estimation algorithm first removes undirected edges by testing conditional independence. The direction of the causal relationship is determined by using v-structures [12] and orientation rule [18] for the remaining undirected skeleton. When using a PC to orient undirected edges, the directions may not be determined in the end, and the graph may be estimated as a partially oriented graph. In that case, our evaluation was based on the causal graph with all directed edges derived from the partially oriented graph that had the maximum (PC_Max) or minimum (PC_Min) Spearman’s rank correlation coefficient. We used the Fisher-$z$ test to investigate the conditional independence.

• •

  DirectLiNGAM [19]: DirectLiNGAM is a causal discovery algorithm that expresses causal relationships using a linear structural equation model called LiNGAM [20], which assumes that the graph is acyclic and that the probability distribution of the error terms is non-Gaussian. DirectLiNGAM estimates the causal structure by repeated regression and evaluating the independence of residuals of each variable.

• •

  RESIT [21]: RESIT is a causal discovery algorithm used when the causal structure is nonlinear and the error variables are additive (Additive Noise Model: ANM). Similar to DirectLiNGAM, the algorithm estimates the causal direction by evaluating the independence of each variable and the residual of nonlinear regressions.

• •

  LiM [22]: LiM causal discovery algorithm extends LiNGAM to handle the mixed data that comprises both continuous and discrete variables. The estimation is performed by first globally optimizing the log-likelihood function on the joint distribution of data with the acyclicity constraint and then applying a local combinatorial search to output a causal graph.

• •

  NOTEARS [23]: NOTEARS reformulates DAG structure learning as a continuous optimization problem over real matrices, avoiding combinatorial acyclicity constraints. It introduces a smooth characterization of acyclicity as an equality constraint $h(W)=0$ on the weighted adjacency matrix W. This equality-constrained program is solved using augmented Lagrangian methods and numerical optimizers. This method assumes that the probability distribution of the error term has equal variance. Empirically, it outperforms state-of-the-art methods, especially for linear.

• •

  NOTEARS-MLP[24]: NOTEARS-MLP is that DAG structure learning algorithm extends NOTEARS to handle non-linear functional relationships using MLP which consists of hidden layer units and an activation that sigmoid function. Especially when the causal structure is an additive noise model, this method is identifiable that assuming the nonlinear function are three times differentiable and not linear in any of its arguments.

The following estimates were based on either prior information (a), prior information (b), or no prior information, denoted by (a), (b), and (0), respectively. PC and DirectLiNGAM can incorporate prior information (a) and (b). However, RESIT, LiM, NOTEARS and NOTEARS-MLP cannot incorporate prior information (b). In addition, we evaluated cases wherein causal discovery was performed without a causal graph (No graph). Table IV presents the experimental settings used in the numerical experiment.

The experimental results of $\overline{\mathrm{MAE}}$ $\pm$ standard error and $\overline{\mathrm{SPR}}$ are presented in Tables V–X. First, regarding the linear causal structure in Tables V and VI, when the causal discovery matched the true graph, the $\overline{\mathrm{MAE}}$ approached 0 and the $\overline{\mathrm{SPR}}$ approached 1. DirectLiNGAM assumed that the functional relationship was linear and that the probability distribution of the error terms was non-Gaussian. In such cases, $\overline{\mathrm{MAE}}$ was smaller and the $\overline{\mathrm{SPR}}$ was greater than that in the case No graph (Table V). As shown in Table VI, when a Gaussian distribution was assumed with no prior information or No graph, the $\overline{\mathrm{MAE}}$ was large and the $\overline{\mathrm{SPR}}$ was small for DirectLiNGAM. However, these scores were improved with prior information (a) and (b). Furthermore, the PC and NOTEARS algorithm consistently achieved high accuracy for both distributions. In all cases with a linear structure, we could estimate the true value and order of Nesuf scores better than that case when No graph was assumed.

Tables VII and VIII present the results in the case of a nonlinear causal structure. When the error term was uniform distribution, RESIT with no prior information and PC with prior information yielded smaller $\overline{\mathrm{MAE}}$ values and greater $\overline{\mathrm{SPR}}$ values than that for No graph in Table VII. For the Gaussian distribution causal structure as shown in Table VIII, NOTEARS-MLP with prior information had the highest performance. However, in this experimental setup, the performance of all methods was lower compared to other setups. The reason may be that monotonicity, another assumption of LEWIS, is not satisfied in nonlinear cases.

Finally, Tables IX and X present the results of the mixed data. In Table IX, the assumptions of LiM were satisfied, thus, the $\overline{\mathrm{MAE}}$ was smaller and the $\overline{\mathrm{SPR}}$ was larger than that when no causal discovery was performed. In addition, in Table X, the LiM assumption that the probability distribution of the error terms was non-Gaussian was not satisfied, the score was the same as the case of No graph. Even in the case of linear mixed data, it can be said that the true Nesuf can be partially restored using the proposed prior information of causal structure.

This result confirms that prior information (a) is at least the same and more than that as when no causal graph is used. Futhermore prior information (0) and (b) provide relatively high performance compared to the case with No graph. On the other hand, depending on the nature of the data and the causal discovery method, it may be worse than the case with No graph, so it is necessary to consider multiple causal discovery methods.

We applied our method to the anonymized credit rating data of $14\,018$ business customers provided by the Shiga Bank, Ltd. A credit rating was assigned by a bank to a debtor based on the analysis of its financial statements [25]. Although there were several grades in the credit rating, we simplified the grades to high (excellent) and low (poor), which facilitated its modeling as a binary classification problem. We used the industry type, amount of capital stock, number of employees, most recent annual sales, and total liabilities and equity as the explanatory variables. We performed equal-frequency discretization with 10 bins because the capital stock, number of employees, most recent annual sales, and total liabilities and equity had highly skewed distributions. These discretized variables are ordinal measures with enough many levels and can be seen as continuous variables. As a machine learning model, we used Random Forest, which is an algorithm that performs classification by combining the results of multiple decision trees constructed from randomly selected learning data and explanatory variables; it is an ensemble learning algorithm [26]. Random Forest was expected to have sufficient predictive accuracy for real data in this study.

Although the data involved mixed data where only the industry type is discrete variable and the others are regarded as continuous variables, the industry type could not be caused by the other explanatory variables. Thus, we used DirectLiNGAM for continuous variables, assuming that industry type was an exogenous discrete variable that affected all other variables. In this case, DirectLiNGAM can handle discrete variable similarity to the conventional structural equation modeling [27]. Because we aimed at a quantitative analysis of all explanatory variables, we conducted our analysis assuming that the target variable exhibited a direct parent-child relationship with all explanatory variables in the causal structure, that is, prior information (b).

The black lines of Figure 5 show the results with DirectLiNGAM for the entire dataset. The interpretation of the causal direction of the results is as follows. The causal direction from capital stock to total liabilities and equity was consistent with domain knowledge because the total liabilities and equity are the sum of debt and equity on the balance sheet. In addition, the causal direction from capital stock to sales was consistent with the domain knowledge that companies conducted business activities based on capital and that this influenced sales. These variables can affect the credit rating on the basis of prior information of the causal structure in Figure 5 (red lines).

Next, Nesuf estimated using the causal graph and No graph is shown in Figure 6. The importance ranking of each of these variables obtained by LEWIS can be explained as follows using the causal graph shown in Figure 5. The industry type was an exogenous variable that affected the other four variables on the causal graph. If the industry type were to be changed, the values in the other four variables would also change, which is likely to affect the final predicted value. Thus, its importance in LEWIS is the greatest. Similarly, the value of capital stock changed the value of the sales and total liabilities and equity, and the total liabilities and equity changed the value of sales. The number of employees and sales were less important because changing these values did not change the values of other variables. However, the Nesuf scores estimated with No graph were smaller, and those of the capital stock in particular were significantly smaller, which contradicted domain knowledge.

The LEWIS reversal probability score is shown in Figure 7. Overall, the Suf score (orange) was high, and there was a high probability that changing the value of the variable would change the company from a low-rated one to a high-rated one. Industry type also significantly affected the predicted value if it were changed. Reversal probabilities can assist the decision-making about which variables are likely to change the rating for a low-rated company and can quantify the company’s strengths and weaknesses based on each score.