Evaluation Metrics for Symbolic Knowledge Extracted from Machine Learning Black Boxes: A Discussion Paper

Artificial Intelligence (AI) for finance is a reality including, but not limited to, consumer credit, credit risk, and anti-money laundering applications (Aziz and Dowling, 2019). Usually, to obtain high predictive capabilities, machine learning techniques exploited for predictions rely on sub-symbolic techniques, that offer advantages related to performances but also come along with some drawbacks (Rocha et al., 2012). The main drawback lies in the opacity of these models, which are obscure in the decisions they make and lack human interpretability. For such a reason models of this kind are also called black boxes (BB).

When dealing with critical applications – such as the finance ones – lack of human awareness is even more unacceptable and makes BB models impracticable. Finance is a domain where trustworthiness must be ensured (Ng et al., 2015). Unique ethical, legal, and regulatory challenges arise as decisions can immediately impact people’s lives (Weiss, 2021). However, it is still unclear how to implement and test trustworthy AI systems in practice. Following the definition of the High-Level Expert Group on AI (European Commission et al., 2019), trustworthy AI should satisfy three main pillars: comply with all applicable laws and regulations (lawfulness), adhere to ethical principles and values (ethicality), and be safe, secure and reliable (robustness). However, how to translate these principles into practice is still an open question (Mittelstadt, 2019). A possible step towards trustworthy AI is to develop explainable AI (XAI), as explainability is one of the 7 more specific requirements that refine the main pillars (European Commission et al., 2019). XAI aims to create insight into how and why AI models produce predictions while maintaining high predictive performance levels.

Different strategies are present in the literature to tackle XAI via explainable models (Guidotti et al., 2018): (i) human-interpretable models, such as shallow decision trees (Rudin, 2019); or (ii) symbolic knowledge-extraction (SKE) techniques to the BB (Kenny et al., 2021). While i) is not always applicable, ii) is gaining a momentum. The amount of SKE algorithms available in the literature is constantly increasing, also thanks to the software support provided by object-oriented frameworks for machine learning as well as to the development of dedicated frameworks (e.g., PSyKE (Sabbatini et al., 2021b, 2022b, 2022a)). Amongst the application areas of SKE there are financial (Baesens et al., 2001, 2003; Steiner et al., 2006) and medical fields (Bologna and Pellegrini, 1997; Franco et al., 2007; Hayashi et al., 2000) and others (Setiono et al., 2011; Sabbatini and Grimani, 2022; Azcarraga et al., 2012; Hofmann et al., 2003). Examples of extractors generally achieving good results are Cart (Breiman et al., 1984), Trepan (Craven and Shavlik, 1996), GridEx (Sabbatini et al., 2021a) and GridREx (Sabbatini and Calegari, 2022). Of course a number of alternatives are present in the literature (Craven and Shavlik, 1994; Huysmans et al., 2006; Barakat and Diederich, 2008; Martens et al., 2007), depending on the task at hand.

A common trait of every SKE extractor is the output, which has to be some sort of human-interpretable knowledge. The knowledge usually follows a list or tree representation, where each item (leaf) of the list (tree) corresponds to a specific rule having predictive power. However, how to select the best SKE extractor for the case at hand – possibly as an automatic task – is still an open issue and the level of human readability of the extracted knowledge should be evaluated via a fair metric (Sovrano et al., 2022; De Mulder and Valcke, 2021). The level of human-readability of the extracted knowledge is bounded to the number of rules it contains and, recursively, to the readability of each rule, in turn. Indeed, rules may follow several formats with different levels of conciseness even when having the same semantics. For this reason, the quantitative assessment of knowledge readability – through an unbiased score and relative metric – is a quite challenging task. The lacking of such a metric prevents the design and implementation of procedures to automatically tune extractor parameters since an unbiased comparison of extractors’ outputs is not possible. Accordingly, in this paper we provide a detailed discussion about the readability assessment, highlighting the most challenging issues. The discussion is clustered based on the knowledge representation of the extraction algorithm.

Following research discussions on the topic (Garcez et al., 2001; Tran and Garcez, 2013), the most common indicators to evaluate the knowledge extracted via SKE techniques are: (i) the predictive performance of the knowledge rules w.r.t. the data or the BB predictions (the latter is called fidelity); (ii) the readability level from a human perspective; and (iii) the input space coverage (completeness). Having good scores according to these 3 dimensions ensures that the extracted knowledge represents well the underlying domain, that it is interpretable for humans, and that it may be used as a surrogate model to draw predictions.

It is quite easy to evaluate the knowledge predictive performance with the same scoring function adopted to evaluate the underlying BB. Adequate scoring metrics are, for instance, the accuracy and F₁ scores for classification tasks and the mean absolute/squared error and R² score for regression tasks. This indicator is pretty relevant since it represents the best way to assess if the knowledge is suitable to draw predictions in lieu of the BB.

The input space coverage can also be measured trivially, e.g., by observing how many data set instances are covered by the extracted rules, or by performing some kind of equal-spaced or random sampling inside the input space and then checking how many data points are covered. The latter solution is more demanding for high-dimensional data sets and may have little significance in the case of sparse data sets (characterised by empty regions). This indicator is also important because extremely human readable knowledge (e.g., composed of a single, simple rule) having excellent predictive performance (e.g., 100% accuracy) is useless if it covers an infinitesimally small region of the input feature space.

Human-readability assessment and evaluation could turn out not to be so trivial. First of all, we have to distinguish between two different levels of readability, both related to the presentation of the extracted knowledge to users. On the one hand, we can consider the macrolevel of readability of the knowledge as a whole, related to the knowledge shape and dimension – e.g., if it is represented as a tree, a list, a table, etc. – and to the number of leaves, rules, rows/columns present (aka complexity). On the other hand, we can focus on the microlevel of readability, to recursively take into account the same indicators – shape and complexity – for each element in the knowledge. As an example of the microlevel we can consider for instance the kind of preconditions of each element (e.g., oblique rules, M-of-N tree nodes) and how many variables, constants, and predicates appear inside them. As can be noticed, this is far from being a trivial task, since equivalent concepts may be represented according to different notations, each one having its conciseness and expressiveness. In the following, both macrolevel and microlevel are examined in detail, showing that the most challenging issues are related to the latter.

To show in practice the differences amongst the output of different knowledge extractors, we applied the Cart, CReEPy, GridEx and GridREx extractors to the Istanbul Stock Exchange data set¹¹1https://archive.ics.uci.edu/ml/datasets/ISTANBUL+STOCK+EXCHANGE [Online; last accessed August 24, 2022], reported in Figure 1a. The data set is composed of 8 continuous features (7 as input + 1 as output). We considered all of them during the training phase of black boxes and extractors, however, for the sake of readability, we reported in Figures 1 and 2 only the most relevant two—i.e., the MSCI European index (EU) and the stock market return index of UK (FTSE).

The selected extractors have been applied to a linear regressor (LR) and a random forest (RF) BB. Mean absolute error (MAE) and R² score of the BB predictions are reported in Table 1. Corresponding decision boundaries are visually represented in Figures 1b and 1c. Predictions obtained through the extracted knowledge have been evaluated with the same scores adopted for the underlying BB outputs and are also reported in Table 1. The corresponding decision boundaries are shown in Figures 2a to 2l.

Considerations about the macrolevel knowledge readability can be drawn by analysing the plots in Figure 2. First of all, it is noticeable that Cart, GridEx and CReEPy with constant outputs introduce an undesired discretisation in the output predictions, thus implying a worsening in the overall predictive performance. Conversely, GridREx and CReEPy with linear outputs can provide linear combinations of the input variables as output predictions, resulting in knowledge having fewer rules with smaller predictive errors.

Observing the same plots it is noticeable that CReEPy outputs a hierarchical knowledge, thus mappable in an ordered list of rules. The same holds for Cart’s knowledge – if simplified – since it is based on a decision tree. Differently, GridEx and GridREx provide rules associated with hypercubic and non-overlapping regions, so they are not ordered and each one contains the same amount of variable occurrences, predicates (i.e., interval inclusion), and constant values (i.e., 2 per variable).

Macrolevel readability comparisons can be easily performed by observing the ‘Rules’ column in Table 1. GridREx is the best choice since it outputs a single rule, followed by the CReEPy. Their superiority is confirmed also from the predictive performance standpoint, since they have the highest R² score. GridREx doesn’t meet the completeness requirement, i.e., results are non-exhaustive in the input space coverage since a portion is considered negligible due to the absence of training instances (so there are test instances that cannot be predicted). This should penalise the final metric score.

To perform microlevel comparisons it is necessary to observe the output knowledge rules provided by the various extractors. We exemplify the difference between constant and linear outputs by considering the knowledge extracted via Cart and CReEPy from the linear regressor. Cart trained with a maximum of 6 leaves provides the following rules:

Cart rules may be simplified in the following without modifying their semantics:

Rules extracted with CReEPy (linear output) for the Istanbul Stock Exchange data set are the following:

On one hand, the two versions of the Cart’s knowledge are equivalent in terms of predictive accuracy and input space coverage, i.e., they provide the same predictions. However, the first has redundant literals, whereas the second has implicit constraints. This means that, when measuring the knowledge microlevel readability, the first should be penalised for the presence of literals in single clauses that are trivially true. For instance, the conjunction EU > -0.00, EU > 0.01, EM =< 0.01 hinders readability if compared to the equivalent EU > 0.01, EM =< 0.01. Vice versa, the simplified version of the same rule represented as EM =< 0.01 implies implicitly acknowledging as false the constraint EU =< 0.01, requiring humans to check all the previous rules. An ideal microlevel readability metric should be able to assign a good score to EU > 0.01, EM =< 0.01 as well as penalise the redundant and implicit alternatives.

On the other hand, by comparing Cart’s and CReEPy’s extracted knowledge it is clear that they exhibit (i) exactly the same input space coverage; (ii) similar predictive performance (CReEPy is slightly better in the fidelity w.r.t. the BB predictions); (iii) different macrolevel readability, since CReEPy extracts only 2 rules instead of 6, as Cart; (iv) different microlevel readability, since CReEPy’s predictions may be constants as well as linear combinations of up to 7 variables. An ideal microlevel readability metric should be able to assign a score taking into account these differences. The lack of a quantitative metric to evaluate the microlevel readability – and thus the overall knowledge readability and, in turn, quality – makes it impracticable to automatically decide whether longer rule lists with simple outputs are preferable or not over shorter lists with more complex outputs.

In this paper we discuss the major issues in defining possible metrics to evaluate in a quantitative way the quality of the symbolic knowledge extracted via SKE techniques out of a BB model as well as the need to define such metrics. We show that quality evaluation should consider different indicators, namely: i) predictive performance, ii) input space coverage and iii) knowledge readability. While for the first two the definition of a metric and the comparison of a score is quite trivial, this does not happen for the last one. As for the latter, readability can be observed at a macrolevel – as the shape and complexity of the knowledge as a whole – and at a microlevel—as shape and complexity of each knowledge rule. We showed that at a macrolevel the knowledge may be easily compared since it is quite easy to convert one format into another. Conversely, we pointed out the most tricky characteristics of microlevel considerations, to favour further studies and discussion on the topic.

Future works will be devoted to the definition of such a suitable metric to assess the quality of knowledge extracted via SKE in a fair, quantitative, and measurable way. In particular, we plan to formulate a sound scoring function for microlevel comparisons and to study how the user interaction may help the customisation and integration of the 3 aforementioned indicators in a unique and effective metric assessing the knowledge quality.

Moreover, adherence to the trustworthy AI requirements should be enlarged to evaluate also other requirements and, in particular, our findings might be subject to change due to the institutional debate about the AI Act Proposal. Further research is needed to consolidate the interpretation of the Act in light of its future changes and to define metrics that can be used in the course of the standardisation process while being respectful of the AI Act and its Annexes.

Another open question left to further research is to take into account that the definition of explainability metrics may lead to consider concurring features that are compliant with the legislative goals of the AI Act and, eventually, the thresholds that meet the Act’s requirements and expectations. Each feature needs to be balanced via parameters that can be set taking into consideration both the scenario at hand and its regulation. This is the first step for converging to trustworthy automated machine learning techniques that we believe to be promising in the near future.