An AI Architecture with the Capability to Explain Recognition Results

Fig. 1 depicts the explainable architecture with explainable and unexplainable contributions. Consider each horizontal element in the architecture incident on the Decision Making Process as a Pre-Decision Flow (PDF) of the explainable system. An explainable PDF would consist of one Property Transform and one Inference Engine (IE). An unexplainable PDF would consist of only one IE. The $1$ through $n$ explainable PDFs in the architecture are depicted as green while the single unexplainable PDF is represented in orange as $IE_{u}$. Explainable PDFs are related to properties that contribute to the explainability of the system. Often, explainable properties provide mediocre recognition results compared to an IE trained against the untransformed data. In many cases, this is because a property may not pertain to each class. E.g., the ellipse property does not apply to a well-formed digit one.

In this architecture, the unexplainable PDF is the untransformed input image. The unexplainable flow contributes excellent recognition performance but deters from the explainability of the system since the IE is considered an opaque box, without an explainable property for justification of a decision. An explainability metric is introduced to quantify the impact of unexplainable PDFs on decisions in Section III-B.

In Fig. 1, input flows into the system at the left. The initial stage of the architecture, for explainable PDFs, is the Transform phase. In the Transform phase, the input image is transformed according to the explainable property. Transforms emphasize particular explainable properties in the input and are essential to explainability.

The second stage of the architecture is the Property Inferencing stage. In Property Inferencing, IEs are used to make decisions on local data. IEs in this work are Multilayer Perceptrons (MLP) or Support Vector Machines (SVM). Explainable IEs are trained on transformed training data. The unexplainable IE, $IE_{u}$, is trained on untransformed input. Training data and training results are stored in a knowledgebase (KB) as depicted at the bottom of Fig. 1.

Local decisions from Property Inferencing are passed to Decision Making, where a global decision is made. This global decision is merely an ordering of the classes voted on based upon Confidence. The Confidence, $C_{d}$, is shown in (5) where $W_{d}$, in (6), represents the weighted effectiveness that voted for a particular class of $d$ over the sum of weighted effectiveness for all classes $k$ voted for. Effectiveness, $E_{j,d}$, indicates how well an IE is at recognizing a class and is obtained from data stored in the KB. Section III-C details effectiveness and outlines new methods of gauging effectiveness while Section IV-A provides results of the new effectiveness metrics.

The Explainability phase in the explainable Architecture, Fig. 1, composes the textual rationale, related to the explainable properties contributing to decisions. This is done by constructing phrases indicating the properties that voted for each class. The explainability measure, $Ex_{d}$ from (7), is also determined in this phase of the system. The measure indicates how explainable each decision for a class, $d$, is based on the flow’s relation to an explainable property. The user is finally presented with the ranked recognition results, the confidence of the decisions, the rationale for a decision, and the level of explainability for each decision. Sample explainable results are presented in Section IV-C.

The skeleton fill flow was not used due to poor performance, almost no $TP$s for nine out of ten MNIST classes, as observed in Table I. A new property representing corners or inflection points in a character was introduced. The corner transform utilizes the Harris corner detection algorithm [20].

Explainability, in the property-based system, is provided by explainable properties, used to justify a local decision in plain terms to a user. With the introduction of unexplainable flows, each flow, $j$, is assigned an explainability metric $X_{j}$, where $0\leq X_{j}\leq 1$. An $X_{j}$ close to zero would signify an unexplainable flow while an $X_{j}$ near one would indicate an explainable flow. Examples in this work use $X_{d}=1$ for explainable and $X_{d}=0$ for the unexplainable flows.

The explainability of a decision for class $d$ is given by $Ex_{d}$ in (7) where the numerator is the sum of the product of the explainability metric and effectiveness for the flows that voted for class $d$, and the denominator is the sum of effectiveness for flows that voted for class $d$.

An important area of the architecture, influencing the overall performance, concerns assigning weights to the IE votes. A key concept, related to assigning of weights to the IE votes, is the Effectiveness of an IE at recognizing a particular class. Effectiveness, $E_{j,d}$, was posed as a metric for how well the $j^{th}$ IE performs at recognizing class $d$ and is used to weight IE votes. Recall, given in (3), was previously used for Effectiveness. While Recall had acceptable results, the system exhibited false positives for some of the poor-performing flows.

It is not enough to know that an IE is effective overall. Effectiveness must be examined at the class level. This requires that metrics are taken as one class versus others. The one versus others strategy imposes a significant imbalance between the single class and others with the MNIST and EMNIST datasets [1, 2]. Some metrics are sensitive to imbalanced data and produce misleading results. This is evident in Table I where Accuracy ($ACC$), for nine out of ten classes, is very high, around 90% despite no $TP$.

Particular metrics from literature that were examined and evaluated as effectiveness in the explainable architecture include Accuracy, Recall, Specificity, Precision, F1-Score, Cohen’s Kappa, Matthews Correlation Coefficient, Balanced Accuracy, and the Area Under the Curve of the Receiver Operator Characteristic (AUC).

In addition to evaluating metrics from literature, some metrics were combined and compared. The combination of metrics found to have the best performance as per-class effectiveness was the product of class vs others Precision ($P$), Accuracy ($ACC$), Recall ($R$), and Specificity ($S$) as shown in (8) as the new effectiveness metric, $E_{PARS}$. Equation (9) shows the expansion of $E_{PARS}$ in terms of $TP$, $TN$, $FP$, and $FN$.

SVM IEs used the Support Vector Classification implementation and NN IEs used the MLP Classifier implementation in scikit-learn version 1.2.2 [21]. In the SVMs, radial basis function kernels were used as they performed better than alternative kernels with the transformed data. MLPs used two hidden layers of 128 neurons with a rectified linear unit activation function. This MLP architecture performed well on the handwritten data and was fast to train without specialized hardware.

Results shown in examples were obtained using discrete classification of the MLP and SVM models, which outputs either a one or zero, not continuous probability estimates p, where $0\leq p\leq 1$. The default behavior for the scikit-learn SVM model is discrete classification. When the SVMs were trained to enable probability estimates and the estimates were used along with effectiveness, there was much less variance between the metrics and higher system accuracy.

Adding an accurate, but unexplainable, classifier to the system improves performance. This is illustrated in Table II where combined results (E+U) are greater than the strictly explainable ($E$) results. A marked improvement in accuracy was shown by moving to combined explainable and unexplainable flows.

The combined (E+U) results were greater than explainable (E) in all cases. In the MNIST example using Recall as Effectiveness, the increase using MLP flows was over 15%. Typical increases with combined flows ranged from two to ten percentage points better than strictly explainable flows.

Some of the performance metrics from the literature, especially those that are resilient to imbalanced data such as Cohen’s Kappa, provide outstanding results as a measure of effectiveness in the explainable architecture. Surprisingly, Precision alone as well as the product of Precision and other metrics perform among the highest of those attempted. The best-performing metric found is given in (8) as $E_{PARS}$, the product of Precision, Accuracy, Recall, and Specificity. The previously reported accuracy observed, using the explainable architecture, on MNIST with MLPs was about 92% using Recall as Effectiveness and probability estimates. Employing $E_{PARS}$ as effectiveness and without using probability estimates, accuracy was observed at about 95.5%, an improvement of over 3%. When probability estimates are used with $E_{PARS}$ as effectiveness, accuracy was increased to 96.1%. Adding an unexplainable component to the system and using probability estimates increased accuracy results to as high as 98.0%.

Observing the numerator in (9), $E_{PARS}$ performs well because $TP$ to $TN$ balance is maintained since $TP^{3}$ ($\approx 2.2\text{x}10^{11}$ for MNIST and $\approx 1.4\text{x}10^{10}$ for EMNIST) and $TN^{2}$ ($\approx 2.7\text{x}10^{9}$ for MNIST and $\approx 1.2\text{x}10^{10}$ for EMNIST). A larger dataset with more classes may not result in similar performance.

This section presents an example and outlines how the explainable architecture comes to a decision. The example illustrates the confidence, rationale, and explainability provided to a user. Also included are results with differing effectiveness metrics. Where the terms low, medium, and high are used in this section, they denote below 25%, 25% to 75%, and above 75%, respectively.

Fig. III depicts the example, an EMNIST balanced test, handwritten letter ‘S’ sample (index 18) labeled as a capital ‘S’. The explainable architecture used for the example is composed of eleven PDFs with SVM IEs. Data from multiple effectiveness metrics (such as Recall, Accuracy, and $E_{PARS}$) will be presented to demonstrate how the explainable system benefits from more robust effectiveness metrics.

The results of processing the example using the Recall metric are in Tables III and IV, the results of using the Accuracy metric are depicted in Tables V and VI, and the results of using the $E_{PARS}$ metric are in Tables VII and VIII. Table IX illustrates the metric particulars for the PDFs in the example in one place. Note that four classes were voted on for the example. Each PDF’s vote and explainability metric is the same across the various tables. Only the effectiveness varies.

When examining the Explainability data from Tables III, V, and VII, each flow identifier, $F_{j}$, is in the first column where $j$ denotes the flow number. The second column indicates the property name. Flow $F_{11}$ is labeled as no property because it represents an unexplainable flow, without an explainable property. The Class Vote column indicates the class that was selected by each flow.

The remaining columns are related to effectiveness and explainability. Since four classes were voted on in this example, there will be four columns each for effectiveness and explainability, representing each class with a vote. The columns labeled $E_{j,d}$, represent the effectiveness of the flows for each class, $d$, that were voted upon. The columns labeled $Ex_{d}$ represent the explainability metric for each flow. Only the column representing the class that a flow voted for will have a value in the table for effectiveness and explainability. The last two rows of the Explainability tables represent the weights, of effectiveness and explainability, as well as the Confidence, $C_{d}$, and Explainability, $Ex_{d}$, for each class $d$.

Ordered explanation results are represented in Tables IV, VI, and VIII. The first column indicates the class, $d$. The second column the Confidence. The third column, represents the explainability associated with the decision. The final column is the rationale provided to the user.

Observe in Tables III and IV that the four PDFs that voted for class one had high Recall, reflected in the Effectiveness column $E_{j,1}$ of Table III in bold. This resulted in the digit one winning with medium confidence, $51.7\%$, due to the enclosed region, circle, ellipse-circle, and crossing properties. The second choice was the ‘S’ with medium confidence, $43.5\%$. Since Recall in (3) has $TP$ in the numerator, high $TP$ and very low $FN$ counts of the PDFs that voted for the digit one observed in Table IX explains why the digit one wins using the Recall metric.

Tables V and VI contain data and results from using the Accuracy metric on the example. Accuracy is given in (1). Due to the comparatively high $TN$ rate of the PDFs that voted for the ‘S’, as shown in bold in Table IX it wins with medium confidence, $51.0\%$, due to the stroke, corner, endpoint, and line properties. The ‘S’ also received a vote from the unexplainable PDF. The lower explainability metric, at $79.8\%$, reflects diminished explainability due to no explainable property associated with part ($20.2\%$) of the contribution for the decision.

The final set of Explainability and Explanations Tables, VII and VIII, involve the $E_{PARS}$ metric. The stroke and unexplainable flows have a much higher effectiveness than other metrics. High effectiveness from the Accuracy metric for the flows voting for the digit one were due to insensitivity of Accuracy to $FP$s. The same flows voting for the digit one have a reduced effectiveness due to relative sensitivity of $E_{PARS}$ to $FP$ counts, as observed in Table IX. The capital ‘S’  correctly wins with high confidence, $86.9\%$, using $E_{PARS}$.