Learning Performance Maximizing Ensembles with Explainability Guarantees

First, we define the underlying task as the estimation of the function $f(x)=y$ where $x\in\mathcal{X},y\in\mathcal{Y}$. We also define observations as $z=(x,y)\in\mathcal{X}\times\mathcal{Y}=\mathcal{Z}$, the training dataset $D^{n}=\{z_{i}:i\in\{1,...,n\},z_{i}\in\mathcal{Z}\}$, and loss function for the underlying task $l_{U}:\mathcal{Y}\times\mathcal{Y}\rightarrow\mathbb{R}$. Next, we define the ensemble component models - first, the intrinsically explainable glass box model as $g:\mathcal{X}\rightarrow\mathcal{Y}$ and the alternate, black box model as $b:\mathcal{X}\rightarrow\mathcal{Y}$, both of which are learned independently on the full training dataset $D^{n}$.

Next, we define the allocation task. We define the class of all allocator functions as $A=\{a:\mathcal{V}\rightarrow\{0,1\}\}$ and the class of all “proper” allocator functions as $\hat{A}=\{a:\hat{\mathcal{V}}\rightarrow\{0,1\}\}$, where $\mathcal{V}$ is a general space of inputs (typically $\mathcal{Z}$) and $\hat{\mathcal{V}}\subseteq\mathcal{V}$ containing only information available at allocation time. Next we define the class of $q$-explainable allocators as $A_{q}=\{a_{q}:a_{q}\in A,\frac{1}{n}\sum^{n}_{i=1}a_{q}(v_{i})=q\}$ and the corresponding class of “proper” $q$-explainable allocators as $\hat{A}_{q}=\{a_{q}:a_{q}\in\hat{A},\frac{1}{n}\sum^{n}_{i=1}a_{q}(v_{i})=q\}$, for $q\in\mathcal{Q}=\{\frac{i}{n}:i\in\{1,...,n\}\}\subseteq[0,1]$, with $q$ being the explainability level. Note, the set $A$ is used to define the optimal allocator, whereas the set $\hat{A}$ is searched to obtain an estimator of this optimum.

We next define indicators of performance sufficiency. These functions $s:\mathcal{Z}\rightarrow\{0,1\}$ should be thought of as context-dependent indicators of whether performance within a region of the feature space is sufficiently high to use the model in question reliably for explanation. For classification tasks, we define performance sufficiency as $s^{\prime}_{f}(z)=\mathbb{I}{\{f(x)=y\}}$, and for regression tasks as $s^{\prime}_{f}(z)=\mathbb{I}{\{l_{U}(f(x),y)<\epsilon\}}$, with $f:\mathcal{X}\rightarrow\mathcal{Y}$. In practice $\epsilon$ should be selected based on problem-specific context, however, lacking such context in the regression experiments conducted for this study, $\epsilon$ was selected to be the lower of the average validation losses of $g$ and $b$, as a reasonable threshold for prediction correctness. These sufficiency indicators generate the following partition of the data: $Z^{\prime}_{0}=\{z:z\in\mathcal{Z},s^{\prime}_{g}(z)+s^{\prime}_{b}(z)=0\},Z^{\prime}_{2}=\{z:z\in\mathcal{Z},s^{\prime}_{g}(z)+s^{\prime}_{b}(z)=2\},Z^{\prime}_{g}=\{z:z\in\mathcal{Z},s^{\prime}_{g}(z)=1,s^{\prime}_{b}(z)=0\}$, and $Z^{\prime}_{b}=\{z:z\in\mathcal{Z},s^{\prime}_{g}(z)=0,s^{\prime}_{b}(z)=1\}$, with $n_{0}=|Z^{\prime}_{0}|,n_{2}=|Z^{\prime}_{2}|,n_{g}=|Z^{\prime}_{g}|,n_{b}=|Z^{\prime}_{b}|$, and $n_{q}=nq$.

Next we motivate the use of the sufficiency perspective. Sufficiency functions are critical for defining coherent allocations when, as is often the case, the absolute performance measures used to learn ensemble component models do not match allocation preference (e.g. loss minimization vs accuracy maximization). Consider the following constrained allocation decision in which only one observation can be allocated to $g$:

In this case, loss minimization dictates an allocation of $z_{1}$ to $g$ and $z_{2}$ to $b$, which would allocate $z_{2}$ to an insufficient prediction. Sufficiency maximization would however yield a more satisfactory allocation of $z_{2}$ to $g$ and $z_{1}$ to $b$. This example demonstrates the utility of sufficiency allocation - distinguishing between a case where the user is willing to sacrifice “a bit of performance” (as quantified by sufficiency) for explanation ($z_{1}$), and a case were even a small performance drop results in an explanation that is not sufficiently trustworthy to use ($z_{2}$).

In the next section, we define the objective of the allocation task and introduce our proposed approach for addressing it.

In the allocation task, the objective is to construct an allocator $a_{q}$ that will determine which model, either the explainable $g$ or the black box $b$, is used for prediction on any given observation $z$, in a manner that is optimal relative to the following criteria. Firstly, for any given explainability level $q$, the allocator should distribute observations in a way that maximizes sufficient ensemble performance, defined as $\bar{t}^{\prime}(a_{q})=\frac{1}{n}\sum_{i=1}^{n}s_{g}^{\prime}(z_{i})a_{q}(v_{i})+s_{b}^{\prime}(z_{i})(1-a_{q}(v_{i}))$. Secondly, and again for any $q$, the allocator should maximize sufficient explainable prediction $\bar{t}^{\prime}_{g}(a_{q})=\frac{1}{n}\sum_{i=1}^{n}s_{g}^{\prime}(z_{i})a_{q}(v_{i})$, i.e. the performance of the model $g$ on the subset of observations it has been allocated, subject to maintaining maximal $\bar{t}^{\prime}(a_{q})$. Finally, the allocator should also be consistent in its allocations across the values of $q$, meaning that if an observation is allocated to $g$ for a given $q$, it should remain allocated to the glass box for all higher explainability levels as well. Next, we define our allocator, and show that it meets the three criteria introduced above.

Our proposed allocation function is defined as $a^{\prime}_{q}(z)=\mathbb{I}\{r^{\prime}(z)>1-q\}$, where rescaled ranking $r^{\prime}(z)=\frac{\mathrm{rank}_{D^{n}}(\tilde{r}^{\prime}(z))}{n}$, and ranking $\tilde{r}^{\prime}(z)=2s^{\prime}_{g}(z)-s^{\prime}_{b}(z)-\sigma(l_{U}(b(x),y)-l_{U}(g(x),y))$, with $\sigma(x)=\frac{1}{1+e^{-x}}$.

The intuition behind the allocator is as follows. First, all observations are sorted in sufficient performance maximizing order, i.e. allocation of observations in $Z_{g}^{\prime}$ to $g$ is prioritized over allocation of observations in $Z_{2}^{\prime}$ and $Z_{0}^{\prime}$, which in turn are prioritized over $Z_{b}^{\prime}$. Next, observations are sorted in explainable sufficient performance maximizing order, i.e. $Z_{2}^{\prime}$ is prioritized ahead of $Z_{0}^{\prime}$ for allocation to $g$. Then, within each sufficiency category, observations are ordered in absolute performance maximizing order, i.e. observations with large relative performance of $g$ over $b$ are prioritized for allocation to $g$. Next, this ranking is normalized, yielding the glass box allocation desirability percentile $r^{\prime}$. An important feature of this percentile is that it is constant with respect to $q$, thus the optimal observations to allocate to $g$, for any level of $q$, are simply the $n_{q}$ most highly ranked, resulting in the allocator $a^{\prime}_{q}$.

Note that in the described methodology, sufficiency based allocation can be viewed as a generalization of allocation via absolute performance, and can thus be reduced to the latter by selecting either $s_{f}(z)=0$ or $s_{f}(z)=1$, $\forall z,f$.

Next, we state the optimality properties of the proposed allocator $a^{\prime}_{q}$. The proofs are available in the long form paper on arxiv.org in the Theoretical Results section of the Appendix.

Tabular data is used to evaluate the proposed methodology as it the setting for which the required intrinsically explainable glass box models are most readily available. Following the tabular data benchmarking framework proposed by (Grinsztajn, Oyallon, and Varoquaux 2022), we conduct experiments on a set of 31 datasets (13 classification, 18 regression). These datasets represent the full set of provided datasets with quantitative features less the four largest scale datasets (omitted due to computational limitations). These datasets are summarized in Table 4.

Each dataset is split (70%, 9%, 21%) into training, validation, and test sets respectively, following (Grinsztajn, Oyallon, and Varoquaux 2022). All features and regression response variables are rescaled to the range [-1,1].

Both glass box and black box models are learned on the full training dataset for each underlying task. For classification datasets, two types of glass box model are fitted, a logistic regression and a classification tree, as well as two types of black box model, a gradient boosting trees classifier and a neural network classifier. Analogously, for regression datasets, two types of glass box model are fitted, a linear regression and a regression tree, as well as two types of black box model, a gradient boosting trees regressor and a neural network regressor. In all cases, the architecture of the neural networks is the “Wide ResNet-28” model (Zagoruyko and Komodakis 2016) adapted to tabular data with the replacement of convolutional layers with fully connected layers.

An allocator is subsequently also learned on the full training dataset. Both gradient boosting trees regressors and neural networks are fitted as allocators for each allocation task.
For allocator training, the features $x$ are augmented with four additional constructed features, the predictions $g(x)$ and $b(x)$, and two distance measures $d(g(x),b(x))$ between them, the cross-entropy and MSE. In our experiments, inclusion of these features improved allocator learning - likely by removing the need for the allocator to attempt to learn these quantities on its own. Allocation performance is further improved by ensembling the feature-dependent learned allocator $a^{\prime}_{q}$ with a strong feature-independent allocator $a^{\prime\prime}_{q}$, where $a^{\prime\prime}_{q}(d(g(z),b(z)))=\mathbb{I}\{\frac{\mathrm{rank}_{D^{n}}d(g(z),b(z))}{n}>1-q\}$. $a^{\prime\prime}_{q}$ can be viewed as an “assume the black box is correct” allocation rule which is more likely to assign an observation to $g$ if the distance between the predictions of $g$ and $b$ is high. Which of the two allocators is used for a given $q$ is determined by their respective performances on the validation set.

Hyperparameter tuning for all models is done using 4-fold cross-validation, with the exception of the neural network tuning which is done using the validation set. A grid search is done to select the best hyperparameters for each model with search values available in Tables 5, 6, and 7.
Each glass box and black box model is tuned on the full set of hyperparameters each time it is replicated. The gradient boosting trees allocator models are retuned on the full hyperparameter set each time as well. The neural network allocator is not retuned however, and instead uses the optimal settings found in the fitting of the black box on each dataset.

We define the following metrics which are used to measure performance of our method. First we define the Percentage Performance Captured over Random (PPCR) for a given allocator as follows: $PPCR(a_{q})=\frac{AUC(a_{q})-AUC(r_{q})}{AUC(o_{q})-AUC(r_{q})}$ where $AUC(f_{q})$ is the area under the curve of function $f_{q}$ over all values of $q$ in its domain, $o_{q}$ is the oracle allocator which has perfect information on the whole dataset, and $r_{q}$ is the random allocator which selects a subset from the data being allocated uniformly at random. The PPCR metric is a percentage and represents the proportion of the oracle AUC, in excess of that also covered by random allocation, that the learned allocator is able to capture. Thus a value of zero indicates performance on par with $r_{q}$ and a value of one represents perfect allocation.

Next, we define the Percent Q Equal or Over Max (PQEOM) as the percentage of $q$ values for which the allocator is performing at least as well as the most accurate ensemble member model (i.e. $g$ or $b$) and Percent Q Over Max (PQOM) as the percentage of q values for which the allocator is performing better than the most accurate ensemble member model (i.e. $g$ or $b$).

Next, we define the Percent Contribution of Feature-dependent Allocator (PCFA) as the percentage of $q$ values for which the feature dependent allocator $a^{\prime}_{q}$ is used for allocation decisions as opposed to the feature-independent allocator $a^{\prime\prime}_{q}$. A value close to one indicates that $a^{\prime}_{q}$ is used often, while a value close to zero indicates it is $a^{\prime\prime}_{q}$ instead.

Next, we define the $95\%$ Threshold Q Max (95TQM) as the highest value of $q$ for which the ensemble performance meets or exceeds $95\%$ of the performance of the better of $g$ and $b$. Thus this is a measure of how much explainability can be utilized before the performance price becomes material.

Next, we define the maximum accuracy achieved by the allocator across all $q$ (Max Acc), and the highest value of $q$ for which this accuracy is maintained (Argmax $q$). The Max Acc can be benchmarked against the AUC, interpretable as the average accuracy across $q$. Each of these metrics is a percentage and higher values correspond with higher performance and higher explainability at this maximum performance level, respectively.

Finally, we define the accuracy with which the four sufficiency categories ($Z^{\prime}_{g}$, $Z^{\prime}_{b}$, $Z^{\prime}_{2}$, and $Z^{\prime}_{0}$) can be estimated as the sufficiency accuracy ($s$ Acc). The higher this accuracy, the better able the allocator is to inform the user of which category a given observation is likely to be a member of.

Evaluation of allocator performance using the metrics defined previously as well as visual inspection of the performance vs explainability trade-off curves (Fig. 2) revealed both the benefits and some of the limitations of learned allocation in the tabular data setting.

Firstly, performance was found to consistently and significantly outperform random allocation, as quantified by a cross-dataset PPCR of $37\%$ (Table 1), indicating that the learned allocation captured close to $40\%$ of the area under the curve available and in excess of random allocation. It was also found that on some datasets in particular, learned allocation performed close to oracle allocation (e.g. $89\%$ and $71\%$ on the IsoletR and BrazilianHousesR datasets).

Learned allocation was also found to perform at least at the level of the best ensemble member model across an average of $74\%$ of the explainability range (PQEOM in Table 1). This indicates that for many datasets, there is a substantial explainability “free lunch” to be taken advantage of without performance loss. On a few datasets, performance of the allocated ensemble was found to outperform both $g$ and $b$ for a majority ($93\%$) of the $q$ range (PolR and FifaR PQOM). The 95TQM metric also supported these conclusions, with a cross-dataset average value of $94\%$ indicating that allocation performance was within $5\%$ of maximal individual model performance across approximately all values of $q$.

Assessing the PCFA metric suggests some limits to the upside of learned, feature dependent allocation - at least in the tested tabular data setting. A cross-dataset average value of $35\%\pm 34\%$ indicates that on average, the range for which the feature dependent allocator is used over the feature-independent one is indistinguishable from zero. This is consistent with a visual inspection of the representative performance-explainability curve e.g. Fig. 2 (b) where there is no improvement to be had in excess of $a^{\prime\prime}_{q}$. However, it is noted that the only possibility for “homerun” allocations is through the feature dependent $a^{\prime}_{q}$ as seen in Fig 2 (a) with the PolR dataset and also in Table 1 for datasets SulfurR, BikeSharingR, and FifaR. Thus the ensembled allocation scheme offers this upside without downside risk of low performance in either $a^{\prime}_{q}$ or $a^{\prime\prime}_{q}$.

Evaluation of the case in which a single allocation is needed is also positive. On a cross-dataset average, the $84\%$ maximal accuracy achieved is quite high, and is also achieved at a high average explainability level ($64\%$). Particularly strong individual results can be seen in the Pol and SulfurR datasets (Table 1). We also find that on a observation-level, the allocation is an accurate estimator of sufficiency category, with a cross-dataset average of $76\%$ and with few datasets with accuracy under $60\%$.