Deletion and Insertion Tests in Regression Models

Explainable AI methods are used help humans learn from patterns that a machine learning or artificial intelligence model has found, or to judge whether those patterns are scientifically reasonable or whether they treat subjects fairly. As Hooker et al., (2019) note, there is no ground truth for explanations. Mase et al., (2022) attribute this to the greater difficulty of identifying causes of effects compared to effects of causes (Dawid and Musio,, 2021). Lacking a ground truth, researchers turn to axioms and sanity checks to motivate and vet explanatory methods. There are also some numerical measures that one can use to compute a quality measure for methods that rank variables from most to least important. These include the Area Over Perturbation Curve (AOPC) of Samek et al., (2016) and the Area Under the Curve (AUC) measure of Petsiuk et al., (2018) that we focus on. They have the potential to augment intuitive and philosophical distinctions among methods with precise numerical comparisons. In this paper we make a careful study of the properties of those measures and we illustrate their use on two datasets.

Insertion and deletion tests were used by Petsiuk et al., (2018) to compare variable importance methods for black box functions. In their specific case they had an image classifier that would, for example, conclude with high confidence that a given image contains a mountain bike. Then the question of interest was to identify which pixels are most important to that decision. They propose to delete pixels, replacing them by a plain default value (constant values such as black or the average of all pixels from many images) in order from most to least important for the decision that the image was of the given class. If they have ordered the pixels well, then the confidence level of the predicted class should drop quickly as more pixels are deleted. By that measure, their Randomized Input Sampling for Explanation (RISE) performed well compared to alternatives such as GradCAM (Selvaraju et al.,, 2017) and LIME (Ribeiro et al.,, 2016) when explaining outputs of a ResNet50 classifier (Zhang et al.,, 2018). For instance, Figure 2 of Petsiuk et al., (2018) has an example where occlusion of about 4% of pixels, as sorted by their RISE criterion can overturn the classification of an image. They also considered an insertion test starting from a blurred image of the original one and inserting pixels from the real image in order from most to least important. An ordering where the confidence rises most quickly is then to be preferred. Petsiuk et al., (2018) scored their methods by an area under the curve (AUC) metric that we will describe in detail below. The idea to change features in order of importance and score how quickly predictions change is quite natural, and we expect many others have used it. We believe that our analysis of the methods in Petsiuk et al., (2018) will shed light on other similar proposals.

Figure 1 shows an example where an image is correctly and confidently classified as an albatross by an algorithm described in Appendix A.1. The integrated gradients (IG) method of Sundararajan et al., (2017) that we define below can be used to rank the pixels by importance. In this instance the deletion AUC is $0.27$ which can be interpreted as meaning that about 27% of the pixels have to be deleted before the algorithm completely forgets that the image is of an albatross. The model we used accepts square images of size $224\times 224$ pixels with 3 color channels. As a preprocessing step, we cropped the leftmost image in Figure 1 to its central $224\times 224$ pixels. The IG feature attributions from each pixel (summed over red, green and blue channels) of this square image are presented in the rightmost panel of Figure 1. A saliency map shows that pixels in the bird’s face, especially the eye and beak are rated as most important.

In this paper we study insertion and deletion metrics for uses that include regression problems in addition to classification. We consider a function $f(\bm{x})$ such as an estimate of a response $y$ given $n$ predictors represented as components of $\bm{x}$. The regression context is different from classification. The trajectory taken by $f(\cdot)$ as inputs are switched one by one from a point $\bm{x}$ to a baseline point $\bm{x}^{\prime}$ can be much less monotone than in the image classification problems that motivated Petsiuk et al., (2018). We don’t generally find that either $f(\bm{x})$ or $f(\bm{x}^{\prime})$ is near zero. There are also use cases where $\bm{x}$ and $\bm{x}^{\prime}$ are both actual observations; it is not necessary for one of them to be an analogue of a completely gray image or otherwise null data value. Sometimes $f(\bm{x})\approx f(\bm{x}^{\prime})$ and yet it can still be interesting to understand what happens to $f$ when some components of $\bm{x}$ are changed to corresponding values of $\bm{x}^{\prime}$.

Our main contributions are as follows. Despite these differences between classification and regression, we find that insertion and deletion metrics can be naturally extended to regression problems. We then develop expressions for the resulting AUC in terms of certain main effects and interactions in $f$ building on the anchored decomposition from Kuo et al., (2010) and others. This anchored decomposition is a counterpart to the better known analysis of variance (ANOVA) decomposition. The anchored decomposition does not require a distribution on its inputs, much less the independence of those inputs that the ANOVA requires. In the regression context we prefer to change the AUC computation replacing the horizontal axis by a straight line connecting $f(\bm{x})$ to $f(\bm{x}^{\prime})$. We obtain an expression for the average AUC in a case where variables were inserted in a uniform random order over all possible permutations. In settings without interactions the area between the variable change curve (that we define below) and the straight line has expected value zero under those permutations, but interactions change this. We also show that the expected area between the insertion curve and an analogous deletion curve that we define below does have expected value zero, even in the presence of interactions of any order. Some other contributions described below show that in some widely used models the same ordering that optimizes an area criterion also optimizes a Shapley value.

We take a special interest in the integrated gradients (IG) method of Sundararajan et al., (2017) because it is very fast. The number of function or derivative evaluations that it requires grows only linearly in the number of input variables, for any fixed number of evaluation nodes in the Riemann sum it uses to approximate an integral. The cost of exact computation for kernel SHAP (KS) of Lundberg and Lee, (2017) grows exponentially with the number of variables, although it can be approximated by sampling. We also include LIME of Ribeiro et al., (2016), DeepLIFT of Shrikumar et al., (2017), Vanilla Grad of Simonyan et al., (2013) and input times gradient method of Shrikumar et al., (2016). In the datasets we considered, KS is generally best overall. We note that the term ‘Vanilla’ is not used by Simonyan et al., (2013) to describe their methods but it has been used by others, such as Agarwal et al., (2022).

It is very common for machine learning functions to include binary inputs. For this reason we discuss how to extend IG to handle some dichotomous variables and then compare it to the other methods, especially KS. Simply extending the domain of $f$ for such variables from $\{0,1\}$ to $[0,1]$ is easy to do and it avoids the exponential cost that some more principled choices have.

The remainder of this paper is organized as following. Section 2 cites related works, introduces some notation and places our paper in the context of explainable AI (XAI), while also citing some works that express misgivings about XAI. Section 3 defines the AUC and gives an expression for it in terms of main effects and interactions derived from an anchored decomposition of the prediction function $f$. The expected AUC is obtained for a random ordering of variables. We also introduce an area between the curves (ABC) quantity using a linear interpolation baseline curve instead of the horizontal axis. We show that arranging input variables in decreasing order by Shapley value does not necessarily give the order that maximizes AUC, due to the presence of interactions. Models that, like logistic regression are represented by an increasing function of an additive model do get their greatest AUC from the Shapley ordering depsite the interactions introduced by the increasing function. When that function is differentiable, then IG finds the optimal order. Section 4 discusses how to extend IG to some dichotomous variables. We consider three schemes: simply treating binary variables in $\{0,1\}$ as if they were continuous values in $[0,1]$, multilinear interpolation of the function values at binary points, and using paths that jump from $x_{j}=0$ to $x_{j}=1$. The simple strategy of casting the binary inputs to $[0,1]$, which many prediction functions can do, is preferable on grounds of speed. Section 5 presents some empirical work. We choose a regression problem about explaining the value of a house in Bangalore using some data from Kaggle (Section 5.1). This is a challenging prediction problem because the values are heavily skewed. It is especially challenging for IG because all but two of the input variables are binary and IG is defined for continuous variables. The model we use is a multilayer perceptron. We compare variable rankings from six methods. KS is overall best but IG is much faster and nearly as good. Section 5.2 looks at a problem from high energy physics using data from CERN. Section 6 includes a ROAR analysis that compares how well the KS and IG measures rank the importance of variables in a setting where the model is to be retrained without its most important variables. Section 7 has some final comments. Appendix A gives some details of the data and models we study. Theorem 1 is proved in Appendix B.

The insertion and deletion measures we study are part of XAI. Methods from machine learning and artificial intelligence are being deployed in electronic commerce, finance and other industries. Some of those applications are mission critical (e.g., in medicine, security or autonomous driving). When models are selected based on their accuracy on holdout sets, the winning algorithms can be very complicated. There is then a strong need for human understanding of how the methods work in order to reason about their accuracy on future data. For discussions on the motivations and methods for XAI see recent surveys such as Liao and Varshney, (2021), Saeed and Omlin, (2021) and Bodria et al., (2021).

One of the most prominent XAI tasks is attribution in which one quantifies and compares importance of the model inputs to the resulting prediction. Since there is no ground-truth for explanations, these attributions are usually compared based on theoretical justifications. From this viewpoint, SHAP (SHapley Additive exPlanations) (Lundberg and Lee,, 2017) and IG (Sundararajan et al.,, 2017) are popular feature importance methods due to their grounding in cooperative game theory. Given some reasonable axioms, game theory can produce unique variable importance measures in terms of Shapley values and Aumann-Shapley values respectively.

As a complementary method to theoretical a priori justification, one can also examine the outputs of attribution methods numerically, either applying sanity checks or computing quantitative quality measures. The insertion and deletion tests of Petsiuk et al., (2018) that we study are of this type. Quantitative metrics for XAI were recently surveyed in Nauta et al., (2022) who also discuss metrics based on whether the importance ratings align with those of human judges.

As an example of sanity checks, Adebayo et al., (2018) find that some algorithms to produce saliency maps are nearly unchanged when one scrambles the category labels on which the algorithm was trained, or when one replaces trained weights in a neural network by random weights. They also find that some saliency maps are very close to what edge detectors would produce and therefore do not make much use of the predicted or actual class of an image. Another example from Adebayo et al., (2020) checks whether saliency mathods can detect spurious correlation in a setting where the backgrounds in training images are artificially correlated to output labels and the model learns this spurious correlation correctly.

One important method is the ROAR (RemOve And Retrain) approach of Hooker et al., (2019). It sequentially removes information from the least important columns in each observation in the training data and retrains the model with these partially removed training data iteratively. A good variable ranking will show rapidly decreasing performance of the classifier as the most important inputs are removed. The obvious downside of this method is that it requires a lot of expensive retraining that insertion and deletion methods avoid.

The insertion and deletion tests we study have been criticized by Gomez et al., (2022) who note that the synthesized images generated in these tests are unnatural and do not resemble the images on which the algorithms were trained. The same issue of unnatural inputs was raised by Mase et al., (2019). Gomez et al., (2022) also point out that insertion and deletion tests only compare the rankings of the inputs.

Fel et al., (2021) noted that scores on insertion tests can be strongly influenced by the first few pixels inserted into a background image. They then note that images with only a few non-background pixels are quite different from the target image. The policy choice of what baseline to compare an image to affects whether the result can be manipulated. An adversarily selected image might differ from a target image in only a few pixels, and yet have a very different classification. At the same time both of these images differ greatly from a neutral background image. An insertion test comparing the target image to a blurred background might show that the classification depends on many pixels, while a deletion test with the adversarial image will show that only a few pixels need to change. The two tests will thus disagree on whether the classification was influenced by few or many pixels. Both are correct because they address different issues.

A crucial choice in using insertion and deletion tests is which reference input to use when deleting or inserting variables. Petsiuk et al., (2018) decided to insert real pixels into a blurred image instead of inserting them into a solid gray image because, when the most important pixels form an ellipsoidal shape, then inserting them into a gray background might lead a spurious classification (such as ‘balloon’). On the other hand, deleting pixels via blurring might underestimate the salience of those pixels if the algorithm is good at inferring from blurred images. In our albatross example of Figure 1 we used an all black image with $224\times 224\times 3$ zeros. We call such choices ‘policies’ and note that a good policy choice depends on the scientific goals of the study and the strengths and weaknesses of the algorithm under study.

Haug et al., (2021) study different kinds of baseline images for image classification problems noting that the choice of background affects how well a method performs. Among the backgrounds they mention are constants, blurred images, uniform or Gaussian noise, average images and baseline images maximally distant from the target image. They also mention the neutral backgrounds of Izzo et al., (2020) that lie on a decision boundary. Sundararajan and Najmi, (2020) discuss taking every available data point as a baseline and averaging the resulting Shapley values. Sturmfels et al., (2020) note the importance of choosing baselines carefully, pointing out that zero would be a bad baseline for blood sugar and that if a solid color image is used as a baseline in image classification, then the result cannot attribute importance to pixels of that color. They discuss many of the baselines that Haug et al., (2021) do and they propose the ‘farthest image’ baseline.

There are also broader criticisms of XAI methods. Kumar et al., (2020) point out some difficulties in formulating a variable importance problem as a game suitable for use in Shapley value. Their view is that those explanations do not match what people expect from an explanation. They also identify a catch-22 where either allowing or disallowing a variable not used by a model to be important causes difficulties. Rudin, (2019) says that one should not use a black box model to make high stakes decisions but should instead only use interpretable models. She also disputes that this would necessarily cause a loss in accuracy. We see a lot of value in that view, and we know that there are examples where interpretable models perform essentially as well as the best black boxes. However black boxes are very widely used. There is then value in XAI methods that can reveal and quantify any of their flaws. Furthermore, an explanation depends on not just the form of the model but also on the joint distribution of the predictor variables. For example an interpretable model that does not use the race of a subject might still have a discriminatory impact due to associations among the predictors and XAI methods can be used to evaluate such bias.

Prior uses of insertion and deletion tests have mostly been about image or text classification. There have been a few papers using them for tabular data or time series data, such as Cai et al., (2021), Hsieh et al., (2021), Parvatharaju et al., (2021), and Ismail et al., (2020).

Ancona et al., (2017) also describe a strategy of changing variables one at a time and observing how the quality of a prediction changes in response independently of Petsiuk et al., (2018). Their Figure 3c compares the trajectories taken by several different methods on some image classification problems. They have insertion and deletion curves to compare an occlusion method to integrated gradients. Unlike Petsiuk et al., (2018), they do not report an AUC quantity.

The work of Samek et al., (2016) precedes Petsiuk et al., (2018) and uses the same global ablation strategy of deleting information in order from most to least important. Their deletions involve replacing a whole block of pixels (e.g., a $9\times 9$ block) by uniformly distributed noise. One motivation for using such noise was to generate images outside the manifold of natural images. Their average over a perturbation curve is comparable to the average under the deletion curve of Petsiuk et al., (2018). Where Petsiuk et al., (2018) focus on curves for individual images, Samek et al., (2016) study the average of such curves over many images. In their numerical work they only make the first 100 such perturbations, affecting about $1/6$ of the pixels.

For us the advantage of the approach in Petsiuk et al., (2018) is that their AUC is defined by running the variable substitution to completion, changing every feature $x_{j}$ to the baseline value $x^{\prime}_{j}$. Then we use two orders, one that seeks to increase $f$ as fast as possible and one that seeks to decrease it as fast as possible, and study the area between those two curves.

The AUC method for comparing variable rankings has not had much theoretical analysis yet. This section develops some of its properties. The development is quite technical in places. We begin with a non technical account emphasizing intuition. Readers may use the intuitive development as orientation to the technical parts or they may prefer to skip the technical parts.

If we order the inputs to a function $f$ and then change them one at a time from the value in point $\bm{x}$ to that in a baseline value $\bm{x}^{\prime}$, the resulting function values trace out a curve over the interval $[0,n]$. If we have tried to order the variables starting with those that will most increase $f$ and ending with those that least increase (most decrease) $f$ then a better ordering is one that gives a larger area under the curve (AUC). We will find it useful to consider the signed area under this curve but above a straight line connecting the end points. This area between the curves (ABC) is the original AUC minus the area under the line segment.

A deletion measure orders the variables from those thought to be most decreasing of $f$ to those that are least decreasing, i.e., the opposite order to an insertion test. For deletion we like to use the area ABC below the straight line but above the curve that the deletion process traces out. When we need to refer to insertion and deletion ABCs in the same expression we use ABC^′ for the deletion case.

Additive functions are of special interest because additivity simplifies explanation and such models often come close to the full predictive power of a more complicated model. For an additive model, the incremental change in $f$ from replacing $x_{j}$ by $x^{\prime}_{j}$, call it $\Delta_{j}$, does not depend on $x_{k}$ for $k\neq j$. In this case the AUC is maximized by ordering the variables so that $\Delta_{1}\geqslant\Delta_{2}\geqslant\cdots\geqslant\Delta_{n}$. In this case the Shapley values are $\phi_{j}=\Delta_{j}$ and so ordering by Shapley value maximizes both AUC and ABC. We also show that if one orders the predictors randomly, then the expected value of ABC under this randomization is zero for an additive function.

Prediction functions must also capture interactions among the input variables. We study those using some higher order differences of differences. This way of quantifying interactions comes from an anchored decomposition that we present. This anchored decomposition is a less well known alternative to the analysis of variance (ANOVA) decomposition. The interaction quantity for a set $u\subseteq\{1,2,\dots,n\}$ of variables is denoted $\Delta_{u}$. This interaction does not contribute to any points along the curve until the ‘last’ member of the set $u$, denoted $\lceil u\rceil$, has been changed. It is thus present only in the final $n+1-\lceil u\rceil$ points of the insertion curve. A large AUC comes not just from bringing the large main effects to the front of the list. It also helps to have all elements in a positive interaction $\Delta_{u}$ included early, and at least one element in a negative interaction $\Delta_{u}$ appear very late in the ordering.

When there are interactions present, it is no longer true that $\mathbb{E}(ABC)$ must be zero under random ordering. We show in Appendix B.1 that the area between the insertion and deletion curves $\mathrm{ABC}+\mathrm{ABC}^{\prime}$ satisfies $\mathbb{E}(\mathrm{ABC}+\mathrm{ABC}^{\prime})=0$ under random ordering of inputs whether or not interactions are present.

Section 3.4 shows that if we sort variables in decreasing order of their Shapley values, then we do not necessarily get the ordering that maximizes the AUC. This is natural: the Shapley value $\phi_{j}$ of variable $j$ is defined as a weighted sum of $2^{n-1}$ incremental values for changing $x_{j}$, while the AUC, uses only one of those incremental values for variable $j$. The anchored decomposition that we present below makes it simple to construct an example with $n=3$ where the Shapley values are $\phi_{1}>\phi_{2}>\phi_{3}$ while the order $(1,3,2)$ has greater AUC than the order $(1,2,3)$. This is not to say that insertion AUCs are somehow in error for not being optimized by the Shapley ordering, nor that Shapley value is in error for not optimizing the AUC. The two measures have different definitions and interpretations. They can reasonably be considered proxies for each other, but the Shapley value weights a variable’s interactions in a different way than the AUC does.

In Appendix B.2 we consider the logistic regression model $f(\bm{x})=\Pr(Y=1\!\mid\!\bm{x})=(1+\exp(-\beta_{0}-\bm{x}^{\mathsf{T}}\beta))^{-1}$. Because of the curvature of the logistic transformation, this function has interactions of all orders. At the same time $\tilde{f}(\bm{x})=\log(f(\bm{x})/(1-f(\bm{x})))=\beta_{0}+\bm{x}^{\mathsf{T}}\beta$ is additive so on this scale the Shapley ordering does maximize AUC. The AUC on the original probability scale is perhaps the more interpretable choice. We show that due to the monotonicity of the logistic transformation, the Shapley ordering for $f(\bm{x})$ is the same as for $\tilde{f}(\bm{x})$ and so it also maximizes the $\mathrm{AUC}$ for $f(\bm{x})=\Pr(Y=1\!\mid\!\bm{x})$. Because $\exp(\cdot)$ is strictly monotone the Shapley ordering also optimizes AUC for loglinear models and for naive Bayes. It is also shown there that for a differentiable increasing function of an additive function that integrated gradients will compute the optimal order.

The next subsections present the above findings in more detail. Some of the derivations are in an appendix. We use well known properties of the Shapley value. Those are discussed in many places. We can recommend the recent reference by Plischke et al., (2021) because it also discusses Harsanyi dividends and is motivated by variable importance.

IG avoids the exponential computational costs that arise for Shapley value. However, as defined it is only available for variables with continuous values. Many problems have binary variables and so we describe here some approaches to including them.

IG is based on the Aumann-Shapley value from Aumann and Shapley, (1974) who present an axiomatic derivation for it. We omit those axioms. See Sundararajan et al., (2017) and Lundberg and Lee, (2017) for the axioms in a variable importance context.

We orient our discussion around Figure 3 that shows a setting with 3 variables. Panel (a) shows a target data point that differs from a baseline point in three coordinates. Panel (b) shows the diagonal path taken by integrated gradients. The gradient of $f(\bm{x})$ is integrated along that path to get the IG attributions:

If one of the variables is binary then one approach, shown in panel (c) is to simply jump from one value to another at some intermediate point, such as the midpoint. For differentiable $f$ the integral of the gradient along the line segment given by the jump would, by the fundamental theorem of calculus, be the difference between $f$ at the ends of that interval (times the Euclidean basis vector $(0,\dots,0,1,0,\dots,0)$ corresponding to that variable). Such a difference is computable for binary variables even though the points on the path are ill defined. Finally, the vector of Shapley values is an average over $n!$ paths making jumps from baseline to target in all possible variable orders (Sundararajan et al.,, 2017). Panel (d) shows two of those paths. For differentiable $f$ one could integrate gradients along those paths as a way to compute the jumps, again by the fundamental theorem of calculus, and then average the path integrals.

Now suppose that we have $m>0$ binary variables in a set $v\subset 1{:}n$. Without loss of generality suppose that $v=1{:}m$. Then any data $\bm{x}$ and $\bm{x}^{\prime}$ are in $\{0,1\}^{m}\times\mathbb{R}^{n-m}$ and for IG we need to consider arguments to $f$ in $\mathbb{R}^{n}$. We consider three choices that we describe in more detail below:

1.  a)

   Use the fitted $f$ as if the binary $x_{j}\in[0,1]$, for $j\in 1{:}m$.

2.  b)

   Replace $f$ by a multilinear interpolation

   $$g(\bm{x})=\sum_{u\subseteq 1{:}m}f(\bm{1}_{u}{:}\bm{0}_{1:m-u}{:}\bm{x}_{(m+1):n})\prod_{j\in u}x_{j}\prod_{j\in 1:m-u}(1-x_{j})$$

   and compute the integrated gradients of $g$.

3.  c)

   Take paths that jump for binary $x_{j}$ as shown in Figure 3(c).

We call these choices, casting, interpolating and jumping, respectively.

Option a is commonly available as many machine learning models cast binary variables to real values when fitting. Option b interpolates: if $\bm{x}_{1:m}\in\{0,1\}^{m}$ then $g(\bm{x})=f(\bm{x})$. Sundararajan et al., (2017) show that integrated gradients match Shapley values for functions that are a sum of a multilinear interpolation like $g$ above plus a differentiable additive function. Unfortunately, the cost of evaluating $g(\bm{x})$ is $\Omega(2^{m})$, which is exponential in the number of binary inputs. For option c we have to choose where to make the $m$ jumps. For $m=1$ we would naturally jump half way along the path though there is not an axiomatic reason for that choice. For $m>1$ we have to choose $m$ points on the curve at which to jump. Even if we decide that all of those jumps should be at the midpoint, we are left with $m!$ possible orders in which to make those jumps. By symmetry we might want to average over those orders but that produces a cost which is exponential in $m$.

Based on the above considerations we think that the best way to apply IG to binary variables is also the simplest. We cast the corresponding booleans to floats.

In this section we illustrate insertion and deletion tests for regression. We compare our variance importance measures on two tabular datasets. The first one is about predicting the value of houses in India. It has mostly binary predictors and two continuous ones. The second dataset is from CERN and it computes the invariant mass produced from some electron collisions.

The methods we compare are Kernel SHAP (Lundberg and Lee,, 2017), integrated gradients (Sundararajan et al.,, 2017), DeepLIFT (Shrikumar et al.,, 2017), Vanilla Grad (Simonyan et al.,, 2013), Input$\times$Gradient (Shrikumar et al.,, 2016) and LIME (Ribeiro et al.,, 2016). Ancona et al., (2017) make a detailed study of backpropagation-based gradient methods including all of the above ones (this excludes LIME) as well as layer-wise relevance propagation (LRP) of Bach et al., (2015) that we did not include in our computations.

In this section we compare KS and IG via ROAR (RemOve And Retrain) (Hooker et al.,, 2019). ROAR is significantly more expensive to study than the other methods we consider as it requires retraining the models, and so we did not apply it to all of the methods. We opted to apply it just to Kernel SHAP and IG. We chose IG as our representative fast method because IG can be used on more general models than DeepLIFT can. We chose Kernel SHAP as the other method because it had best or near best ABC values on our numerical examples. The task in ROAR is about which variables are important to the model’s accuracy and not about which variables are important to any specific prediction. As a result the values in ROAR are not comparable to the other AUC and ABC values that we have computed.

The original proposal of ROAR measures the drop in accuracy for image classification tasks and applying it to regression tasks with tabular data raises the same issues as extending insertion/deletion tests to regression. We measure the test loss on held out data as a measure of retrained model performance. Retraining procedures are taken with an increasing number of removed features at each quantile in $\{0.1,0.3,0.5,0.7,0.9,1.0\}$, where $1.0$ means that all features are removed. The model architecture and hyperparameters of retrained models are the original models as described in Sections A.2 and A.4.

To use ROAR we must decide how to remove features in the data. Removing in the original ROAR algorithm means padding pixels of the original images with noninformative values such as gray levels. In our experiments, the important features are overwritten by average values for continuous features and by modes for discrete features. These average and modal values are taken from the training data.

The results of our ROAR calculations are shown in Figure 13. Features to be removed are sorted both by their absolute values and by their original signed values for both KS and IG. The error bars show plus or minus one standard error computed from five replicates.

Since ROAR in this experiment measures the Huber loss on the test points, it should be unaffected by the signs of the attributions and sensitive to their magnitude. For this reason, sorting features by the absolute values of their attributions should give a better score than sorting by their signed values. This is a contrasting point to insertion and deletion tests.

The results in Figure 13 are surprising to us. The curves we see are very nearly straight lines connecting the loss with all features present to the loss with no features present, so there is very little area between them and a straight line connecting the end points. This means that the loss in accuracy from variables deemed most important is about the same as those deemed less important. This could be because neither KS nor IG are able to identify important predictors for this task. It could also be that the majority of predictor variables in this data can be replaced by a combination of some other predictors which then prevents a large reduction in accuracy from removing a subset of predictors prior to retraining.

A second surprise is that the signed ordering, that we used as a control that should have been beaten by the absolute ordering was nearly as good as the absolute ordering.

As before KS and IG are comparable, though here both seem disappointing. Using IG with variables sorted by their absolute values even came out superior to KS in the Bangalore housing dataset.

In this paper we have extended insertion and deletion tests to regression problems. That includes getting formulas for the effects of interactions on the AUC and ABC measures, finding the expected area between the insertion/deletion curve under random variable ordering, and replacing the horizontal axis by a more appropriate straight line reference. We gave a condition under which sorting variables by their Shapley value will optimize ABC as well as constructing an example where that does not happen.

We compared six methods and several policies on two datasets. We find that overall the Kernel SHAP gave the best areas. The much faster Integrated Gradients method was nearly as good. In order to even run IG in settings with binary variables, some strategy for using continuum values must be employed. We opted for the simplest choice of just casting the booleans to real values.

A very natural policy question is whether to prefer insertion or deletion. Petsiuk et al., (2018) consider both and do not show a strong preference for one over the other. They use deletion when comparing a real image to a blank image (deleting real pixels by replacing them with zeros). They use insertion when comparing a real image to a blurred one (inserting real pixels into the blurred image). In other words the choice between insertion and deletion is driven by the counterfactual point. In the regression setting both inputs points could be real data. By studying both insertion and deletion we have seen that they can differ. A natural way to break the tie is to sum the ABC values for both insertion and deletion. Under a completely random permutation the expected value of that sum is zero. See Appendix B.1. In our examples, IG closely matches KS for both insertion and deletion, so it also matches their sum.

This work was supported by the U.S. National Science Foundation grants IIS-1837931 and DMS-2152780, and by Hitachi, Ltd. We thank Benjamin Seiler for helpful comments. Comments from three anonymous reviewers have helped us improve this paper.