Attribution Mask: Filtering Out Irrelevant Features By Recursively Focusing Attention on Inputs of DNNs

A single-layer neural network is an easy-to-interpret model because it simplifies the relationship between the target class and the input features to an affine transform. Therefore, to create an interpretable DNN, we compress a DNN into a single-layer neural network through local linearization. Suppose that a pre-trained DNN to be compressed has a total of $L$ layers, and the number of dimensions of the $l$-th layer is $d_{l}$, where $l\in\{1,\dots,L\}$. Let $\mathbf{z}_{l}=\phi(\mathbf{s}_{l})$ be a hidden output of each layer. Here, the input of nonlinearity $\phi(\cdot)$ is $\mathbf{s}_{l}=\mathbf{W}_{l}\mathbf{z}_{l-1}+\mathbf{b}_{l}$, the weight and the bias are $\mathbf{W}_{l}\in R^{d_{l},d_{l-1}}$ and $\mathbf{b}_{l}\in R^{d_{l}}$, and the input features are $\mathbf{z}_{0}\in R^{d_{0}}$.

To locally linearize a DNN, we focus on the nonlinearity of the DNN. The nonlinearity can be classified into two categories, element-wise and non-element-wise nonlinearity. The element-wise nonlinearity (e.g., Rectified Linear Unit (ReLU), Tanh, etc.) denotes a function that does not change the dimensions of the input features, and the non-element-wise nonlinearity (e.g., MaxPool, AvgPool, etc.) means that the input dimension changes. First, applying the Taylor decomposition (Kindermans et al., 2016; Montavon et al., 2017) to the element-wise nonlinearity function $\phi:R^{d_{l}}\to R^{d_{l}}$ with respect to an input of the function $\mathbf{s}_{l}\in R^{d_{l}}$, we obtain

where $\mathbf{a}_{l}=\nabla_{\mathbf{s}_{l}}\phi(\mathbf{s}_{l})$ and $\mathbf{c}_{l}=\phi(\mathbf{e}_{l})-\mathbf{a}_{l}\circ\mathbf{e}_{l}+(\nabla^{2}_{\mathbf{s}_{l}}\phi(\mathbf{s}_{l})\circ(\mathbf{s}_{l}-\mathbf{e}_{l})^{2}+\dots)$. The baseline is $\mathbf{e}_{l}\in R^{d_{l}}$ and $\circ$ is an element-wise product. Next, for a non-element-wise nonlinearity $\phi:R^{d_{l}}\to R^{d_{l+1}}$, the Taylor decomposition provides

where $\mathbf{W}_{l+1}\in R^{d_{l+1},d_{l}}$. Because there is no bias in the result of the local linearization, we can absorb the function in another linear layer. Therefore, we only need to consider the element-wise nonlinearity containing bias and apply the linearized nonlinearity to the DNN $f:R^{d_{0}}\to R^{d_{L}}$ including CNNs (Nie et al., 2018). We then have

Lemma 1 is applied to Equation 3 of the locally linearized DNN for the input features and is derived as follows:

where

Assuming there is $\mathbf{\Delta}^{*}_{t}\in R^{d_{0}}$ that satisfies $\mathbf{\Delta}_{t}=\mathbf{W}_{t}\mathbf{\Delta}^{*}_{t}$ for compressing the DNN into a weight matrix, Equation 4 is derived as follows:

Thus, the brew matrix $\mathbf{W}_{t}$ and the additive input feature $\mathbf{\Delta}_{t}^{*}$ can be obtained for the input features $\mathbf{z}_{0}$.

The additive input features $\mathbf{\Delta}_{t}^{*}$ is generated by the total bias $\mathbf{\Delta}_{t}$, which can work as noise for the input features $\mathbf{z}_{0}$ and is consistent with what was claimed in (Mohan et al., 2019; Wang et al., 2019). To remove $\mathbf{\Delta}_{t}^{*}$, we use a no implicit bias (NIB) model, a condition that forces all $\mathbf{c_{l}}$ and $\mathbf{b_{l}}$ in Equation 6 to zeros.

First, to force $\mathbf{b}_{l}=\mathbf{0}$, we eliminate the biases of all layers as well as the mean parameters of batch normalization (Ioffe & Szegedy, 2015). Next, we use the Taylor decomposition under the baselines $\mathbf{e}_{l}=\mathbf{0}$ to find the nonlinearity such that it satisfies the condition $\mathbf{c}_{l}=\mathbf{0}$. Therefore, we only consider functions such that satisfy this condition (e.g., ReLU or MaxPool). For example, in the case of ReLU, $\phi(\mathbf{0})=\mathbf{0}$ and $\nabla^{(n)}_{\mathbf{s}_{l}}\phi(\mathbf{s}_{l})=\mathbf{0}$ for $n\geq 2$, and thus Equation 1 is calculated as $\mathbf{a}_{l}\circ\mathbf{s}_{l}$. In the other case, as in Equation 2, MaxPool satisfies the condition because the function is locally linearized as a linear layer without bias. These two conditions, $\mathbf{b}_{l}=\mathbf{0}$ and $\mathbf{c}_{l}=\mathbf{0}$, lead $\mathbf{\Delta}_{t}$ to $\mathbf{0}$ in Equation 6. We thus can derive Equation 7 as follows:

Consequently, $\mathbf{W}_{t}\in R^{d_{0},d_{L}}$ denotes the noise-reduced brew matrix. When using a DNN as a classifier, which means that $d_{L}$ is the same as the number of classes $K$, the simplified matrix fully contains the information regarding the target class $k\in\{1,2,\dots,K\}$. Therefore, because the matrix is calculated using only the input features excluding $\mathbf{\Delta}_{t}^{*}$, we simply use the $k$-th row of the matrix to obtain information about the target label.

In terms of the brew matrix, attributions $[\mathbf{W}_{t}]_{k}\circ\mathbf{z}_{0}$ satisfy the efficiency properties of Shapley when the attributions are defined as Shapley values (Sundararajan & Najmi, 2020). In other words, the sum of all elements is equivalent to the prediction for the $k$-th class of the DNN.

Existing GBA methods calculate attributions along with input features by modifying the gradient obtained from a DNN. Among such methods, Gradient * Input, which is the basis, obtains attributions by applying the gradient without additional manipulation. Given the $L$-layer DNN $f$ and the input features $\mathbf{z}_{0}$, the gradient of this method is calculated through an automatic differentiation as follows:

Here, $\nabla_{\mathbf{s}_{l}}\mathbf{z}_{l}$ is a locally linearized nonlinearity $\bar{\mathbf{a}}$ and $\nabla_{\mathbf{z}_{l-1}}\mathbf{s}_{l}$ is a weight matrix $\mathbf{W}_{l}$. Therefore, Equation 9 is identical to Equation 5 under two conditions, $\mathbf{c}_{l}=0$ and $\mathbf{b}_{l}=0$. Thus, the brew matrix can also be computed as follows:

Based on this result, the gradient calculated using the GBA methods ignores the additive noise term, which means that this noise is always included in the attributions when the NIB model is not used.

The NIB condition guarantees that the gradient does not contain additive noise. DeepLIFT and Integrated Gradients, which are designed for Completeness (Sundararajan et al., 2017), satisfy this condition by introducing the baselines, although finding such baselines is a difficult problem. Therefore, we compute the gradient under the NIB model to enhance the model accuracy through attributions. Under this model, Gradient * Input, DeepLIFT and Integrated Gradients satisfy the NIB condition (see Appendix A).

The gradient, which is denoted as a specific row of the brew matrix, is attributed to the class $k$. Each row is trained using several input features corresponding to each class within the dataset. Because these features contain some redundant information other than that related to the target class, we suppress this information by focusing attention on important parts of input features through the attributions.

First, we propose an attribution mask that recursively focuses attention on parts of an input of a pre-trained DNN $f$. Let attributions $\mathbf{A}_{i,k}=\text{Expl}(f,\mathbf{z}_{0}\circ\prod_{j=0}^{i-1}\mathbf{m}_{j,k},k)$ with Expl denotes the GBA method, where the attribution mask for the target class is $k$ and the number of iterations is $i$. The mask is calculated using MinMax normalization (Patro & Sahu, 2015) as follows:

where the initial mask $\mathbf{m}_{0,k}$ is $\mathbf{1}$, and $\mathbf{m}_{i,k},\mathbf{A}_{i}\in R^{d_{0}}$. This method normalizes the mask value to between zero and 1, leaving only the ratio of the relative values between the attributions, and thus the repeated multiplications of the mask does not diverge. In the case of an image, which is the target data of this study, $d_{0}$ is $channel\times height\times width$, and we conduct normalization for each $channel$.

Next, the attribution mask is multiplied by the input features and then the masked features are fed into the DNN. Let $\mathcal{D}$ be a test dataset that has the input features $\mathbf{z}_{0}$ paired with the target label $t\in\{1,2,\dots,K\}$. Then, the accuracy of the DNN with the mask is calculated as follows:

where $\mathbbm{1}_{t}[.]$ is an indicator function, which is $1$ when $t=k$ and has a value of $0$ for the others. This accuracy, i.e., the MIA, is used to quantify the loss of information that occurs when target label information passes through DNN layers by the backpropagation process. For example, if $Q$ decreases from the baseline accuracy ($i=0$) for $i>0$, the target label information is not properly backpropagated to the attributions, indicating that the irrelevant part is incorrectly removed.

The existing GBA methods use the gradient that already has the physical dimension of the input features and create nonlinear attributions under the perspective of the brew matrix. To make linear attributions, we only use the sign of the input features. Next, we propose the attribution method, which is calculated as follows:

We label this method Gradient * Sign-of-Input (GxSI), which increases the attribution value when the gradient element and the input feature have the same sign. This method is based on the intuition that the input features that improve the prediction of the DNN are an important part. Unlike the Gradient * Input, this method is designed so that the gradient value affects attributions by only using the sign of the input instead of its value.

NIB VGG To construct the NIB model, we removed all parameters corresponding to the mean of batch normalization in VGG (Simonyan & Zisserman, 2014) and the additive bias of all layers. In this model, a kernel size of 3 and a padding of 1 were used for all conv2d layers, and a kernel size of 2 and a stride size of 2 were used for MaxPool. To experiment with NIB models of different depths, we started with NIB VGG13 and constructed the model by reducing the number of conv2d layer by 2 to a total of 7 layers (see Table 1). Their models are labeled NIB VGG13-2, NIB VGG13-4, and NIB VGG13-6, respectively.

WideResNet WideResNet (Zagoruyko & Komodakis, 2016) was only used as a student model in Section 5.4, and thus it was not modified as an NIB model. We used WRN-28-10 and WRN-40-4, both of which achieve a stable performance on CIFAR datasets.

We targeted the following GBA methods: Gradient * Input (GxI), Guide Backpropagation (GBP), Integrated Gradient (IG), DeepLIFT (DL), Positive-Gradient * Input (PGxI) and Gradient x Sign-of-Input (GxSI). Our implementations of GBA method are identical to those of (Adebayo et al., 2018), except for PGxI and GxSI. First, in the case of the proposed attribution method, GxSI, we implemented Equation 13. Next, in the case of PGxI, we implemented the saliency map of (simonyan2013deep) by multiplying the input by the following element: $[\nabla_{\mathbf{z}_{0}}f(\mathbf{z}_{0})]_{k}^{+}\circ\mathbf{z}_{0}$, where $k$ is a target class and $[\cdot]^{+}=\max(\cdot,0)$. Among the other methods, IG and DL need the baselines, and because we use the NIB model, the attributions were calculated with the baselines set to zeros. In particular, in the case of IG, we calculated the attributions with a step size of 50 to numerically calculate the integral.

To verify the effect of the NIB condition on the MIA score, we measured this score of the existing GBA methods. Whether the methods satisfy the condition or not depends on the gradient modification and a pre-trained model. We used NIB VGG16 among the pre-trained NIB models to demonstrate the effect of the gradient modification (see the details in Section 4.1).

As shown in Table 2, GxI, DL, and IG have the same MIA value under the NIB model. We also see that these methods produce better performance than other methods. In particular, on CIFAR-10, the performance was 96.3%, which is close to 100%. However, as shown in Figure 3, PGxI and GBP methods reduce the performance despite the attribution mask looking more similar to the input image than with the other approaches. This deterioration in performance is because the object in this image disappears as the mask recursively modifies the input image (Figure 3 col. 1, col. 2). The baseline accuracy (i.e., the accuracy at $i=0$) of the DNN on CIFAR-100 is inferior to that of CIFAR-10. However, we can see that the methods, which satisfy the NIB condition, have a higher performance than the other methods.

According to the above results, methods using gradients that satisfy the NIB condition can increase the MIA compared to the other methods. Meanwhile, these results are contrary to the fact that GxI and other GBA methods failed the sanity check (Adebayo et al., 2018; Sixt et al., 2020), the reason for which is that these methods do not satisfy the NIB condition because the model used in the previous study was not the NIB model.

To validate that GxSI improves the performance, we compare its MIA score with that of the methods discussed in Section 5.1. As we can see in Table 2, the proposed method achieves superior performance compared to the other attribution methods, and the gradient satisfies the property. This superiority becomes more pronounced when the number of iterations increase to 3 or more.

In Figure 4 (a), we can see that the proposed method, unlike the other approaches, continuously increased its MIA and reached a performance close to the ideal value of 100%. In particular, on CIFAR-10, the performance reached 99.8% with 7 iterations, 99.9% with 10 iterations, and then fluctuated between 99.8% and 99.9%. Meanwhile, for GxI, IG, and DL, the measured value increased only until the second iteration and then gradually decreased to 52%. Moreover, for PGxI and GBP, the performance decreased continuously to 18.3% and 10.9%, respectively. In Figure 4 (b), on CIFAR-100, we can also see that the MIA value measured using the GxSI increases continuously, and we achieved 90.2% of the MIA with 12 iterations despite the relatively low baseline accuracy (see Appendix B).

These results imply that the attribution mask of the GxSI can recursively filter information not related to the target label more effectively than existing methods.

To measure the performance of GxSI according to the configurations of a DNN, we constructed the NIB model of various depths (see Table 1).

As shown in Table 3, in general, the depth of the DNN and the baseline accuracy are proportional, but the performance of the proposed method was rather reduced. In particular, at $i=2$, the performance of the proposed method for NIB VGG13-6 was 1.7% higher than that for NIB VGG16. However, as shown in Figure 4, the performance of GxSI for NIB VGG16 increased to 99.9 %.

Therefore, these results indicates that the convergence speed of performance decreases according to the model depth, and the deeper DNNs lead to a difficulty in the attributions required to obtain the information of the target label. However, we can overcome this problem by applying the attribution mask to the input feature recursively.

We modified the input features with the attribution mask obtained using GxSI under the NIB model and applied it for training other models. In this way, we tested for arbitrary DNNs in which the mask transforms the input features into easy-to-understand features. For this experiment, the model that obtains the attribution mask is called the teacher model, and the model to be trained from the input features modified with this mask is called the student model. For this teacher model, NIB VGG13 and NIB VGG16 were used to use the attribution mask showing the high MIA values. Specifically, for NIB VGG16, to introduce an attribution mask showing various MIA scores, this mask was obtained by repeating the process 3 and 10 times. The CIFAR-10 dataset was modified by multiplying the mask to the image of the training and test set. The student model NIB VGGs and WRNs were trained using this dataset.

As shown in Table 4, the accuracy of all student models was increased compared to the original accuracy. For the student model NIB VGG16, the accuracy was 99.9%, 99.6%, and 99.8% with respect to each teacher model, whereas it was 99.7%, 99.1%, and 99.5% for the student model NIB VGG13-6. We can see that the difference in classification accuracy between the two student models was not large. For the student model WRN-40-4 and WRN-28-10, through the teacher model with an MIA score of 99.8% and 99.9%, 100% accuracy was achieved when the input features contained all of the target label information. As we can see in the last two columns of Table 4, the accuracy of $i=10$ NIB VGG16 was higher on average than $i=3$ NIB VGG16.

These results indicate that the MIA score is a value that generally represents the ability to filter information other than the target label of the attribution mask from the input features. If the mask is only effective for the DNN from which it was obtained, the masked input features should depends on the model and lose the generality. Thus, the masked input features, which allow to achieve a high accuracy, are generally easy-to-understand features for an arbitrary DNN. Furthermore, we can see that the configuration of the model does not need to be complicated when the arbitrary dataset is masked with the attribution mask.

In the previous Sections, we conducted experiments using the NIB model and saw that the NIB condition is necessary for improving the MIA score. We revoked this condition and then conducted an experiment to measure the change in the MIA score on CIFAR-10. Accordingly, layer bias (LB) and batch normalization bias (BNB) were added to NIB VGG16 for GxI and GxSI among GBA methods, and we measured this score for 10 iterations and compared the maximum MIA score with the baseline accuracy.

As described in Table 5, there were two problems, one was that the MIA was small compared to the baseline accuracy, and the other was that the MIA decreased as the iterations increased. For the model with both LB and BNB added, the MIA scores of GxI and GxSI were decreased to 10.5% and 12.5%, respectively, and the maximum MIA was also lower than the baseline accuracy. The same phenomenon was observed for the model with BNB added, and the two GBA methods showed 16.6% and 11.5% MIA scores at 9th iteration, respectively. For the model with LB added, the maximum MIA of GxSI was slightly increased, but the score eventually fell to 10.0.

This result empirically shows that when the gradient calculated by the GBA methods does not satisfy the NIB condition, the performance is reduced by additive noise as we expected.