Deeply Explain CNN via Hierarchical Decomposition

Feature attribution methods typically generate a saliency map to locate the input locations important to the output. We classify them into three categories: backpropagation-based methods, perturbation-based methods, and activation-based methods.

The aforementioned attribution methods mostly focus on generating saliency/activation maps to study how the input affects the output prediction. Although some attribution methods can measure the importance of intermediate features to the output prediction, they usually neglect to study the relationships among different intermediate features. As pointed by Olah et al. [23], the relationships among different intermediate features are also important to interpret a prediction. We decompose not only the network decision but also the intermediate features to find their supporting evidence from previous layers, explaining how these associated intermediate features affect each other. While LRP [34] method propagates feature importance to intermediate features, the feature importance for different channels is coupled in the back-propagation process. This method generates simple explanations for the entire network behavior rather than hierarchical explanations.

Visualizing the CNN features of the intermediate layers can provide insight into what these layers learn. For the first layer of the CNN, we can directly project its three-channel weights into the image space. To visualize the features from higher layers, researchers have proposed many alternative approaches. Among them, Erhan et al. [51] and Simonyan et al. [20] utilize the gradient ascent algorithm to find the optimal stimuli in the image space that maximizes the neuron activations. Other methods [24, 30, 52] identify the image patches from the dataset that maximize the neuron activation of the CNN layers, as shown in Fig. 2 (a). Guided Backpropagation [30] and Deconvnet [24] also utilize the top-down gradients to discover the patterns that the intermediate layers learn. Using the natural image prior, feature inversion methods [53, 54, 55, 56, 57] learn an image to reconstruct the neuron activation. Furthermore, the recent methods [58, 59, 60] attempt to detect the concepts learned by intermediate CNN layers. The above feature visualization methods explore what the intermediate features detect, but they do not answer how the network assembles individual features to make a prediction.

Recently, another research line has attempted to transfer the powerful ability of CNN to explainable models, such as the decision tree or linear model, to approximate the behavior of the original model. Chen et al. [61] distill the knowledge into an explainable additive model. Ribeiro et al. [45] utilize a local linear model to approximate the original model, studying how the input affects any classifier’s decisions. Frosst et al. [62] and Liu et al. [63] distill the learned knowledge of CNN into the decision tree. These methods only bridge between the network decision and the input. They cannot help the user understand how the internal features of CNNs affect the network decision and each other. Our hierarchical decomposition is also an approximation to the original model. Unlike the above methods, our hierarchical decomposition not only highlights the important features for the network decision but builds the relationships among the feature channels from different layers. From our method, we can obtain the states of the internal features and how the internal features affect each other and the network decision.

Except for the post-hoc interpretability analysis for a trained CNN, some researchers have attempted to explore inherently interpretable models. Chen et al. [64] proposed a deep network architecture called prototypical part network. The network has a transparent reasoning process that first computes the similarity scores between the image patches and the learned prototypes. Then the network makes predictions based on a weighted sum of the similarity scores. Concept bottleneck models [65, 66, 67] are also inherently interpretable. Unlike those post-hoc methods [58, 59] that utilize human-specific concepts to generate explanations, they directly predict a set of human-specific concepts at training time and then use these concepts to make predictions, where the reasoning process is interpretable. Some recent intrinsic interpretable models [65, 64] utilize VGG [1] or ResNet [2] to extract high-level features firstly and perform the reasoning process on the high-level features. Our method is complementary to these CNN-based intrinsic interpretable models because one can use the hierarchical decomposition to provide more hierarchical evidence from the feature extractor if needed.

We begin by defining the notation for the CNN, as illustrated in Fig. 3. In the $l^{th}$ CNN layer, the features $\mathbf{F}^{l}$, partial gradients $\mathbf{G}^{l}$, and corresponding neuron activations $\mathbf{A}^{l}$ are 3D tensors with the same size, i.e., $\mathbf{G}^{l},\mathbf{A}^{l},\mathbf{F}^{l}\in\mathbb{R}^{K^{l}\times H^{l}\times W^{l}}$, where $K^{l}$ is the number of channels and $H^{l}\times W^{l}$ is the spatial size in the CNN layer $l$. To find supporting evidence for the final CNN decision or any intermediate feature response, we propose a gradient-based activation propagation (gAP) method. Using the gAP module, we can understand a decision of interest at a CNN layer by localizing the most related evidence in its previous layer.

As shown in Fig. 3, we decompose a CNN feature (i.e., a decision of interest) $\mathbf{F}^{l+1}_{k^{\prime},x,y}$ at the convolutional layer $l+1$, channel $k^{\prime}$, and spatial position $(x,y)$, to find the supporting evidence in its previous convolutional layer $l$. In this work, we are interested in understanding the strong feature response $\mathbf{F}^{l+1}_{k^{\prime},x,y}$ that has the largest contribution to the decision among the feature channel. In typical CNNs, a certain feature at layer $l+1$ is computed as a linear combination of features from its previous layer $l$ and a ReLU. For the strong feature $\mathbf{F}^{l+1}_{k^{\prime},x,y}$, we have

where $\bf{w}_{k}^{l}$ is the linear weight for combining $k^{th}$ feature channel of $\mathbf{F}^{l}$. To obtain the weight, we first use backpropagation to compute the partial gradient map $\mathbf{G}^{l}_{k}$ of the feature $\mathbf{F}^{l+1}_{k^{\prime},x,y}$ w.r.t. the feature map $\mathbf{F}^{l}_{k}$ by

The gradient map $\mathbf{G}_{k}^{l}$ captures the ‘importance’ of the feature map $\mathbf{F}_{k}^{l}$ for the decision $\mathbf{F}^{l+1}_{k^{\prime},x,y}$.

We employ the gradient map $\mathbf{G}_{k}^{l}$ to generate an activation map

The activation map indicates the contribution of each feature in $\mathbf{F}_{k}^{l}$ to the decision $\mathbf{F}^{l+1}_{k^{\prime},x,y}$. Based on its corresponding activation map, each channel’s contribution to the decision can be computed by

where $Z^{l}=H^{l}\times W^{l}$ denotes the number of spatial positions in the activation map $\mathbf{A}^{l}_{k}$. We can also identify the feature $\mathbf{F}^{l}_{k,\hat{x},\hat{y}}$ in $k^{th}$ feature channel that contributes the most to the decision in which

Thus, for each decision, we can find the most important feature channel $\mathbf{F}_{k}^{l}$ according to the contribution $\alpha_{k}^{l}$ computed by Eqn. (4). In the most important channel, we can also identify the feature $\mathbf{F}^{l}_{k,\hat{x},\hat{y}}$ that contributes most to the decision according to Eqn. (5). In the top row of Fig. 4, we show the three most important activation maps $\mathbf{A}^{4}_{131}$, $\mathbf{A}^{4}_{255}$, and $\mathbf{A}^{4}_{452}$ in layer conv4_3 for the decision from $\mathbf{F}_{277}^{5}$. These activation maps provide spatial channel responses to the decision, benefiting human understanding. Using Guided Backpropagation [30], we visualize the most contributing feature by generating sharp visualizations, which highlight the associated input. An example is shown in the bottom row of Fig. 4.

Discussion. Our gAP module is inspired by CAM [28] and Grad-CAM [29], which explain CNN decisions by class activation localization. To explain the relation and difference to our gAP module, we first revisit CAM and Grad-CAM. Smilkov et al. have proofed that Grad-CAM is a strict generalization of CAM. Without loss of generality, we consider the same network discussed in [28]. For an image classification CNN, the CNN features $\mathbf{F}^{L}$ of the last convolutional layer are spatially pooled using the global average pooling layer to obtain feature vectors. The network performs a linear combination of the feature vectors by feeding them into a fully connected layer before the softmax. Let $C$ denote the number of classes. The classification score before softmax $S^{c}$ for each class $c\in\{1,2,\dots,C\}$ is

where $w_{k}^{c}$ is the weight connecting the $k^{th}$ feature map with the $c^{th}$ class. The contribution of a feature $\mathbf{F}^{L}_{k,x,y}$ to $S^{c}$ is $w^{c}_{k}\mathbf{F}^{L}_{k,x,y}$. CAM generates a class activation map $M^{c}$ by summing over all feature maps,

where each value in $\mathbf{M}^{c}$ indicates the contribution of each spatial location to $S^{c}$.

For the linear function, the importance weight is also equal to the gradient. Thus, we can also obtain the weight by computing the back-propagating gradients,

The detailed derivations of $w^{c}_{k}$ is depicted in [29]. Eqn. (8) is also the way that Grad-CAM computes the weight $w^{c}_{k}$. A little difference is that Grad-CAM multiplies $w^{c}_{k}$ by a proportionality constant, i.e.,

where the proportionality constant $1/Z^{L}$ will be normalized.

Considering the scores $\{S^{c}\}$ as CNN features with $C$ channels and spatial size $1\times 1$, i.e., $\mathbf{F}_{c}^{L+1}(1,1)=S^{c}$, we can plug Eqn. (2) into Eqn. (9) and get

Due to the global average pooling layer, the gradient of each element in $\mathbf{F}^{L}_{k}$ is the same, i.e., $\forall_{x,y},\mathbf{G}_{k,x,y}^{L}=w^{c}_{k}$. The class activation map $\mathbf{M}^{c}$ can be written as

Eqn. (11) suggests that the activation map $\mathbf{M}^{c}$ for Grad-CAM can be generated by simply adding the activations maps $\mathbf{A}^{L}_{k}$ from our gAP.

The differences between gAP and Grad-CAM/CAM are:

• •

  Grad-CAM/CAM combine all activation maps to generate a single class activation map $\mathbf{M}^{c}$, which highlights important regions supporting the prediction. Our gAP method explains a decision of interest by generating a group of activation maps $\{\mathbf{A}_{k}\}$. Each activation map corresponds to a feature channel, which is crucial for our iterative decomposition process.

• •

  Grad-CAM/CAM generate class activation maps from the last convolutional layer to explain the prediction. Our gAP generalizes this idea and iteratively decomposes a decision of any CNN layer to its lower layer.

While the above derivations apply to adjacent layers, we empirically find that satisfactory decomposition results can also be obtained when applying the gAP module between two layers from different stages of CNN (see Sec. 4.1). In the following, we will describe how we build hierarchical explanations for the network decisions.

Fig. 5 demonstrates an example of our hierarchical decomposition process. First, we decompose the network decision to the last convolutional layer and find the top few most crucial supporting features. Then, we decompose each of the supporting features to their previous layer and iteratively repeat the decomposition process until the bottom layer. As mentioned in Sec. 1, the key challenge is that naively building the hierarchical decomposition will generate too many visualizations, which will be a cognitive burden for humans. Even if we only decompose a single maximal contributed feature in each channel (see also Eqn. (5)), directly decomposing all channels in VGG-16 will generate $512^{3}\times 512^{3}\times 256^{3}\times 128^{2}\times 64^{2}\approx 2.5\times 10^{22}$ visualizations.

To obtain human-scale visualizations, we propose two strategies to reduce the number of visualizations. Firstly, we only decompose the top few most important features at each layer. Experiment (see Sec. 4.1) has verified that a small subset of feature channels in a layer accounts for the majority of the contributions to a decision. Thus, we select the top few most important channels. We simplify the top-down decision decomposition process by utilizing the last convolutional layer of each stage. Current popular CNNs [1, 2] usually reduce the spatial size of feature maps after each stage, where a stage is composed of a set of convolution layers with the same output resolution. Each stage learns different patterns, such as blobs/edges, textures, and object parts [30, 24]. Experiments verify that when using the gAP module between two layers from two consecutive stages, we can obtain visually meaningful decomposition results (see Fig. 5). By these two strategies, we can largely reduce the number of visualizations to obtain human-scale explanations.

An example of the VGG-16 classification network is shown in Fig. 5. We select conv1_2, conv2_2, conv3_3, conv4_3, conv5_3 and index these layers as $\{1,2,\dots,L\}$, where $L=5$. The network output before softmax could be considered as the $6^{th}$ CNN layer, with features $\mathbf{F}^{6}\in\mathbb{R}^{C\times 1\times 1}$. The decomposition process starts from the CNN decision $\mathbf{F}^{6}_{c}(1,1)$, where $c$ corresponds to the ‘person’ class. Using gAP, we first decompose the CNN decision $\mathbf{F}^{6}_{c}(1,1)$ to $5^{th}$ layer. The decomposition generates a set of activation maps $\{\mathbf{A}^{L}\}$ at $5^{th}$ layer for $\mathbf{F}^{6}_{c}(1,1)$. We use Eqn. (4) to select the top $N$ (e.g., $N$=3) important activation maps, i.e., $\mathbf{A}^{5}_{389}$, $\mathbf{A}^{5}_{277}$, and $\mathbf{A}^{5}_{259}$. We continue to decompose the decisions from $\mathbf{F}^{5}_{389}$, $\mathbf{F}^{5}_{277}$, $\mathbf{F}^{5}_{259}$, and find the top $N$ most important activation maps at $4^{th}$ layer for them, respectively. However, directly decomposing the feature map is not easy. Because not all of the features in a feature map contribute to decision (see the activation maps in the top row of Fig. 4). We select the most representative feature that contributes most to a decision and decompose this feature. We utilize Eqn. (5) to find the feature $\mathbf{F}_{k,\hat{x},\hat{y}}^{l}$ corresponding to the maximum activation. Then we decompose it to layer $l-1$ using gAP. This hierarchical decomposition process recursively runs until we decompose the CNN decision to the lowest layer.

The number of visualizations $N$ is a flexible parameter, which controls how many top response feature channels will be selected during each decomposition. To make human cognition easier, $N$ is set to 3 in Fig. 5. Moreover, we make the hierarchical decomposition interactive, so that the users can choose the features to be decomposed, easily accessing the information they need. We also provide a video about the interactive demo, shown in supplementary materials. In Fig. 5, we can see that the features detected in high-level layers can be decomposed to different parts detected in low-level layers. The hierarchical decomposition process tracks important features and recursively explains the evidence using evidence from lower layers. For instance, the classification results of ‘person’ have been decomposed to ‘face’ and ‘hand’ evidence. The ‘face’ evidence is then decomposed to ‘eye’, ‘nose’, and ‘lower jaw’. This process continues until we reach the lowest layer, which usually detects edge and blob features.

The Pearson Correlation Coefficient (PCC) metric [71] is utilized to measure the linear correlations between ground truth contribution $\hat{\alpha}^{l}\in\mathbb{R}^{K^{l}}$ and the contribution $\alpha^{l}\in\mathbb{R}^{K^{l}}$ estimated by Eqn. (4). When the PCC value equals 1, there are linear correlations between the two variables (0 denotes no linear correlations, -1 denotes total negative linear correlations). The PCC metric is computed by

where $\mu$ and $\sigma$ denote the mean and variance, respectively.

As shown in Tab. I, we study several strategies of calculating the contribution of a feature channel to the decision of interest. It can be seen that the contribution computed by averaging activations (i.e., Eqn. (4)) obtains the highest PCC value with the ground truth. For all stages in VGG-16, there are strong linear correlations between the computed contributions and the ground truth. This high correlation verifies the effectiveness of the gAP module.

Taking the computational efficiency into account, it is rather a time-consuming style of measuring channel contributions by calculating the score drop when removing feature channels iteratively in a layer [21, 70, 60]. In comparison, only one backpropagation process is needed when gAP calculates the channel contributions to a decision. On a VGG-16 backbone, calculating the ground truth channel contributions of an image takes about 10s, while the gAP module only takes about 50ms, nearly 200x faster. With the efficiency advantages of the gAP module, our hierarchical decomposition process can immediately yield detailed explanations of a network decision.

In Fig. 7(a), the low spearman metric indicates that the attribution maps from the original model and the randomly initialized model differ substantially, which demonstrates that gAP is sensitive to model parameters. In Fig. 7(b), the low spearman metric also indicates gAP is sensitive to the labeling of the data. The experimental results verify that our method can be used for debugging models. The visual comparisons are shown in supplementary materials.

As shown in Fig. 10, we use the hierarchical decomposition to examine the CNN’s wrong decision. Fig. 10 demonstrates a failure case. A dog image misclassified to the cat category with a probability of 99%. We first decompose the network decision to layer conv5_3 and find the most important channel, i.e., the $328^{th}$ channel, with a 32.3% contribution. We further present the decomposition from channel $328^{th}$ to layer conv4_3 and find the most important channel, i.e., the $330^{th}$ channel. The activation map $\mathbf{A}_{330}^{4}$ has strong activations at the ear region. Moreover, the patterns that maximumly activate the $330^{th}$ channel are the ear image patches of the cat category. We find the dog’s ear of this example has a similar shape to those ear image patches of the cat category. We further occlude the image region of the dog’s ear and observe that the CNN correctly predicts the dog category with a probability of 65%. With the hierarchical decomposition, we found that CNN makes the wrong decision because it takes the dog’s ear as the cat’s ear in this example.

As shown in Fig. 11(b), when comparing the adversarial image to the original image, we observe that the peak feature responses of important channels for the picket$\_$fence category, i.e., the $181^{st}$, $146^{th}$, $100^{th}$, $499^{th}$ channels, largely decrease by 11.3, 14.5, 4.7, and 11.1. However, the peak feature responses of important channels for the church category, i.e., the $33^{rd}$, $440^{th}$, $188^{th}$, $466^{th}$ channels, largely increase by 8.7, 10.4, 16.4, and 5.5. As shown in Fig. 11(c), we also compute the average peak responses of important channels on the whole ILSVRC validation dataset [75]. The adversarial attack algorithms change the feature responses of important channels to affect the final network decision. For the important channels, they reduce the correct category’s feature responses and increase the wrong category’s feature responses.

To quantitatively analyze the context information contained in the activation maps, we utilize the PASCAL-Context dataset [79] for evaluation. We select the images with context annotations from the PASCAL VOC validation set [69] and compute the most frequent context labels for each category. Specifically, we perform the hierarchical decomposition to layer conv4_3, obtaining the activation map for each selected channel. The activation map is first thresholded to a binary map. Then we compute the IoU between the binary activation map and each context region. The activation map is assigned with the label of the context region corresponding to the largest IoU. In Fig. 13, we have shown the top few most frequent context labels for three categories, i.e., bird, boat, and train. These categories usually appear in a specific environment. This fact suggests that the context of the objects is critical for recognition. The context information of other categories and the qualitative examples are shown in the supplementary materials.

We apply the hierarchical decomposition to different CNNs. As shown in Fig. 14, the channels’ discriminative degrees in low-level layers are very small. They usually have strong activations for multiple categories. This fact indicates that the basic features detected by channels in low-level layers are shared among different categories, which lacks discriminative information for classification. However, in high-level layers of CNNs, the channels’ discriminative degrees are much larger than those in low-level layers. Because the high-level layers in CNNs gradually combine basic features from low-level layers to form more discriminative features. In high-level layers, different categories tend to highlight their own discriminative channels. These results provide additional evidence for the conclusion found by Zeiler et al. [24].

Moreover, for the high-level layers of different CNNs, the discriminative degrees of the channels gradually increase with the growth of the network depth (ResNet-50 [2] $>$ VGG-16 [1] $>$ AlexNet [75]). Such difference suggests that the high-level layers of ResNet-50 have a stronger discriminative ability. The strong discriminative ability of the channels can effectively reduce confusion among different categories, which helps ResNet-50 achieve higher classification accuracy than VGG-16 and AlexNet.