SCOUTER: Slot Attention-based Classifier for Explainable Image Recognition

There are mainly three XAI paradigms [43], i.e. post-hoc, intrinsic, and distillation. The post-hoc paradigm usually provides a heat map highlighting important regions for the decision (e.g. [31, 30]). The heat map is computed besides the forward path of the model. The intrinsic paradigm explores the important piece of information within the forward path of the model, e.g., as attention maps (e.g. [21, 40, 24, 42]). Distillation methods are built upon model distillation [16]. The basic idea is to use an inherently transparent model to mimic the behaviors of a black-box model (e.g. [48, 29]).

The post-hoc paradigm has been extensively studied among them. The most popular type of methods is based on channel activation or back-propagation, including CAM [49], GradCAM [31], DeepLIFT [32], and their extensions [3, 25, 36, 35, 7]. Another type of method is perturbation-based, including Occlusion [45], RISE [27], meaningful perturbations [11], real-time saliency [5], extremal perturbations [10], I-GOS [28], IBA [30], etc. These methods basically give attributive explanation, which visualizes support regions of learned patterns for each category ${l}$ in the set of all possible categories $\mathcal{L}$. This visualization can be done by finding regions in feature maps or the input image that give large impact on the score $g_{l}$. By definition, attributive explanation is the same as our positive explanation.

Some of the methods for attributive explanation thus can be extended to provide negative explanations by negating the sign of the score, feature maps, or gradients. It should be noted that the interpretation of visual explanation by gradient-based methods [31, 3, 25] may not be straightforward because of linearization of $g_{l}$ for the given image; and thus the resulting visualization may not highlight the support regions for negative patterns. GradCAM [31] refers to its negative variant as counterfactual explanation that gives regions that can change the decision, emphasizing how it should be interpreted.

Discriminant explanation is a new type of XAI in the post-hoc paradigm, which appeared in [38] to show “why image $x$ belongs to category ${l}$ rather than ${l}^{\prime}$.” This can be interpreted using set $\mathcal{S}_{l}^{+}$ of all possible positive patterns for ${l}$ and set $\mathcal{S}_{{l}^{\prime}}^{-}$ of all possible negative patterns for ${l}^{\prime}$: It try to spot a (combination of) discriminative pattern $s$ that is in the intersection $\mathcal{S}_{l}^{+}\cap\mathcal{S}_{l^{\prime}}^{-}$. Due to the unavailability of negative patterns, the method [38] first finds (a combination of) positive patterns and uses the complementary of the region containing the positive patterns as a proxy of negative patterns. Goyal et al. gives another line of counterfactual explanation in [13]. Given two images of categories $l$ and $l^{\prime}$, they find the region in the image of $l$, of which replacement to a certain region in the image of $l^{\prime}$ changes the prediction from $l$ to $l^{\prime}$. This can be also implemented using discriminant explanation.

SCOUTER computes a heat map to spot regions important for the decision in the forward path, so it falls into the intrinsic paradigm. Together with the dedicated loss, it can directly identify positive and negative patterns with control over the size of patterns.

Self-attention is first introduced in the Transformers [34], in which self-attention layers scan through the input elements one by one and update them using the aggregation over the whole input. Initially, self-attention is used in place of recurrent neural networks for sequential data, e.g., natural language processing [8]. Recently, self-attention is adopted to the computer vision field, e.g., Image Transformer [26], Axial-DeepLab [37], DEtection TRansformer (DETR) [1], Image Generative Pre-trained Transformer (Image GPT) [4], etc. Slot attention [23] is also based on this mechanism to extract object-centric features from images (there are some other works [15, 12] using the concept of slot); however, the original slot attention is tested only on some synthetic image datasets. SCOUTER is based on slot attention but is designed to be an explainable classifier applicable to natural images.

In the original slot attention mechanism [23], a slot is a representation of a local region aggregated based on the attention over the feature maps. A single slot attention module with multiple slots is attached on top of the backbone network $B$. Each slot produces its own feature as output. This configuration is handy when there are multiple objects of interest. This idea can be transferred to spot the supports in the input image that leads to a certain decision.

The main building block of SCOUTER is the xSlot attention module, which is a variant of the slot attention module tailored for SCOUTER. Each slot of the xSlot attention module is associated with a category and gives the confidence that the input image falls into the category. With the slot attention mechanism, the slot for category $l$ is required to find support $\mathcal{S}_{l}$ in the image that directly correlates to $l$.

Given feature $F$, the xSlot attention module updates slot $w^{(t)}_{l}$ for $T$ times, where $w^{(t)}_{l}$ represents the slot after $t$ updates and $l\in\mathcal{L}$ is the category associated to this slot. The slot is initialized with random weights, i.e.,

$$w^{(0)}_{l}\sim\mathcal{N}(\mu,\text{diag}(\sigma))\in\mathbb{R}^{1\times c^{\prime}},$$

(2)

where $\mu$ and $\sigma$ are the mean and variance of a Gaussian, and $c^{\prime}$ is the size of the weight vector. We denote the slots for all categories by $W^{(t)}\in\mathbb{R}^{n\times c^{\prime}}$.

The slot $W^{(t+1)}$ is updated using $W^{(t)}$ and feature $F$. Firstly, $F$ goes through a $1\times 1$ convolutional layer to reduce the number of channels and the ReLU nonlinearity as $F^{\prime}=\operatorname{ReLU}(\operatorname{Conv}(F))\in\mathbb{R}_{+}^{c^{\prime}\times d}$, with $F$’s spatial dimensions being flattened ($d=hw$). $F^{\prime}$ is augmented by adding the position embedding to take the spatial information into account, following [34, 1], i.e. $\tilde{F}=F^{\prime}+\text{PE}$, where PE is the position embedding. We then use two multilayer perceptrons (MLPs) $Q$ and $K$, each of which has three FC layers and the ReLU nonlinearity between them. This design is for giving more flexibility in the computation of query and key in the self-attention mechanism. Using

$$Q(W^{(t)})\in\mathbb{R}^{n\times c^{\prime}},\quad K(\tilde{F})\in\mathbb{R}^{c^{\prime}\times d},$$

(3)

we obtain the dot-product attention $A^{(t)}$ using sigmoid $\sigma$ as

$$A^{(t)}=\sigma(Q(W^{(t)})K(\tilde{F}))\quad\in(0,1)^{n\times d}.$$

(4)

The attention is used to compute the weighted sum of features in the spatial dimensions by

$$U^{(t)}=A^{(t)}{F^{\prime}}^{\top}\quad\in\mathbb{R}^{n\times c^{\prime}},$$

(5)

and slot $W^{(t)}$ is eventually updated through a gated recurrent unit (GRU) as

$$W^{(t+1)}=\operatorname{GRU}(U^{(t)},W^{(t)}),$$

(6)

taking $U^{(t)}$ and $W^{(t)}$ as input and hidden state, respectively. Following the original slot attention module, we update the slot for $T=3$ times.

The output of the xSlot attention module is the sum of all elements for category $l$ in $U^{(T)}$, which is a function of $F$. Formally, the output of xSlot Attention module is:

$$\text{xSlot}(F)=U^{(T)}\mathbf{1}_{c^{\prime}}\quad\in\mathbb{R}_{+}^{n},$$

(7)

where $\mathbf{1}$ is the column vector with all $c^{\prime}$ elements being 1.

From Eqs. (5) and (7), we have $\text{xSlot}(F)=A^{(T)}F^{\prime\top}\mathbf{1}_{c^{\prime}}$, where $F^{\prime\top}\mathbf{1}_{c^{\prime}}\in\mathbb{R}^{d}$ is a reduction of $F$ and is a class-agnostic map. The $l$-th row of $A^{(T)}$ can then be viewed as spatial weights over map $F^{\prime\top}\mathbf{1}_{c^{\prime}}$ to spot where the support regions for category $l$ is²²2$F^{\prime\top}\mathbf{1}_{c^{\prime}}$ can be viewed as a single map that includes a mixture of supports for all categories.. In order for visualizing the support regions, we reshape and resize each row of $A^{(T)}$ to the input image size.

Note that in the original slot attention module, a linear transformation is applied to the features, i.e., $V(\tilde{F})$, which is then weighted using Eq. (5). However, the xSlot attention module omits this transformation as it already has a sufficient number of learnable parameters in $Q$, $K$, GRU, etc., and thus the flexibility. Also, the confidences, given by Eq. (7), are typically computed by an FC layer, while SCOUTER just sums up the output of xSlot attention module, which is actually the presence of learned supports for each category. This simplicity is essential for a transparent classifier as discussed in Section 3.3.

The whole model, including the backbone network, can be trained by simply applying softmax to xSlot$(F)$ and minimizing cross-entropy loss $\ell_{\text{CE}}$. However, there is a phenomenon that, in some cases, the model prefers attending to a broad area (e.g., a slot covers a combination of several supports that occupy large areas in the image) depending on the content of the image. As argued in Section 1, it can be beneficial to have control over the area of support regions to constrain the coverage of a single slot.

Therefore, we design the SCOUTER loss to limit the area of support regions. The SCOUTER loss is defined by

$$\ell_{\operatorname{SCOUTER}}=\ell_{\operatorname{CE}}+\lambda\ell_{\operatorname{Area}},$$

(8)

where $\ell_{\operatorname{Area}}$ is the area loss, $\lambda$ is a hyper-parameter to adjust the importance of the area loss. The area loss is defined by

$$\ell_{\operatorname{Area}}=\mathbf{1}_{n}^{\top}A^{(T)}\mathbf{1}_{d},$$

(9)

which simply sums up all the elements in $A^{(T)}$. With larger $\lambda$, SCOUTER attends smaller regions by selecting fewer and smaller supports. On the contrary, it prefers a larger area with smaller $\lambda$.

The model with the SCOUTER loss in Eq. (8) can only provide positive explanation since larger elements in $A^{(T)}$ means the prediction is made based on the corresponding features. We introduce a hyper-parameter $e\in\{+1,-1\}$ in Eq. (7), i.e.,

$$o=\text{xSlot}_{e}(F)=e\cdot U^{(T)}\mathbf{1}_{c^{\prime}}\quad\in\mathbb{R}_{+}^{n},$$

(10)

where $o=\{o_{1},\dots,o_{n}\}$. This hyper-parameter configures the xSlot attention module to learn to find either positive or negative supports.

With the softmax cross-entropy loss, the model learns to give the largest confidence $o_{l}$ corresponding to ground-truth (GT) category $l$ and a smaller value $o_{l^{\prime}}$ to wrong category $l^{\prime}\neq l$. For $e=+1$, all elements given by $\operatorname{xSlot}$ is non-negative since both $A^{(T)}$ and $F^{\prime}$ are non-negative and thus $U^{(T)}$ is. For arbitrary non-negative $F^{\prime}$, thanks to simple reduction in Eq. (7), larger $o_{l}$ can be produced only when some elements in $a^{(T)}_{l}$, the row vector in $A^{(T)}$ corresponding to $l$, is close to 1, whereas a smaller $o_{l^{\prime}}$ is given when all elements in $a^{(T)}_{l}$ are close to 0. Therefore, by setting $e$ to $+1$, the model learns to find the positive supports $\mathcal{S}^{+}_{l}$ among the images of the GT category. The visualization of $a^{(T)}_{l}$ thus serves as positive explanation, as in Fig. LABEL:fig_story (left).

On the contrary, for $e=-1$, all elements in $o$ are negative and thus the prediction by Eq. (10) gives $o_{l}$ close to 0 for correct category $l$ and smaller $o_{l^{\prime}}$ for non-GT category $l^{\prime}$. To make $o_{l}$ close to 0, all elements in $a^{(T)}_{l}$ must be close to 0, and a smaller $o_{l^{\prime}}$ is given when $a^{(T)}_{l^{\prime}}$ has some elements close to 1. For this, the model learns to find the negative supports $\mathcal{S}_{-}$ that do not appear in the images of the GT category. As a result, $a^{(T)}_{l^{\prime}}$ can be used as negative explanation, as shown in Fig. LABEL:fig_story (right).

We chose to use the ImageNet dataset [6] for a detailed evaluation of SCOUTER, because of the following three reasons: (i) It is commonly used in the evaluation of classification models. (ii) There are many classes with similar semantics and appearances, and the relationships among them are available in the synsets of the WordNet, which can be used to evaluate positive and negative explanations. (iii) Bounding boxes are available for foreground objects, which helps measure the accuracy of visual explanation. In experiments, we use subsets of ImageNet by extracting the first $n$ ($0<n\leq 1,000$) categories in the ascending order of the category IDs. Also, we present classification performance on Con-text [20] and CUB-200 [39] datasets and illustrate glaucoma diagnosis using quantitative and qualitative results on ACRIMA [9] dataset.

The size of images is $260\times 260$. The feature $F$ extracted by the backbone network is mapped into a new feature $F^{\prime}$ with the channel number $c^{\prime}=64$. The models were trained on the training set for $20$ epochs and the performance scores are computed on the validation set with the trained models after the last epoch. All the quantitative results are obtained by averaging the scores from three independent runs.

To evaluate the quality of visual explanation, we use bounding boxes provided in ImageNet as a proxy of the object regions and compute the percentage of the pixels located inside the bounding box over the total pixel numbers in the whole explanation. Specifically, for set $I$ of all pixels in the input image and set $D$ of all pixels in the bounding box, we define the explanation precision as $\operatorname{Precision}_{l}=\frac{\sum_{i\in D}a^{l}_{i}}{\sum_{i\in I}a^{l}_{i}}$, where category $l\in\mathcal{L}$ and $a^{l}_{i}\in[0,1]$ is the value of pixel $i$ in $\bar{A_{l}}$, which is attention map $A$ resized to the same size as the input image by bilinear interpolation. We compute this metric on the visualization results of the GT category for positive explanations and on the least similar class (LSC) (using Eq. 11) for negative explanations, as LSC images usually show strong and consistent negative explanations. This precision metric actually is a generalization of the pointing game [47], which counts one hit when the point with the largest value on the heat map locates in the bounding box and the final score is calculated as $\frac{\operatorname{\#Hits}}{\operatorname{\#Hits}+\operatorname{\#Misses}}$.

We also adopt several other metrics, i.e., (i) insertion area under curve (IAUC) [27], which measures the accuracy gain of a model when gradually adding pixels according to their importance given in the explanation (heat map) to a synthesized input image; (ii) deletion area under curve (DAUC) [27], which measures the performance drop when gradually removing important pixels from the input image; (iii) infidelity [44], which measures the degree to which the explanation captures how the prediction changes in response to input perturbations; and (iv) sensitivity [44], which measures the degree to which the explanation is affected by the input perturbations. In addition, we calculate the (v) overall size of the explanation areas by $\operatorname{Area}_{l}=\sum_{i\in I}a^{l}_{i}$, as for some applications, a smaller value is better to pinpoint the supports to differentiate one class from the others.

We conduct the explainability experiments with the ImageNet subset with the first 100 classes. We train seven models with (1) an FC classifier, (2)–(4) SCOUTER₊ ($\lambda=1,3,10$), and (5)–(7) SCOUTER_- ($\lambda=1,3,10$) using ResNeSt 26 [46] as the backbone. The results of competing methods are obtained from the FC classifier-based model. In addition, as introduced in Section 2.1, some of the existing works can give negative explanations. Therefore, we also implement and compare our results with their negative variants by using negative feature maps/gradients or modifying their objective functions.

The numerical results are shown in Table 1. We can see that SCOUTER can generate explanations with different area sizes while achieving good scores in all metrics. These results demonstrate that the visualization by SCOUTER is preferable in terms of controlling area sizes, high precision, insensitive to noises (sensitivity), and with good explainability (infidelity, IAUC, and DAUC). Among the competing methods, extremal perturbation [10], I-GOS [28], and IBA [30] also take the size of support regions into account, and thus some of them give smaller explanatory regions. Extremal perturbation’s explanatory regions cover some parts of foreground objects. This leads to a high precision score, but the performance over other metrics is not satisfactory. I-GOS and IBA give small explanation areas. I-GOS results in low IAUC and sensitivity scores. IBA gets relatively low scores of IAUC and DAUC, which means its explanations cannot give correct attention to the pixels and thus does not have enough explainability.

It is arguable that area size can be controlled by thresholding the heatmap. In order to verify this, we set a threshold ($a\geq 0.2$) to one of the back-propagation-based methods (SS-CAM) to get explanations with smaller size. We can see that this variant suffers a deterioration in IAUC and DAUC (significantly worse than I-GOS, IBA, and SCOUTER), which represents a large explainability drop and hampers its uses in actual applications requiring small explanations.

To further explore the explanation for non-GT categories, we define the semantic similarity between categories based on [41], which uses WordNet, as

$$\operatorname{Similarity}=2\frac{d(\operatorname{LCS}(l,l^{\prime}))}{d(l)+d(l^{\prime})},$$

(11)

where $d(\cdot)$ gives the depth of category $l$ in WordNet, and $\operatorname{LCS}(l,l^{\prime})$ is to find the least common subsumer of two arbitrary categories $l$ and $l^{\prime}$. We define the highly-similar categories as the category pairs with a similarity score no less than $0.9$, similar categories as with a score in $[0.7,0.9)$, and the remaining categories are regarded as dissimilar categories. Table 2 summarizes the area sizes of the explanatory regions for GT, highly-similar, similar, and dissimilar categories. We see a clear trend: SCOUTER₊ decreases the area size when the inter-category similarity becomes lower, while SCOUTER_- gives larger explanatory regions for the dissimilar categories.

Some visualization results are given in Figs. 2 and 3. It can be seen that SCOUTER gives reasonable and accurate explanations. Comparing SCOUTER₊’s explanation with SS-CAM [36], and IBA [30], we find that SCOUTER₊ can give explanations with more flexible shapes which fit the target objects better. For example, in the first row of Fig. 2, SCOUTER₊ gives more accurate attention around the neck. In the second row, it accurately finds the individual entities. Compared with SS-CAM, IBA shows smaller explanatory regions. However, IBA is less precise and less reasonable, which is consistent with the numerical results in Table 1.

In Fig. 3, SCOUTER_- can also find the negative supports, e.g., the wattle of the hen, and the hammerhead and the fin of the shark. In addition, although the negative variation of S-GradCAM++ performs well on the first row, its explanation in the second row does not well fit the object’s shape and fails to pinpoint the key difference (the head).

We compare SCOUTER and FC classifiers with several commonly used backbone networks with respect to the classification accuracy. The results are summarized in Fig. 4. With the increase of the category number, both the FC classifier and SCOUTER show a performance drop. They show similar trends with respect to the category number.

The relationship between $\lambda$, which controls the size of explanatory regions, and the classification accuracy is shown in Fig. 5 for ResNeSt 26 model with $n=100$. A clear pattern is that the area sizes of both SCOUTER₊ and SCOUTER_- drop quickly with the increase of $\lambda$. However, there is no significant trend in the classification accuracy, which should be because the cross-entropy loss term works well regardless of $\lambda$.

Also, according to the visualization results in Figs. 2 and 3, a larger $\lambda$ does not simply decrease the explanatory regions’ sizes. Instead, SCOUTER shifts its focus from some larger supports to fewer, smaller yet also decisive supports. For example, in the first row of Fig. 3, when $\lambda$ is small, SCOUTER_- can easily make a decision that the input image is not a cock because of unique feathers on the neck. With a larger $\lambda$, SCOUTER finds smaller combinations of supports (i.e., its wattle) and thus the explanation changes from the (larger) neck to the (smaller) wattle region.

We also summarize the classification performance of the FC classifier, SCOUTER₊ ($\lambda=10$), and SCOUTER_- ($\lambda=10$) over ImageNet [6], Con-text [20], and CUB-200 [39] datasets in Table 3. The subsets with $n=100$ are adopted for ImageNet and CUB-200, while all 30 categories are used for the Con-text. The results show that SCOUTER can be generalized to different domains and has a comparable performance with the FC classifier over all datasets. Also, SCOUTER’s number of parameters is comparable to FC’s (more details in supplementary material).

One drawback of SCOUTER is that its training is unstable when $n>100$. This is possibly because of the increasing difficulty in finding effective supports that consistently appear in all images of the same category but are not shared by other categories. This drawback limits the application of SCOUTER to small- or medium-sized datasets.

SCOUTER uses the area loss, which constrains the size of support. This constraint can benefit some applications, including the classification of medical images, since small support regions can precisely show the symptoms and are more informative in some cases. Also, there was no method that could give the negative explanation but it is actually needed (doctors need reasons to deny some diseases). SCOUTER was designed upon these needs and is being tested in hospitals for glaucoma (Fig. 6), artery hardening (supplementary material), etc.

For glaucoma diagnosis, we tested SCOUTER with $\lambda=10$ over a publicly available dataset, i.e., ACRIMA [9], which has two categories (normal and glaucoma). ResNeSt 26 is used as backbone. The results are shown in Table 4. We can see that both SCOUTER₊ and SCOUTER_- get better performances than the FC classifier. Besides, SCOUTER is preferred in this task as doctors are eager to know the precise regions in the optic disc that lead to the machine diagnosis. We can see that, in the visualization results in Fig. 6, SCOUTER shows more precise and reasonable explanations that locate on some vessels in the optic disc and show clinical meanings (vessel shape change due to the enlarged optic cup), which are verified by doctors. Although IBA also gives small regions, they cover some unrelated or uninformative locations. In addition, it is notable that the facts to admit category “Glaucoma” need not to match with the facts to deny “Normal”, as they are only subsets of the support sets and an on-purpose negative explanation is especially helpful for the doctors when the machine decisions are against their expectations.