Toward Scalable and Unified Example-based Explanation and Outlier Detection

For the purpose of providing meaningful explanations and detecting outliers, we introduce three generic head architectures²²2The term head architecture refers to the architecture used in the prototype network which is the portion of the network located after the CNN encoder. Figures 4-4 illustrate this. for our prototype student network. Each architecture is evaluated for its performance in providing explanation and detecting outliers in Section V. For simplicity, we assume in this section that the prototypical examples are already selected based on our iterative prototype replacement algorithm. We will introduce the algorithm in Section III-B later.

Let $f(\cdot\ ;\theta_{1}):\mathbb{R}^{C_{0}\times H_{0}\times W_{0}}\rightarrow\mathbb{R}^{C\times H\times W}$ be the CNN feature extractor before the spatial pooling layer with the learnable parameters $\theta_{1}$. Let $e(\cdot\ ;\theta_{2}):\mathbb{R}^{C\times H\times W}\rightarrow\mathbb{R}^{\tilde{C}}$ be the prototype network with the learnable parameters $\theta_{2}$, and let $\tilde{C}$ be the number of classes in the classification task. The overall student network is represented by $e\ \circ f$. For simplicity, we omit the parameters $\theta_{1}$ and $\theta_{2}$ and denote the feature map representation of the input sample $x$ as $f(x)$ in this section.

To make a prediction, we have to solve the problem of computing a similarity between the feature map $f(x)$ for a test sample $x$ and the feature $f(p)$ for a prototype $p$. The feature maps $f_{c,h,w}(\cdot)$ have a channel dimension $c$ and spatial dimensions, usually the two $h,w$. We observe that three larger categories of similarities can be defined, which correspond to the head classes I, II and III. The Head I follows the simplest idea to compute a similarity without any spatial dimensions; thus, it pools over the spatial dimensions before computing an inner product over the channel dimension. The Head II class computes similarities that incorporate the spatial dimensions. The difference between the two heads in the Head II class is whether one matches each spatial coordinate $(h,w)$ in the test sample to the same coordinate in the prototype as in Head II-A or whether one searches for every coordinate in the test sample for the best matching coordinate in the prototype using a maximum-operation as in Head II-B. Finally, the Head III class also uses the spatial dimensions of feature maps but additionally learns an attention weight over the spatial coordinates of the features in the test sample. This serves to weight the spatial regions in the test samples in the final similarity measures, whereas the Head II class treats each spatial coordinate $(h,w)$ of the test sample feature map with equal weight.

Figure 1 shows the taxonomy of the various head architectures for the student network and Figures 4-4 are visualizations of the head architectures. Our task differs from the standard image retrieval task as our task uses prototypes from the training set that participate in the prediction and thus share quantifiable prediction reasoning as the test sample, whereas the image retrieval task involves an exhaustive nearest neighbor search over the entire database that does not participate in the prediction and thus does not guarantee similar prediction reasoning between the test sample and its nearest neighbor.

Our proposed head architectures do not require any modification to the CNN encoder compared to [14], which requires additional linear layers for queries and keys that add to the overhead during training.

The Head I architecture is based on a spatial average pooling of the feature maps $f(\cdot)$, resulting in a vector $g(\cdot)$ that consists of only $C$ feature channels. This is followed by a cosine similarity over the feature channels $g(\cdot)$, as seen in Figure 4:

The similarity output $s^{\text{(I)}}(p_{k})$ falls in the range of $[-1,1]$ where a higher value indicates higher similarity between the input sample $x$ and the prototype $p_{k}$. The linear classification layer $h(x,\{p_{1},p_{2},...,p_{K}\})=\sum^{K}_{k=1}w_{k}ReLU(s^{\text{(I)}}(p_{k}))+b$ then predicts the output logit $y$. For this head architecture, the learnable prototype network parameters $\theta_{2}$ denote the parameters of the linear layer $h(\cdot)$. The linear layer $h(\cdot)$ in Head I is consistent with the formulation of the dual kernel SVM. The linear layers $h(\cdot)$ in Heads II and III as shown in Figures 4 and 4 will have similar dependency on a set of prototypes $\{p_{1},p_{2},...,p_{K}\}$, but they no longer correspond to kernels.

The Head II architecture in Figure 4, uses the features before spatial pooling to compute similarity scores. Therefore, the similarity layer computes $s^{\text{(II)}}(p_{k})\in\mathbb{R}^{H\times W}$, i.e., one similarity score for each spatial position $(h,w)$. We explore two types of similarity functions for this head architecture: A) similarity between the input and prototype at the same spatial position $(h,w)$ and B) the maximum similarity between input and prototype at a given input spatial location $(h,w)$. We refer to the former as Head II-A and the latter as Head II-B. The similarity operation for Head II-A (Eq.(2)) and Head II-B (Eq.(3)) at the spatial location $(h,w)$ is defined as:

where $f_{c,h,w}(x)$ and $f_{c,h,w}(p_{k})$ are the $c$-th features at the spatial location $(h,w)$ from the feature maps $f(x)$ and $f(p_{k})$, respectively. Head II-A compares the pairwise similarity of the spatial features between $f(x)$ and $f(p_{k})$ along the same spatial location, whereas Head II-B considers all spatial locations of $f(p_{k})$ to find the location that is most similar to the specified spatial location in $f(x)$. Therefore, Eq. (3) ensures that the similarity at the spatial location $(h,w)$ is computed only between the location $(h,w)$ in $x$ and the most relevant location $(h^{\prime},w^{\prime})$ in $p_{k}$, i.e., the pair that locally gives the highest cosine similarity score. Its similarity operation is spatially invariant, as it considers all spatial locations of prototype $p_{k}$ in identifying the most similar feature to the input $x$. Head II-A (Eq. (2)) can be viewed as a special case of Head II-B (Eq. (3)). The operation of Head II-B reduces to Head II-A when the maximum similarity between input and prototype for a given input spatial location $(h,w)$, also occurs at the prototype location $(h,w)$. Thus, the Head II-B architecture is considered the general case of Head II-A. For this category of architectures, the learnable prototype network parameters $\theta_{2}$ denote the parameters of the linear layer $h(\cdot)$. Note that Head II-B, as a maximum over inner products, no longer defines a kernel, as the maximum of two positive definite matrices is not guaranteed to be a positive definite. Thus, we employ a nonkernel-based similarity.

The Head III architecture employs a common attention mechanism to compute weighted similarities. For the Head III architecture as shown in Figure 4, the similarity scores from Eq. (2) and (3) are passed into a softmax $\sigma(\cdot)$ to obtain the following attention maps:

Head III-A and Head III-B are the attention-augmented counterparts of Head II-A and Head II-B, respectively:

The indices $h_{\max}$ and $w_{\max}$ in Eq. (8) are the selected $h^{\prime}$ and $w^{\prime}$ indices used in the computation of $s^{\text{(II-B)}}_{h,w}(p_{k})$. The attention map $a(p_{k})\in\mathbb{R}^{H\times W}$ is applied over the spatial features $f(x)\in\mathbb{R}^{C\times H\times W}$ and $f(p_{k})\in\mathbb{R}^{C\times H\times W}$ to give more weight to the important regions (i.e., spatial regions between $f(x)$ and $f(p_{k})$ with high similarity) for the neural network to learn and focus on these regions. The output from the attended similarity layer which represents a weighted spatial similarity for every channel, is passed into a convolutional-1D layer with kernel size $C$ to compute a weighted sum of the $C$ features. Since $C\gg H\times W$, we avoid using simple average spatial pooling over the $C$ features to obtain better explanation heatmaps. The convolutional-1D layer has a minimum weight clipping at zero to ensure the outputs are always nonnegative in order to apply the explainability method described in Section III-D. Additionally, the use of individual attention maps for the respective features $f(x)$ and $f(p_{k})$ is also investigated in Head III-C:

where $a^{\text{(III-B)}}_{h,w}$ is the attention score from Eq. (6) for $f(x)$ at the input spatial location $(h,w)$, whereas $a^{\text{(III-C)}}_{h,w}$ is the attention score for $f(p_{k})$ at the prototype spatial location $(h,w)$, as can be seen by the computation of $\max_{u,v}(\cdot)$ over the spatial locations of $\hat{f}(x)$. Eq. (9) computes an attention map by considering all spatial locations of $f(x)$ to find the most similar location to the specified location in $f(p_{k})$ as opposed to the attention map based on Eq. (3). The attention map $a^{\text{(III-B)}}(p_{k})\in\mathbb{R}^{H\times W}$ with weights denoting the importance of the spatial positions in $f(x)$ based on its similarity with $f(p_{k})$, is applied over the feature $f(x)$. On the other hand, the attention map $a^{\text{(III-C)}}(p_{k})\in\mathbb{R}^{H\times W}$ with importance weights for the spatial positions in $f(p_{k})$ based on its similarity with $f(x)$, is applied over the feature $f(p_{k})$. For the head architectures in this category, the learnable prototype network parameters $\theta_{2}$ are the parameters of the convolutional-1D and the linear classification $h(\cdot)$ layers.

For the selection of representative prototypes, one can directly learn an importance mask over the entire training set and select those samples with high importance as prototypes. This approach, however, does not scale well to large sample sizes as the number of parameters that must be learned increases proportionally. For approaches such as [14], the database of prototype candidates used must be sufficiently large for successful training as reported in their paper. Instead of learning an importance mask over the entire training set or a large candidate database, we propose to learn a smaller importance mask over a fixed number of $K$ prototypes. We note that $K$ is very much smaller than the size of the candidate database [14] used during training or inference. Our approach scales well to large sample sizes since the number of parameters to be learned can now be constrained by $K$ which also corresponds to the final number of prototypes used during inference. We introduce an auxiliary output branch to guide the network to learn a prototype replacement scheme that replaces uninformative prototypes (out of the $K$ prototypes) with new samples based on the importance or contribution of the current prototypes to the decision boundary of the network. Prototypes that are assigned larger importance weights have higher importance and are unlikely to be replaced. On the other hand, prototypes with smaller importance weights are likely to be replaced with other samples from the training set due to their minimal contribution to the decision boundary of the network.

During training, the network learns a prototype importance weight $\mathcal{M}\in\mathbb{R}^{K}$ via an auxiliary loss to quantify the importance of the $K$ prototypes based on their contribution to the prediction. The auxiliary loss imposes a penalty when informative prototypes that influence the decision boundary of the network are removed, i.e., masked out. The network parameters $\theta_{1}$ and $\theta_{2}$ and the prototype importance weight $\mathcal{M}$ are jointly optimized. The formulation of the auxiliary loss will be elaborated further in Section III-C.

The full algorithm for the iterative prototype replacement method is presented in Algorithm 1. We compute the binary prototype mask $\mathcal{M}^{\prime}(\tau_{t})$ from the prototype importance weight $\mathcal{M}$ using the threshold $\tau_{t}$ at iteration $t$ as:

We use continuous ReLU activation with normalization as inspired by the hard shrinkage operation in [30] to avoid issues with the backpropagation of a discontinuous 0-1 step function. For simplicity, we assign $\tau_{t}$ as the $p$-th smallest weight in $\mathcal{M}$ to zero out a fixed number of $p$ prototypes (in the input of the linear classification layer $h(\cdot)$) with the smallest weight in $\mathcal{M}$. To compute the auxiliary output $y_{mask}$, we multiply the binary prototype mask $\mathcal{M}^{\prime}(\tau_{t})$ elementwise with the features $z(p_{k})\in\mathbb{R}$ at the input of the final linear classification layer $h(\cdot)$ corresponding to each prototype as follows:

where $m^{\prime}_{k}\in\{0,1\}$ is the weight in $\mathcal{M}^{\prime}(\tau_{t})$, and $w_{k}\in\mathbb{R}$ and $b\in\mathbb{R}$ are the parameters of the linear layer $h(\cdot)$. At the end of each epoch, we replace the $p$ prototypes that have the smallest weight in $\mathcal{M}$ with new samples from the training set $D$. The new set of prototypes $P$ also maintains an equal number of samples per class during training.

The prototype importance weight $\mathcal{M}$ can also be used to prune prototypes and relevant neurons. This will be discussed in Section VI-B.

We remark that it is not of interest to iterate through all the training samples when choosing optimal prototypes, as this will not scale well to large datasets. Our algorithm is based on the idea that one can find prototypes that may be similar to a number of training samples and thus can avoid iterating through all training samples when selecting prototypes. In future work, we will consider presorting the prototypes based on their similarity to maintain an order over training samples during learning to speed up model convergence. The prototype-based Head III architectures with higher architectural complexities than Head I and II architectures have greater training overhead than a standard CNN teacher network due to the computation of attention maps and training time which increases proportionally with the number of prototypes used. Additional details are provided in Section S-I of the Supplementary Material.

We define the set of input training samples with their ground truth to be $D=\{(x_{i},\tilde{y}_{i})\ |\ 1\leq i\leq N\}$ and the set of prototypes with their class labels to be $P=\{(p_{k},\tilde{q}_{k})\ |\ 1\leq k\leq K\}$. For an input sample $x$, we denote the output logits of the teacher network as $y_{T}$ and the logits and auxiliary output of the student network as $y$ and $y_{mask}$, respectively. The predicted class label from the student network for the input sample $x$ is represented by $y_{pred}=\arg\max y$. The following defines the objective function for our student network:

where $\mathcal{L}(\cdot)$ is the cross-entropy loss, $\mathcal{J}(\cdot)$ is a loss term based on the squared $L_{2}$-norm, and $\sigma(\cdot)$ is the softmax.

The first two terms are the losses commonly employed in the training of a student network. The second term, also known as the distillation loss, is the cross-entropy loss with the teacher soft labels as ground truth to encourage the student network to learn predictions resembling the teacher network.

The third term $\mathcal{L}(y_{pred},\sigma(y_{mask}))$ is used for the learning of prototype importance weight $\mathcal{M}\in\mathbb{R}^{K}$. We use the standard cross-entropy loss as the auxiliary loss but with the predicted class label and the probability $\sigma(y_{mask})$ as the input argument to the standard cross-entropy loss rather than using the ground truth class $\tilde{y}$ and the probability output $\sigma(y)$ from the main network branch. The auxiliary loss enables the network to learn prototype importance weight $\mathcal{M}\in\mathbb{R}^{K}$ which gives larger weights to prototypes that will cause a large change in the decision boundary of the network when removed, and lower importance weights to prototypes that cause only a small change to the decision boundary if removed. Our proposed solution is based on the idea of removing uninformative prototypes with minimal disruption to the decision boundary using the predicted class label $y_{pred}$ instead of the ground truth class label $\tilde{y}$ to guide the learning of parameters $\mathcal{M}\in\mathbb{R}^{K}$ since we believe the network can learn better by replacing uninformative prototypes with other potentially more informative prototypes. This allows the replacement of lower ranked prototypes that have a lower impact on the prediction accuracy in the prototype replacement step.

The fourth loss term $\mathcal{J}\left((x,\tilde{y}),(p,\tilde{q})\right)$ encourages the feature space of the input sample $x$ to be close to the set of prototypes from the same class but farther away from the other classes. The formulation of this objective term varies slightly between the different head architectures depending on their similarity operation.

For the Head I architecture, the objective term $\mathcal{J}\left((x,\tilde{y}),(p,\tilde{q})\right)$ is defined as:

For the Head II-A and Head III-A architectures, the objective function $\mathcal{J}\left((x,\tilde{y}),(p,\tilde{q})\right)$ is defined as:

The term inside the square bracket is simply the average squared $L_{2}$-norm computed for the channel dimension over all spatial positions $(h,w)$. Similarly, the objective function $\mathcal{J}\left((x,\tilde{y}),(p,\tilde{q})\right)$ for the Head II-B and Head III-B architectures is defined as:

The different formulation for Eq. (15) and Eq. (16) is due to the two different types of similarity operations introduced in the head architectures, as discussed in Section III-A. For Head III-C architecture, the objective function $\mathcal{J}(\cdot)$ is expressed as

where the first term is equivalent to Eq. (16) and the second term is defined accordingly based on the $a^{\text{(III-C)}}_{h,w}(p_{k})$ operation for $f(p_{k})$ in Head III-C, i.e., use $\hat{f}_{c,h_{max},w_{max}}(x_{i})$ instead of $\hat{f}_{c,h,w}(x_{i})$ and vice versa for $p_{k}$ in the loss $\mathcal{J}_{p}(\cdot)$.

The student network provides a prediction explanation in the form of: (1) a prediction explanation with top-$k$ nearest prototypical examples and (2) pixel evidence of similarity between the prediction sample and the prototypical examples. We define a prototype similarity score $u_{k}$ that ranges between $[0,1]$ to identify the top-$k$ nearest prototypical examples for a given input sample. We elaborate on the formulation of the prototype similarity score $u_{k}$ in Section III-E as the scores are also used to formulate the outlier score. In the following paragraph, we recapitulate the explanation method to compute pixel evidence of similarity between each pair of input-prototype.

To show evidence of similarity between the input sample and the prototypes in the pixel space, we use a posthoc explainability method, the Layer-wise Relevance Propagation (LRP) algorithm [3, 31] to compute a pair of LRP heatmaps for each pair of input-prototype. For a pair of input-prototype, their respective heatmaps comprise positive and negative relevance scores (also known as evidence) in the pixel space. The relevance scores represent the pixels’ contribution to the prediction score. For the 2D-convolutional layers, we used the following LRP-$\alpha\beta$ rule to compute relevance $R_{d}^{(l)}$ for the neuron $d$ at the current layer $l$:

where $a_{d}$ is the lower-layer activation from the input neuron $d$ at layer $l$, $w_{dj}$ is the weight connecting the neuron $d$ at layer $l$ to the higher-layer neuron $j$ at the succeeding layer $l+1$, $R_{j}^{(l+1)}$ is the relevance for the neuron $j$ at the succeeding layer $l+1$, $\alpha-\beta=1$, $\alpha>0$, and $\beta\geq 0$. For the other layers, including the similarity layer (Heads I and II) and the attended similarity layer (Head III), we use the following LRP-$\epsilon$ rule:

The similarity/attended similarity layer is treated as a linear layer, which enables the easy application of the LRP-$\epsilon$ rule in the standard manner without resorting to more complex LRP methods such as BiLRP [32]. For Head III architectures, the relevance scores flow through the attended similarity layer branch only while treating the attention weights from the similarity layer as constants.

From the last linear layer up to the similarity/attended similarity layer $l$, we compute the relevance scores $R_{(\cdot)}^{(l)}(x,p_{k})$ for each pair of $(x,p_{k})$. Using $R_{(\cdot)}^{(l)}(x,p_{k})$ at the similarity/attended similarity layer $l$, we compute the LRP heatmap for $x$ by backpropagating the relevances to the input space of $x$ through the CNN encoder. The heatmap for $p_{k}$ is also computed in a similar manner (for Head II-B and Head III-B architectures with $\max_{h^{\prime},w^{\prime}}(\cdot)$ operation, the relevance scores are accumulated at the relevant $h^{\prime}$ and $w^{\prime}$ indices before backpropagating to the input space of $p_{k}$). We refer the reader to [3] for more information on LRP algorithms.

Our proposed prototype-based architectures can naturally quantify the degree of outlierness of an input sample based on the sequence of sorted similarities even though they are not trained primarily for outlier detection. We revisit the method in [23] and generalize the confidence score in [14] to compute an outlier score based on the sum of similarity scores. We use the sum of the prototype similarity scores $u_{k}$ to compute the outlier score. While we see the main contribution in unifying two aspects of explanation and outlier detection, a smaller contribution lies in the different formulation of the prototype similarity scores $u_{k}$ with respect to the student architecture (refer to the output branch for outlier score in Figures 4-4).

For each input sample $x$, we define the set of corresponding prototype similarity scores as $U=\{u_{k}\ |\ 1\leq k\leq K\}$. For all Head I and Head II architectures, the $u_{k}$ score is assigned as $u_{k}=z(p_{k}),$ where $z(p_{k})$ is the input to the linear layer $h(\cdot)$ as shown in Figures 4 and 4. For Head III-A and Head III-B architectures, the $u_{k}$ score is computed as $u_{k}=\frac{1}{HW}\sum_{h=1,w=1}^{H,W}r_{h,w}(p_{k}).$ For the Head III-C architecture, the $u_{k}$ score is computed as $u_{k}=\max_{h,w}(r_{h,w}(p_{k})).$ With the placement of ReLU layers in the CNN, the elements of $U$ lie in the range of $[0,1]$, where a higher value indicates a larger similarity between the input sample $x$ and the prototype $p_{k}$. From the set $U$, top-$k^{\prime}$ highest scores are selected regardless of the prototype class. For sample $x$, the outlier score $o(x)$ is defined as:

Note that $u_{k}$ is in general no inner product, and thus, we cannot compute true metric distances and hence follow the idea in [23]. In this formulation, we assume that the prototype similarity scores $u_{k}$ for outliers are smaller than the normal samples. The above formulation is the general case, which considers prototypes from all classes in the summation, unlike the formulation of the confidence score in [14], which considers only the set of prototypes from the predicted class. We show only the results using Eq. (20), which gives better performance, and omit results using only prototypes from the same predicted class due to lack of space. Larger outlier score $o$ indicates a larger anomaly.

We begin the comparison with the predictive aspect of our proposed architectures and the other methods on both datasets. Based on Table I, it can be observed that the proposed student architectures outperform the teacher network and are better than the other methods on the LSUN dataset. The Head III-B student network, which computes an attention map based on the maximum similarity between input and prototype, has the best classification performance among the other student networks. The Head III-C student network, which computes individual attention maps for the input and the prototype, has similar performance to Head III-B but is computationally more expensive to train due to the additional attention map used. It can be observed that ProtoDNN underperforms severely on the LSUN dataset. A manual inspection of the prototypes from ProtoDNN reveals that the prototypes do not resemble any training samples and that the autoencoder used for prototype decoding fails to reconstruct the LSUN samples, resulting in poor classification performance. This implies that the joint optimization for the layers in ProtoDNN performs well only on simple grayscale datasets such as MNIST and Fashion MNIST [6] but is not generalizable to complex datasets such as LSUN. Our student network does not rely on autoencoders for prototype visualization, since the prototypes are actual training samples selected from the training set. On the basis of our own observation with outlier detection on simpler datasets and the inability of ProtoDNN to generalize to complex datasets such as LSUN, we refrain from using common and simple datasets such as MNIST and Fashion MNIST in our evaluation to provide a more challenging performance benchmark.

For the PCam and Stanford Cars datasets, as shown in Tables II and III, we show only one type of head per category. For the Head II and III categories, we show only performances of Head II-B and Head III-B architectures as they are the general cases of Head II-A and Head III-A, respectively, and have lower computational complexity than the Head III-C architecture. Based on Table II, ProtoPNet and our student network with Head I architecture demonstrate equally good performance on the PCam dataset. The former achieves the highest test accuracy (only marginally better than Head I), and the latter achieves the best AUROC score. The PCam dataset consisting of only 2 classes and similar image statistics has smaller intraclass variations than the LSUN dataset with its 10 classes. In this case, Head I, which computes similarity using spatially pooled features, overfits less, thus performing better than more complex architectures such as Head II-B and Head III-B.

In Table III, we compare the different head architectures only with the most competent state-of-the-art model, i.e., ProtoPNet. For the student networks trained on the Stanford Cars dataset, we observe similar performance trend as the student networks trained on PCam dataset since the Stanford Cars classification task is a fine-grained car model classification with significantly smaller intra- and interclass variations (similar to PCam samples) than the LSUN dataset. The architecture Head I has the best classification performance, surpassing the performance of ProtoPNet, while the attention-guided Head III-B architecture with higher complexity underperforms on this dataset. We used smaller learning rates for the Head III-B network on this dataset as the model overfits severely with higher learning rates as observed in our initial experiment. In an attempt to improve the performance of the model on this dataset, we also investigated several Head III-B variants such as a parameterized spatial attention and uniform attention variants but obtain unsatisfactory classification performance compared to the Head I and Head II-B networks. For more information, we refer the reader to Section S-V of the Supplementary Material. Due to small sample variations in the fine-grained dataset and a larger number of learnable parameters in the Head III-B architecture (the additional parameters in the 1D-CNN layer), the model has a higher tendency to overfit compared to simpler architectures such as Head I and Head II-B. Since fine-grained classification is a task that involves distinguishing between very similar objects, the network may need to focus on a smaller region of the image sample. Thus, Head III-B can potentially be improved further by sampling small prototype patches from the training set, as inspired by the learning of prototype latent patches in ProtoPNet, which permits the attention map in Head III-B to be applied more precisely over a smaller target region. We note that there is a performance gap between the performance of ProtoPNet in Table III and the performance reported in [8] (Table 1 of their Supplementary Material). The performance gap is due to our experimental setup, which is different from [8] as described earlier in Section IV, and the classification problem of the first $50$ classes that are naturally more challenging than the remaining Stanford Cars classes. For a complete discussion on this matter, we refer the reader to Section S-IV of the Supplementary Material.

For the remaining experiments using the PCam and Stanford Cars datasets, we evaluated only Head II-B and Head III-B for the Head II and Head III categories.

We conclude that using prototype-based students generally results in models that are equally competitive in prediction performance as the teacher network except for Head III-B on the Stanford Cars dataset. Employing the same underlying architecture and using soft labels from the teacher network to train the prototype networks allows unconstrained modeling of the original task. We note that our proposed models are built on top of a typical CNN model and thus have lower performance than more advance state-of-the-art methods that are curated to perform well on the task. To achieve performance comparable to Table IV, one may use any state-of-the-art model as the teacher network and use the soft labels and the same underlying architecture to train the prototype networks. We do not include the state-of-the-art performance on the Stanford Cars dataset in Table IV since we used only the first 50 car classes in our experiments due to time constraints.

We provide an explanation using the top-$k$ nearest prototypical examples from different classes and show the pixel evidence of similarity between the prototype and test sample. We first compare the quality of the pixelwise LRP heatmap explanation generated by the different student architectures with the teacher network and then show examples of explanation in the two forms (refer to Section III-D).

We assess the quality of the pixelwise LRP heatmap explanation for the network by performing a series of region perturbations in the input space of the test sample. The LRP heatmap comprises relevance scores associated with the degree of importance for the predicted class. Performing a perturbation on the image pixel associated with high relevance scores will destroy the network logit for the predicted class. In other words, we expect a sharp decrease in the logit score of the predicted class as the relevant features in the input sample are gradually removed in decreasing order of importance. We adopt the generalized region perturbation approach in [49]. We perform a series of perturbations starting from the most relevant region (based on the LRP relevance scores) to the least relevant region, as shown in Figure 5. A steep decrease suggests a good pixel explanation.

Based on the subplot for the LSUN dataset in Figure 5, the Head III-B student architecture with the best classification performance also has the best heatmap explanation, as shown by the largest decline in the logit scores. The attention modules in Head III-A and Head III-B show evidently better explanation qualities than the standard teacher network, as suggested by their respective curves. The Head I architecture shows a poorer explanation than the teacher network due to the early average pooling operation, which may have removed distinctive features that are useful in providing an informative pixel explanation. For the PCam dataset, the subplot suggests better heatmap quality for Head III-B architecture in the first 5 perturbation steps only and is overtaken by the teacher network and the Head II-B student network. Due to a lower diversity of objects in a patch (small cell types, stromal tissue and background) in the PCam dataset compared to the LSUN dataset, there may be a lesser need for attention weighting for explanation; thus, Head II-B shows better LRP explanation than its attention-augmented counterpart Head III-B when an increasing number of relevant regions are perturbed. Likewise, Head I architecture shows poor LRP explanation on the PCam dataset although it achieves the best classification performance in Table II. Generally, architectures with an attention mechanism provide better LRP explanation for datasets with large object diversity, while datasets with low object diversity need lesser attention weighting for explanation. In summary, Head II and Head III architectures produce heatmap quality comparable to the teacher network.

We show only LRP heatmaps from architecture with the best explanation (Head III-B) in Figures 6 and 7. We show pairs of heatmaps between the input sample and the top-1 nearest prototype (based on $u_{k}$ score) for a few selected classes. The positive and negative evidence in the heatmaps are represented by red and blue pixels, respectively. We obtain nonidentical input heatmaps (the columns in the top half of Figures 6 and 7) when comparing the same input sample with different prototype samples due to the different similarity outputs computed and thus unequal relevances for each input-prototype pair. In the left part of Figure 6, heatmaps for mostly all classes are dimmed out (the third row in the upper and lower part) after scaling due to their low similarity $u_{k}$ scores, except class $3$ (classroom), which is the predicted class. The red pixels are the similar regions between the pair of input-prototype that has contributed positively to the prediction classroom. The input heatmaps in the second row in columns 1, 4, and 5 look similar and they correspond to the similarity of the input image (classroom) to prototype examples belonging to the categories bedroom, classroom, and conference room, respectively. The categories bedroom, classroom, and conference room are all indoor scenes containing somewhat similar objects and indoor lighting. Since for each class we look at the nearest prototype to an image (rather than a randomly drawn prototype), we will likely find similar sceneries for these three indoor categories in a large dataset such as LSUN. Thus, the input heatmaps that are computed based on looking at the nearest prototype for each category, appear to reflect a plausible similarity among these three similar indoor scene examples.

In the lower right of Figure 6 showing the top-1 prototypes from different classes, the regions that resemble furniture in the dining room, such as chairs, tables, and stools, generally lit up in the prototype heatmaps for the prediction dining room. They mainly correlate to the input regions with a dining table, chair, and chandelier, as shown in the upper right of Figure 6.

For the explanation on the PCam dataset in Figure 7, we show explanations for a true negative, a true positive, and a false negative. The true negative prediction shows positive relevance scores (in red) on dense clusters of lymphocytes (dark regular nuclei), which is evidence supporting the absence of tumors. For the false negative prediction, the input, which is a positive sample (as indicated by the ground truth), resembles more of the negative class prototype (class 0) rather than the positive class prototype (class 1) in the red pixel areas. One usually looks at the nearest prototype to the test sample for explanation in addition to assessing the heatmap computed for the test sample. In the PCam case, the interpretation of the prototype with its LRP heatmap requires some domain knowledge from the person, such as a pathologist who is trained to recognize certain morphological structures.

In Figure 8, which depicts the false negative explanation extracted from Figure 7 for the input sample with respect to a negative class prototype, one can see in the upper left of the first row (displaying the input sample) a few faint likely cancer nuclei. The heatmap shown in Figure 8 shows very low heatmaps score in these cancer nuclei locations (almost black) and high heatmap scores in their direct surrounding embedding tissue matrix. Moreover, one sees high heatmap scores in some of the black dots, which are tissue invading lymphocytes (TiLs) also present in the prototype, and for which some but not all of them seem to be similar to the TiLs in the prototype (Figure 7, column 5, row 3). By looking at the prototype and its heatmap in the lower part of Figure 7, column 5, one can see that the similarities are found in a part of the tissue invading lymphocytes and a part of the embedding stromal tissue matrix.

The explanations in Figures 6 and 7 allow domain experts to easily validate the reasoning process of the neural networks by verifying that the nearest prototypical training samples are correct and the contributing features are relevant to the predicted class. By analyzing contradictory cases of prediction and explanation, domain experts understand the model limitation and are able to make an informed judgment. We have included more LRP explanation examples in Section S-III of the Supplementary Material.

The prototype similarity scores $u_{k}$, that are used to identify the top-$k$ nearest prototypical examples can also be employed to gauge the explanation confidence of a given input sample. The optimal value of $k$ that we choose to use for explanation depends on the task at hand. We can consider an explanation based on prototypical examples to be of sufficient confidence if the top-$k$ nearest prototypes (i.e., top-$k$ highest $u_{k}$ score) from the predicted class are responsible for a high fixed ratio (e.g., 90%) of the prediction mass; thus, these prototypes can be considered to have sufficiently high similarity scores. The top-$k$ nearest prototypes are considered to have insufficient similarity scores if their total score is below this fixed ratio.

Alternatively, one can also interpret the similarity scores in another manner. We consider the set of top-$k$ nearest prototypes from the predicted class to be too small if the removal of the prototype with the smallest similarity score would change the prediction label such that it differed from the prediction with the full set of prototypes. This is related to the principle of pertinent positives [50], but performed on the prototype level rather than the level of subsets of an input sample.