Toward a Unified Framework for Debugging Concept-based Models

A central tenet of eXplainable AI (XAI) is that explanations of a model’s predictions naturally uncover bugs and biases affecting the model (Lapuschkin et al. 2019; Schramowski et al. 2020). Post-hoc explanations of black-box models, however, can be unfaithful and ambiguous (Dombrowski et al. 2019; Teso 2019; Lakkaraju and Bastani 2020; Sixt, Granz, and Landgraf 2020), and the extraction process can be computationally challenging (Van den Broeck et al. 2021).

Concept-based models (CBMs) are designed to make the extraction step as straightforward as possible while retaining the performance of more opaque alternatives (Rudin 2019). To this end, CBMs learn a set of high-level, interpretable concepts capturing task-salient properties of the inputs, and then obtain predictions by aggregating the concept activations in a (typically) understandable manner (Alvarez-Melis and Jaakkola 2018; Losch, Fritz, and Schiele 2019; Koh et al. 2020; Chen et al. 2019; Hase et al. 2019; Rymarczyk et al. 2020; Nauta, van Bree, and Seifert 2021; Lage and Doshi-Velez 2020). A key feature of CBMs is that they explain their own predictions by supplying faithful concept-level attribution maps, which encompass both the concepts and the aggregation weights, thus facilitating the identification of bugs affecting the model.

Work on interactive troubleshooting of CBMs is, however, sparse and ad hoc: some approaches assume the concepts to be given and high-quality and focus on correcting the aggregation step (Teso 2019; Stammer, Schramowski, and Kersting 2021), others address issues with the learned concepts while ignoring how they are aggregated (Barnett et al. 2021; Lage and Doshi-Velez 2020). Fixing only one set of bugs is however insufficient.

In this paper, we outline a unified framework for debugging CBMs. Our framework stems from the simple observation that the quality of a CBM hinges on both the concepts vocabulary and on how the concepts are aggregated, and that both elements are conveniently captured by the CBM’s explanations. This immediately suggests a human-in-the-loop debugging strategy based on providing supervision on the model’s explanations that is composed of three different phases, namely (i) evaluating concept quality, (ii) correcting the aggregation weights, and (iii) correcting the concepts themselves. This novel yet intuitive setup allows us to identify limitations in existing works and highlight possible ways to overcome them.

As a first step toward implementing this framework, we introduce a new loss function on the aggregation weights that generalizes approaches for aligning local explanations (Ross, Hughes, and Doshi-Velez 2017; Lertvittayakumjorn, Specia, and Toni 2020) to be robust to changes in the underlying concepts during training. We then outline how the same strategy can be applied recursively to align the concepts themselves by correcting both their predictions and explanations.

We are concerned with learning a high-quality classifier $f:\bm{\mathrm{x}}\mapsto y$ that maps instances $\bm{\mathrm{x}}\in\mathbb{R}^{d}$ into labels $y\in[v]:=\{1,\ldots,v\}$. In particular, we focus on concept-based models (CBMs) that fit the following two-level structure.

At the lower level, the model extracts a vector of concept activations $\bm{\mathrm{c}}(\bm{\mathrm{x}})=(c_{1}(\bm{\mathrm{x}}),\ldots,c_{k}(\bm{\mathrm{x}}))\in\mathbb{R}^{k}$ from the raw input $\bm{\mathrm{x}}$. The concepts $\{c_{j}\}$ are usually learned from data so to provide strong indication of specific classes (Chen et al. 2019). For instance, in order to discriminate between images of cars and plants, the model might learn concepts that identify wheels and leaves. The concepts themselves are completely black-box.

At the upper level, the model aggregates the concept activations into per-class scores, typically in a simulatable (Lipton 2018) manner. This is most often implemented by taking a linear combination of the activations:

where $\bm{\mathrm{w}}^{(y)}(\bm{\mathrm{x}})\in\mathbb{R}^{k}$ is the weight vector associated to class $y$. Class probabilities are then obtained by passing the scores through a softmax activation, that is, $P(y\ |\ \bm{\mathrm{x}}):=\mathrm{softmax}(\bm{\mathrm{s}}(\bm{\mathrm{x}}))_{y}$.

CBMs are learned by minimizing an empirical loss $1/|D|\cdot\sum_{i}\ell(f,(\bm{\mathrm{x}}_{i},y_{i}))$, where $i$ runs over the training set $D$, as customary. The loss function $\ell(f,(\bm{\mathrm{x}},y))$ itself combines a standard loss for classification, like the cross-entropy loss, together with additional regularization terms that encourage the learned concepts to be understandable to (sufficiently expert) human stakeholders. Commonly used criteria include similarity to concrete examples (Alvarez-Melis and Jaakkola 2018; Chen et al. 2019), disentanglement (Alvarez-Melis and Jaakkola 2018), and boundedness (Koh et al. 2020).

With CBMs, it is straightforward to extract local explanations that capture how different concepts contribute to a decision $(\bm{\mathrm{x}},y)$. These explanations take the form:

Notice that both the concepts and the aggregation weights are integral to the explanation: the concepts $\{c_{j}\}$ establish a vocabulary that enables communication with stakeholders, while the weights $\{w_{j}(\bm{\mathrm{x}})\}$ convey the relative importance of different concepts. Crucially, the score of class $y$ is independent from $\bm{\mathrm{x}}$ given the explanation $\mathcal{E}(\bm{\mathrm{x}},y)$, ensuring that the latter is faithful to the model’s decision process.

Existing literature on debugging CBMs can be split into two groups. Approaches in the first group are concerned with bugs in the aggregation weights $\{\bm{\mathrm{w}}^{(y)}(\bm{\mathrm{x}})\}_{y}$. These can occur when the data fools the model into assigning non-zero weight to concepts that correlate with – but are not causal for – the label. A prototypical example are class-specific watermarks in image classification (Lapuschkin et al. 2019). Teso (2019) addresses this issue by leveraging explanatory interactive learning (XIL) (Teso and Kersting 2019; Schramowski et al. 2020). As in active learning, in XIL the machine obtains labels by querying a human annotator, but in addition it presents predictions for its queries and local explanations for its predictions. The user then provides corrective feedback on the explanations, for instance by indicating those parts of an image that are irrelevant to the class label but that the machine relies on for its prediction. The model is then penalized whenever its weights do not align with the user’s corrections. This setup assumes that the concepts $\bm{\mathrm{c}}$ are fixed rather than learned from data. Stammer, Schramowski, and Kersting (2021) extend XIL to neuro-symbolic models and structured attention, but also assume the concepts to be fixed.

Conversely, approaches in the second group are only concerned with how the concepts are defined: concepts learned from data, even if discriminative and interpretable, may be misaligned with (the stakeholders’s understanding of) the prediction task and may thus fail to generalize properly. CBNMs work around this issue by leveraging concept-level label supervision (Koh et al. 2020), however this does not ensure that the concepts themselves are “right for the right reasons” (Ross, Hughes, and Doshi-Velez 2017). As a matter of fact, just like all other neural nets (Szegedy et al. 2013), part-prototypes learned by ProtoPNets can pick up and exploit uninterpretable features of the input (Hoffmann et al. 2021). Hoffmann et al. (2021) and Nauta et al. (2020) further argue that it may be difficult for users to understand what a learned prototype (concept) represents, unless the model explains why the concept activates. To this end, Nauta et al. (2020) propose a perturbation-based technique – analogous to LIME (Ribeiro, Singh, and Guestrin 2016) – for explaining why a particular prototype activates on a certain region of an image, which however does nothing to fix the bugs that it highlights. Finally, Barnett et al. (2021) introduce a loss term for ProtoPNets that penalizes concepts that activate on regions annotated as irrelevant by a domain expert.

We propose a new unified framework – based on the right for the right reasons (RRR) principle (Ross, Hughes, and Doshi-Velez 2017) – that, in contrast with existing solutions, is designed for the more realistic case of CBMs affected by multiple, different bugs.

The RRR principle is straightforward: the goal is to ensure that the model outputs accurate predictions that are justified by high-quality explanations. In order to make this concrete, we need to define what we mean for an explanation $\mathcal{E}(\bm{\mathrm{x}},y)$ to be “high-quality”. Intuitively, for CBMs explanation quality depends on the quality of the learned concepts $\bm{\mathrm{c}}$ and aggregation weights $\bm{\mathrm{w}}(\bm{\mathrm{x}})$. We unpack this intuition as follows:

Requirements C2, C3, and C4, on the other hand, are concerned with whether the concept set can be used for communicating with human stakeholders, and lie at the core of CBMs and human-in-the-loop debugging. Notice also that requirements C1–C4 are mutually independent: not all task-relevant concepts are understandable (indeed, this is often not the case), not all semantically meaningful concepts are easy to interpret, etc.

Simply discouraging the model from using incorrectly learned concepts does not prevent it from learning them again with minimal changes. To address this problem, we propose to penalize the model for associating non-zero weight to concepts that are similar to those concepts ${c}$ indicated as irrelevant to the target decision during the debugging session. We assume these concepts to have been collected in a memory $M$. The resulting loss is:

The sum iterates over old concepts ${c}$ irrelevant to $(\bm{\mathrm{x}},y)$, denoted $M(\bm{\mathrm{x}},y)$, and over the current concepts $\bm{\mathrm{c}}$, while $0\leq\kappa(c,c^{\prime})\leq 1$ measures the similarity between concepts.

Notice how Eq. 4, being independent of the concept’s order, prevents the model from re-learning forbidden concepts. Moreover, it automatically deals with redundant concepts, which are sometimes learned in practice (Chen et al. 2019), in which cases all copies of the same irrelevant concept are similarly penalized.

Our work targets well-known concept-based CBMs, including SENNs (Alvarez-Melis and Jaakkola 2018), CBMs (Koh et al. 2020; Losch, Fritz, and Schiele 2019), and ProtoPNets (Li et al. 2018; Chen et al. 2019; Hase et al. 2019; Rymarczyk et al. 2020; Barnett et al. 2021) and related approaches (Nauta, van Bree, and Seifert 2021), but it could be easily extended to similar models and techniques, for instance concept whitening (Chen, Bei, and Rudin 2020). The latter is reminiscent of CBMs, except that the neurons in the bottleneck layer are normalized and decorrelated and feedback on individual concepts is optional. Of course, not all CBMs fit our template. For instance, (Al-Shedivat, Dubey, and Xing 2020) propose a Bayesian model that, although definitely gray-box, does not admit selecting a unique explanation for a given prediction. More work is needed to capture these additional cases.

Our debugging approach is rooted in explanatory interactive learning, a set of techniques that acquire corrective from a user by displaying explanations (either local (Teso and Kersting 2019; Selvaraju et al. 2019; Lertvittayakumjorn, Specia, and Toni 2020; Schramowski et al. 2020; Shao et al. 2021) or global (Popordanoska, Kumar, and Teso 2020)) of the model’s beliefs. Recently, (Stammer, Schramowski, and Kersting 2021) designed a debugging approach for the specific case of attention-based neuro-symbolic models. These approaches assume the concept vocabulary to be given rather than learned and indeed are not robust to changes in the concepts, as illustrated above. Our aggregation loss generalizes these ideas to the case of concept-based CBMs. Other work on XIL has focused on example-based explanations (Teso et al. 2021; Zylberajch, Lertvittayakumjorn, and Toni 2021), which we did not include in our framework, but that could provide an alternative device for controlling a CBM’s reliance on, e.g., noisy examples.

A causal approach for debiasing CBMs has been proposed by (Bahadori and Heckerman 2021). This work is orthogonal to our contribution, and could be indeed integrated with it. The strategy of (Lage and Doshi-Velez 2020) acquires concept-based white box models that are better aligned with the user’s mental model by interactively acquiring concept-attribute dependency information. The machine does so by asking questions like “does depression depend on lorazepam?”, acquiring more impactful dependencies first. This approach is tailored for specific white-box models, but it could and should be extended to CBMs and integrated into our framework.

We proposed a unified framework for debugging concept-based CBMs that fixes bugs by acquiring corrective feedback from a human supervisor. Our key insight is that bugs can affect both how the concepts are defined and how they are aggregated, and that both elements have to be high-quality for a CBM to be effective. We proposed a three-step procedure to achieve this, shown how existing attribution losses are unsuitable for CBMs and proposed a new loss that is robust to changes in the learned concepts, and illustrated how the same schema can be used to correct the concepts themselves by interacting with their labels and explanations. A thorough empirical validation of these ideas is currently underway.

We are grateful to the anonymous reviewers for helping us to improve the manuscript. This research has received funding from the European Union’s Horizon 2020 FET Proactive project “WeNet - The Internet of us”, grant agreement No. 823783, and from the “DELPhi - DiscovEring Life Patterns” project funded by the MIUR Progetti di Ricerca di Rilevante Interesse Nazionale (PRIN) 2017 – DD n. 1062 del 31.05.2019. The research of ST and AP was partially supported by TAILOR, a project funded by EU Horizon 2020 research and innovation programme under GA No 952215.