BayesNAM: Leveraging Inconsistency for Reliable Explanations

As numerous machine learning and deep learning models are black-box, a line of work has attempted to explain the decisions made by a black-box model. We call these methods post-hoc methods since they are applied after the model has been trained. While post-hoc methods offer some interpretability, recent work [3, 4] has argued that these methods can lead to unreliable explanations, which could potentially have detrimental effects on their explainability.

In contrast to post-hoc methods, intrinsic methods aim to develop an inherently explainable model without additional techniques [4]. Agarwal et al.[1] proposed a neural additive model (NAM), which combines a generalized additive model [5] and neural networks. To be specific, given $d$ features, $x_{1},x_{2},\cdots,x_{d}$ and a target $y$, NAM constructs $d$ mapping functions as follows:

where $\beta$ is a bias term and each mapping function $f_{i}$. In Fig. 2, We illustrate an example of NAM. By utilizing the neural network, NAMs capture the non-linear relationship and achieve high performance while maintaining clarity through a straightforward plot.

Despite their strengths, NAMs frequently exhibit inconsistent explanations even when trained on identical datasets with the same architectures, as illustrated in Fig. 1. These inconsistencies can also be observed in the original work [1], where the mapping functions produced by different NAMs within an ensemble show substantial variation, despite being trained under the same experimental conditions.

While this inconsistent phenomenon across NAMs can harm its explainability as they are intended to be XAI models, this phenomenon has received limited attention in the literature. To the best of our knowledge, only one study has explicitly addressed this issue. Radenovic et al. [2] introduced the neural basis model (NBM), which used shared basis functions across features rather than assigning independent mapping functions to each feature. They argued that NBM reduces divergence between models, offering more consistent shape functions compared to NAM, thus mitigating the inconsistency problem.

In contrast, this paper presents a novel view on the inconsistency phenomenon. Rather than treating it as a problem to be solved, we argue that these inconsistencies provide valuable information about the data model.

Although the use of a single model is a fundamental approach, numerous studies have found that a point-estimation is often vulnerable to overfitting and high variance due to its limited representation [6]. To overcome this limitation, Bayesian neural network estimates the model distribution instead of calculating a fixed model. Given the data $({\bm{x}},y)\sim\mathcal{D}$ and the prior $p({\bm{w}})$, we aim to approximate the posterior $p({\bm{w}}|{\bm{x}},y)$. Specifically, rather than using a fixed weight vector ${\bm{w}}_{i}$, it aims to find a distribution of weight vectors $\mathcal{N}(\bm{\mu}_{i},\texttt{diag}({\bm{s}}_{i})^{2})$ and learn the mean vector $\bm{\mu}_{i}$ and the standard deviation vector ${\bm{s}}_{i}$.

Since the distribution $p({\bm{x}},y)$ is generally intractable, several methods have been developed to approximate the posterior, including Markov Chain Monte Carlo (MCMC) [7] and variational inference approaches [8, 9]. While MCMC methods can provide more accurate estimates, their high computational cost [10] has led to the use of variational inference methods across diverse domains [11, 12].

During optimization in variational inference methods, an weight vector ${\bm{w}}_{i}=\bm{\mu}_{i}+{\bm{s}}_{i}\odot\bm{\epsilon}$ is sampled for each forward step where $\bm{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$. The prior distribution can be simply chosen as the isometric Gaussian prior $\mathcal{N}(\mathbf{0},s_{0}^{2}\mathbf{I})$ where $s_{0}$ is a predefined standard deviation to explicitly calculate the KL-divergence [11].

A promising direction in the field of Bayesian neural networks is their integration with other domains to enhance model explainability. Bayesian neural networks provide weight distributions that enable the identification of high-density regions or confidence intervals, which can be used for uncertainty estimation. Researchers and practitioners in several domains that require reliable explanations, such as medicine [13] and finance [14], have also explored the utilization of Bayesian models to measure the confidence of prediction for trustworthy decision-making.

We begin the analysis by identifying and investigating the inconsistent explanations of NAM. To this end, we construct a simple theoretical model. Here, we consider a binary classification task where the target $y$ can have a value in $\{-1,1\}$. Inspired by [15], we construct the input-target pairs $({\bm{x}},y)=(x_{1},x_{2},\cdots,x_{d},y)$ from a distribution $\mathcal{D}$ as follows:

where $x_{2},\cdots,x_{d}$ are independently and identically sampled from a normal distribution $\mathcal{N}$ with the mean $\lambda y$ and the standard deviation $\sigma^{2}$ for positive $\lambda$ and $\sigma$. It is important to note that the features $x_{2},\cdots,x_{d}$ are uncorrelated, as they are drawn independently and identically distributed. By adjusting the values of $p$ and $\lambda$, we can control the significance of $x_{1}$ and $x_{2},\cdots,x_{d}$ in predicting $y$, as stated in the following lemma:

Let $p$ be a sufficiently large positive number. When $\lambda=0$, only $x_{1}$ is useful to predict $y$ and other features $x_{2},\cdots,x_{d}$ are not correlated to $y$. As $\lambda$ increases, $x_{2},\cdots,x_{d}$ become correlated to $y$. By Lemma 1, if (5) is satisfied, a model that only considers $x_{2},\cdots,x_{d}$ can achieve a higher classification accuracy than $p$. In summary, if $\lambda=0$, $x_{1}$ would be the only feature with a high importance in predicting $y$, while $x_{2},\cdots,x_{d}$ is enough to have a significant performance in predicting $y$ for a large $\lambda>0$.

Now, we consider the following two cases with d=3:

• •

  Case-I. Single important feature exists $(\lambda=0)$. In this case, only $x_{1}$ is effective in predicting $y$, while $x_{2}$ and $x_{3}$ are not useful.

• •

  Case-II. Multiple important features exist $(\lambda=3)$. In this case, all the features, $x_{1}$, $x_{2}$, and $x_{3}$ are highly correlated with $y$. The model uses $x_{2}$ and $x_{3}$ can perform better than the model sorely depends on $x_{1}$ since $\Phi_{Z}(\lambda=3)=0.999$.

Given this theoretical model, we generated two sets of data containing 50,000 training examples and 10,000 test examples and trained two different NAMs on each dataset for different random seeds. For simplicity, we fixed the feature dimension to $d=3$, the probability $p=0.95$, and $\sigma^{2}=d-1$, resulting (5) becomes $\Phi_{Z}(\lambda)>p$, where $Z$ is drawn from the standard normal distribution $\mathcal{N}(0,1)$. For each mapping function $f_{i}$ of NAM, we constructed a simple neural network with two linear layers containing 10 hidden neurons and used ReLU as an activation function. The models are trained by SGD with a learning rate of 0.01. One epoch was sufficient to achieve high training accuracy.

Fig. 3 (Case-I) and Fig. 4 (Case-II) illustrate the mapping functions of trained NAMs for each case. Specifically, for Case-I, we observed that the two NAM models trained with different random seeds exhibited similar test accuracy and explanations (94.9% and 95.0%, respectively). As shown in Fig. 3, the mapping functions for each $x_{i}$ have similar shapes, with $f_{1}$ being the only increasing one and the others being almost constant. Therefore, in this case, NAM successfully captures the true importance of features and provides reliable explanations.

In contrast, for Case-II (Fig. 4), the mapping functions $f_{i}$ of the trained NAMs have extremely different shapes. Although both NAMs achieve a test accuracy exceeding 99.99%, $f_{3}$ is much steeper than $f_{2}$ for the first random seed, whereas the relationship is reversed for the second random seed, and vice versa.

Such inconsistency results in inconsistent feature contribution. Figure 5 shows the feature contribution of a sample ${\bm{x}}=[x_{1},x_{2},x_{3}]$ $=[-1,3,3]$. Following [1], we calculate the feature contribution by subtracting the average value of a mapping function across the entire training dataset. Although we use the same example, the feature contribution calculated by NAM with random seed 1 implies that $x_{2}$ appears to be more important than $x_{3}$, while NAM with random seed 2 outputs the opposite result that $x_{3}$ appears to be more significant than $x_{2}$. In summary, NAMs can produce inconsistent explanations when multiple important features are present, a common condition in real-world datasets. Indeed, as discussed later in Figures 10 and 12, this inconsistency is readily observable in widely-used datasets.

At first glance, the observed inconsistency may appear problematic; however, both explanations are not inherently incorrect. Specifically, given the theoretical model, both $x_{2}$ and $x_{3}$ are important features under Case-II, as using only one of them can achieve high performance. Therefore, the distinct mapping functions demonstrate that the different perspectives of trained models and each explanation is a valid interpretation of the data model, where relying solely on either $x_{2}$ or $x_{3}$ is sufficient for high performance.

In Fig. 6, we vary the learning rates ($\eta$) and batch sizes ($B$) during training within the same theoretical model. We linearly increase the learning rate $\eta$ from 0.005 to 0.01, and the batch size $B$ from 5 to 50. In total, we trained 50 models for each experiment, where each NAM exhibits inconsistent mapping functions. However, it is important to note that all models achieved over 99% test accuracy on the dataset. This indicates that the diverse explanations are not incorrect; rather, they offer valuable external insights into the existence of diverse perspectives among high-performing models, complementing the internal explanations of individual models. Therefore, we posit that inconsistency can be a useful indicator of potential external explanatory factors. Based on these findings, we propose a new framework to leverage inconsistency and provide additional explanations within the data model.

In the previous subsection, we explored the inconsistency phenomenon in NAMs and suggested that rather than being a flaw, this inconsistency can serve as a valuable source of additional information, shedding light on underlying external explanations in the data model. In this section, we introduce BayesNAM, a novel framework designed to leverage this inconsistency. BayesNAM incorporates two key approaches: (1) a modeling approach based on Bayesian structure and (2) an optimization approach utilizing feature dropout. Each of these approaches will be detailed in the following paragraphs.

1) Modeling Approach: Bayesian Structure for Inconsistency Exploration. A naive approach to exploring possible inconsistencies in NAMs is by training multiple independent models. Indeed, Agarwal et al.[1] trained several NAMs and visualized the learned shape functions $f_{k}(x_{k})$. However, this requires training $n$ independent models, leading to a computational burden proportional to $n$, making it impractical for large-scale applications.

To address this limitation, we propose using Bayesian neural networks [8, 9], which inherently allow for efficient exploration of model uncertainty without the need to train multiple independent models. Under variational inference and Bayes by Backprop [8, 9], Bayesian neural networks rather train the mean parameter $\bm{\mu}_{i}$ and the standard deviation parameter ${\bm{s}}_{i}$ instead of an weight vector ${\bm{w}}_{i}$. Then, during the training and inference phase, it samples a weight ${\bm{w}}_{i}=\bm{\mu}_{i}+{\bm{s}}_{i}\odot\bm{\epsilon}$ for a random vector $\bm{\epsilon}$ from a predefined distribution. Following prior works [11, 12], we adopt the reparameterization trick [16] for efficient training. This results in the following training objective.

where $\mathcal{L}(\cdot)$ is a given loss function and $\texttt{KL}(\cdot\lVert\cdot)$ is the KL-divergence. For further details, we refer the readers to [9].

In Fig. 7, we present the structural framework that integrates a Bayesian neural network with a neural additive model. For each sampled weight ${\bm{w}}^{(j)}$, we compute the corresponding predictions $f_{i}(x_{i}|{\bm{w}}^{(j)}_{i})$. This sampling approach enables the model to efficiently explore a diverse range of model spaces without needing to train multiple models. By incorporating a Bayesian neural network, the model provides high-density regions of the mapping functions and provides confidence intervals for feature contributions, offering richer interpretability.

2) Optimization Approach: Feature Dropout for Encouraging Diverse Explanations. Although Bayesian neural networks provide an efficient mechanism for exploration, they do not inherently guarantee exploring diverse explanations. Indeed, as shown in Fig. 8a, naive Bayesian neural network alone tends to focus on a single feature, similar to training a single NAM, rather than adequately exploring diverse explanations. As previously noted in related works [11, 12], we observe that increasing the standard deviation hyper-parameter $s_{0}$ within Bayesian neural network tends to degrade model performance and fails to address this issue effectively. Therefore, given the presence of diverse valid explanations shown in Figures 4 and 5, it is evident that a method is needed to encourage the exploration of diverse explanations.

As a potential solution, we propose the use of feature dropout during optimization. Feature dropout, initially introduced by Agarwal et al.[1], extends traditional dropout by selectively omitting individual feature networks during training. The hyperparameter $\tau$ determines the probability of dropping each feature. While the original work focused on improving model performance with feature dropout, we here provide a theoretical analysis showing that feature dropout implicitly encourages diverse explanations, preventing over-relying on any single feature.

Given the theoretical model in Section 3.1, we establish the following theorem.

Fig. 9 empirically verifies the general acceptance of Theorem 1. We plot $\Delta\mathcal{P}(k,\tau)$ with varying $k$. Other settings are same as the Case-II in Section 3.1. The increasing trend of $\Delta\mathcal{P}(k,\tau)$ for $\tau\in[0,\frac{1}{2}]$ aligns with the implications of Theorem 1. Furthermore, the acceptable range of $\tau$ in Theorem 1 expands as $k$ increases. When $k=100$, $\Delta\mathcal{P}(k,\tau)$ is increasing for $\tau\in[0,0.9]$. Even for $k=2$, we observe that $\Delta\mathcal{P}(k,\tau)$ increases until $\tau=0.4$. Since Theorem 1 holds regardless of the values of $\lambda$, we observe similar results for a very small value of $\lambda=0.01$.

In summary, we theoretically and empirically verify that feature dropout encourages the model to explore diverse explanations by using multiple features in the dataset. As shown in Fig. 8b, incorporating feature dropout enables the model to explore explanations across a range of features. Therefore, we introduce the framework that combines Bayesian neural networks with feature dropout as Bayesian Neural Additive Model (BayesNAM).

The capability of BayesNAM to explore diverse explanations further allows us to obtain confidence information of feature contributions. In the left plot of Figure 10, we plot the feature contributions of two randomly drawn offenders from each target value, ‘reoffended’ ($y=1$) or ‘not’ ($y=0$).

Among the features, juv_other_count (which represents the number of non-felony juvenile offenses a person has been convicted of) exhibits high variance in its contributions. This high variance indicates that, with a single NAM, the contribution of juv_other_count can appear either extremely negative or positive, potentially leading to misinterpretation. BayesNAM reveals substantial variability among models, suggesting that juv_other_count can have both positive and negative effects on predictions within certain ranges.

What can we infer from this high variation? In the right plot of Figure 10, we analyze the mapping function of juv_other_count. NAMs (gray) show inconsistent explanations for juv_other_count. The two-sigma interval of mapping functions of BayesNAM (orange) also starts to diverge significantly when juv_other_count $\geq 4$, indicating increased inconsistency in this range. As shown in Figure 11a, a data range where juv_other_count $\geq 4$ indicates a lack of sufficient data. Moreover, this range also shows skewed proportions, especially for juv_other_count $\geq 9$, where all labels are either ’reoffended’ or ’not.’ In summary, we verify that the high inconsistency highlights the need for caution when interpreting examples involving the feature, and suggests potential issues such as a lack of data.

In addition to identifying data insufficiencies, our model can also be used for feature selection. Features with high absolute contributions and small standard deviations, such as priors_count (which represents the total number of prior offenses a person has been convicted of), consistently demonstrate significant impact across different models.

In addition to data insufficiencies, a high level of inconsistency can reveal structural limitations within the model. In Fig. 12a, we plot the results of NAMs and BayesNAM for longitude. These functions show higher housing prices in San Francisco (around -122.5) and Los Angeles (around -118.5), consistent with previous findings [1]. However, BayesNAM finds that there exist inconsistent explanations between these two cities, particularly within the longitude range of -120 to -119.

We hypothesize that this inconsistency is due to significant variations in housing prices (target variable) across different latitudes. Fig. 12b illustrates the distribution of housing prices in California, with red circles indicating higher prices and larger circles representing higher volumes of houses. As shown in Fig. 12b, while Santa Barbara, Yosemite National Park, and Fresno are on similar longitudes, Santa Barbara exhibits a substantial price gap compared to the others. Additionally, since Yosemite National Park and Fresno are near the same latitude as San Francisco, NAM might struggle to accurately predict housing prices without the interaction term between Latitude and Longitude.

Based on this observation, we train a NAM with an interaction term between Latitude and Longitude. This model achieved a much better performance (RMSE: $0.506\pm 0.005$) than without the interaction term (RMSE: $0.556\pm 0.009$). In addition, The significance of the interaction term is particularly evident near Yosemite. When BayesNAM is trained with this interaction term, it also shows reduced variance in the longitude range of -120 to -119. This suggests that the high variance can highlight potential structural limitations within models. Moreover, considering that existing methods such as NA²M [1], which incorporate all possible interaction terms, incur heavy computational costs and diminished explainability, BayesNAM offers a promising alternative by effectively selecting the most important interaction terms.