Multiple Testing Framework for Out-of-Distribution Detection

Given the ubiquitous use of ML models in safety-critical applications such as self-driving and medicine, there is a need to develop methods to detect whether an ML model’s output at inference time can be trusted. This problem is commonly referred to as the Out-of-Distribution (OOD) detection problem. If an output is deemed untrustworthy by an OOD detector, one can abstain from making decisions based on the output, and default to a safe action. There has been a flurry of works on this problem in recent years. A particular area of focus has been on OOD detection for deep learning models ([1, 2, 3, 4, 5, 6]). While neural networks generalize quite well to inputs from the same distribution as the training distribution, recent works have shown that they tend to make incorrect predictions with high confidence, even for unrecognizable or irrelevant inputs (see, e.g., [7, 8, 9]).

In many of the prior works on OOD detection, OOD inputs are considered to be inputs that are not generated from the input training distribution (see, e.g., [2, 3]), which better describes the classical problem of outlier detection. However, in contrast to outlier detection, the goal in OOD detection is to flag untrustworthy outputs from a given ML model. Thus, it is essential for the definition of an OOD sample to involve the ML model. One of the contributions of this paper is a formal definition for the notion of OOD that involves both the input distribution and the ML model.

In a line of work in OOD detection, it is assumed that the detector has access (exposure) to OOD examples, which can be used to train an auxiliary classifier, or to tune hyperparameters for the detection model ([1, 10, 2, 11]). Other works rely on identifying certain patterns observed in the training data distribution, and use these patterns to train the original ML model to help detect OOD examples. For instance, in [6], a neural network is trained to leverage in-distribution equivariance propoerties for OOD detection. There is another line of work in which tests are designed based on statistics from generative models trained for OOD detection. For instance, in [12], statistics from a deep generative model are combined through p-values using the Fisher test. In this paper, we focus exclusively on developing methods that do not use any OOD samples, and can be applied to any pre-trained ML model.

Prior work has primarily been focused on identifying promising test statistics and corresponding thresholds, sometimes motivated by empirical observations of the values taken by these statistics for certain in-distribution and OOD inputs. For instance, in [1], a confidence score is constructed through a weighted sum of Mahalanobis distances across layers, using the class conditional Gaussian distributions of the features of the neural network under Gaussian discriminant analysis. In [2], a statistic based on input perturbations and temperature-scaled softmax scores is proposed. In [5], a free energy score based on the denominator of the temperature-scaled softmax score is proposed. In [3], scores are derived from Gram matrices, through the sum of deviations of the Gram matrix values from their respective range observed over the training data. In [13], the broad goal is to find all candidate functions from a given collection in an offline manner through multiple testing, such that any one of these candidate functions controls some risk at inference time. This approach is applied in [13] to the problem of OOD detection to select suitable thresholds for a given test statistic to control the false alarm rate. In [4], vector norms of gradients from a pre-trained network are used to form a test statistic. In [14], OOD detection in Convolutional Neural Networks (CNNs) is studied; spatial and channel reduction techniques are employed to produce statistics per layer, and these layer statistics are combined to form a final score using a method motivated by the tests proposed by [15] and [16]. Thus, their proposed algorithm computes a single score using all the intermediate features of the CNN and its corresponding empirical p-value. They provide marginal false alarm guarantees averaged over all possible validation datasets used to compute the empirical p-value. Additionally, the proposed method in [14] can be applied only to CNNs, and not any general ML model. To summarize, from prior work, it is unclear which among these scores/statistics is the best for OOD detection, or if there exists such a test statistic that is useful for all possible out-distributions. The latter question was raised in [17], where they posit that one can construct an out-distribution for any single score or statistic that results in poor detection performance.

The false alarm probability or type-I error of a test refers to the probability of a single in-distribution sample being misclassified as OOD, and the detection power refers to the probability of correctly identifying an OOD distribution sample. Note that the detection power is also referred to as detection accuracy in prior OOD works. In much of the prior work on OOD detection, the false alarm probability is estimated using empirical evaluations on certain in-distribution datasets. What is lacking in such works is a rigorous theoretical analysis of the probability of false alarm, which can be used to meet pre-specified false alarm constraints. Such false alarm guarantees are crucial for the responsible deployment of OOD methods in practice. Note that it is not possible to give any theoretical guarantees on the detection power of an OOD detection test without prior information about the class of all possible out-distributions, which is typically not available in practice. Therefore, in prior work on OOD detection, the detection powers of candidate OOD methods that meet the same pre-specified false alarm levels are compared empirically.

In this work, we propose a method inspired by multiple hypothesis testing ([18, 19, 20]) to systematically combine multiple test statistics for OOD detection. Our method works for combining any number of statistics with an arbitrary dependence structure, for instance the Mahalanobis distances ([1]) and the Gram matrix deviations across layers ([3]) of a neural network. We should emphasize there is no obvious way to directly combine such disparate statistics with provable guarantees for OOD detection. Detection procedures for multiple hypothesis testing are usually based on combining p-values across hypotheses [18, 19, 20]. However, in the problem of OOD detection, the probability measures under both the in-distribution (null) and out-of-distribution (alternate) settings are unknown, and thus the actual p-values cannot be computed. In conformal inference methods ([21, 22]) the p-values are replaced with conformal p-values, which are estimates computed from the empirical CDF of the test statistics. These conformal p-values are data-dependent, as they are calculated from in-distribution samples. In the procedure proposed in this paper, we use conformal p-values and provide rigorous theoretical guarantees on the probability of false alarm, conditioned on the dataset used for computing the conformal p-values.

Consider a learning problem with $(X,Y)\sim\mathrm{P}_{X,Y}$, where $(X,Y)$ is the input-output pair and $\mathrm{P}_{X,Y}$ is the distribution of the dataset available at training time. Let the dataset available at training time be denoted by $\mathcal{T}=\{(X_{1},Y_{1}),(X_{2},Y_{2}),...,(X_{n},Y_{n})\}$, where $n$ is the size of the dataset. Let the ML model be denoted by $f(\mathbf{W},.)$, where $\mathbf{W}$ is the random variable denoting the parameters of the ML model. For instance, $\mathbf{W}$ depicts the weights and biases in a neural network. Let $(X_{\mathrm{test}},Y_{\mathrm{test}})$ be a random variable generated from an unknown distribution, and $(x_{\mathrm{test}},y_{\mathrm{test}})$ be an instance of this random variable seen by the ML model at inference time. Given $\mathcal{T}$ and the ML model, the goal is to detect if this new unseen sample might produce an untrustworthy output. This might happen because either the input does not conform to the training data distribution, or if the ML model is unable to capture the true relationship between the input $X_{\mathrm{test}}$ and the true label $Y_{\mathrm{test}}$. Whether a new unseen sample is OOD or not depends on both the ML model and the distribution $\mathrm{P}_{X,Y}$.

A precise mathematical definition of the OOD detection problem that captures both the input distribution and the ML model appears to be lacking in prior work. The most common definition is based on testing between the following hypotheses (see, e.g., [2]):

where $\mathrm{H}_{0}$ corresponds to ‘in-distribution’ and $\mathrm{H}_{1}$ corresponds to ‘out-of-distribution’. However, such a definition does not involve the ML model, and better describes the problem of outlier detection, which is fundamentally different from the problem of OOD detection.

Let $\hat{Y}=f(\mathbf{W},X)$, and consider the distribution $\mathrm{P}_{X,\hat{Y}}=\mathrm{P}_{X}\times\mathrm{P}_{\hat{Y}|X}$ as the joint distribution of the input and the output of the ML model. Using this joint distribution as the ‘in-distribution’, consider the following testing problem:

Note that this is a definition of OOD detection that involves both the input distribution and the ML model (through $\mathrm{P}_{\hat{Y}|X}=\mathrm{P}_{f(\mathbf{W},X=x)|X=x}$). It also captures both the cases where the input is not drawn from $\mathrm{P}_{X}$, and when the ML model is unable to capture the relationship between the unseen input and its label.

The hypothesis test in (2) involves the true label ${Y}_{\mathrm{test}}$ and the distribution $\mathrm{P}_{X,\hat{Y}}$. Since these quantities are unknown, the model prediction $\hat{Y}_{\mathrm{test}}$ and the empirical distribution of $(X,\hat{Y})$ based on the training data, respectively, may be used instead. When the ML model performs well during training, i.e., $Y=\hat{Y}$ for almost all training data points, the empirical versions of $\mathrm{P}_{X,\hat{Y}}$ and $\mathrm{P}_{X,Y}$ agree, and we again arrive at a definition that does not involve the ML model. Thus, we conclude that it is necessary to use other functions of the input derived from the ML model¹¹1In this paper, we use the term statistic or score interchangeably to denote these functions of the input derived from the ML model. in addition to just the final output in constructing test statistics for effective OOD detection. Such a strategy is commonly employed, without theoretical justification, in many OOD detection works, for instance, through the use of intermediate features of a neural network to calculate the Mahalanobis score ([1]) and gram matrix score ([3]), and gradient information to calculate the GradNorm score ([4]). The discussion above provides a qualitative theoretical justification for these strategies developed in prior works.

In this section, we describe our proposed framework formally, and present our algorithm to combine any number of different functions of the input with an arbitrary dependence structure.

In our formulation of OOD detection in (2), we posit that, in addition to the input and the output from the ML model, it is necessary to use other functions²²2Without loss of generality, we may assume that these functions are scalar-valued. of the input, which are dependent on the ML model. We refer to these functions as score functions, denoted by $s^{1}(.),\ldots,s^{K}(.)$. The outputs of the score functions are scalar-valued scores $T^{1},T^{2},\ldots,T^{K}$:

The score functions are assumed to be chosen based on prior information in such a way that the scores are likely to take on small values for in-distribution inputs and larger values for OOD inputs. For a new input $X_{\mathrm{test}}$, let $(T^{1}_{\mathrm{test}},T^{2}_{\mathrm{test}},\ldots,T^{K}_{\mathrm{test}})$ be the corresponding scores. Note that one of scores $T^{k}_{\mathrm{test}}$ could be based on the final output from the learning model $\hat{Y}_{\mathrm{test}}$.

In the previous section, we have provided guarantees on the strong probability of false alarm for Algorithm 1. However, since it is not possible to theoretically analyze the power of such a test (due to the structure of the alternate hypothesis), we evaluate the power of our proposed approach through experiments. In addition, since we do not know beforehand what kind of OOD samples might arise at inference time, an effective OOD detection test must also have low variance across different OOD datasets, for a given Deep Neural Network (DNN) architecture. In our experiments, we evaluate both of these metrics to demonstrate the effectiveness of our approach.

Following the standard protocol for OOD detection ([1, 3, 5]), we consider settings with CIFAR10 and SVHN as the in-distribution datasets.

• •

  For CIFAR10 as the in-distribution dataset, we study SVHN, LSUN, ImageNet, and iSUN as OOD datasets.

• •

  For SVHN as the in-distribution dataset, we study LSUN, ImageNet, CIFAR10 and iSUN as OOD datasets.

We evaluate the performance on two pre-trained architectures: ResNet34 ([26]) and DenseNet ([27]). The calibration dataset in each case is a subset of 5000 samples of the in-distribution training dataset.

We evaluate the proposed approach, and compare it with SOTA methods based on the standard metric of probability of detection $\mathrm{P_{D}}$ or power (i.e., probability of correctly detecting an OOD sample) at probability of false alarm $\mathrm{P_{F}}$ at 0.1. Note that in some prior on OOD detection, the probability of detection is referred to as the True Negative Rate (TNR) and $1-\mathrm{P_{F}}$ as the True Positive Rate (TPR), where the in-distribution samples are considered positives, and OOD samples are considered negatives.

Recall that we focus exclusively on methods that do not have any outlier exposure to OOD samples, such as those in [10, 2, 1], and can be applied to any pre-trained ML model. We compare our approach against baselines: Mahalanobis ([1]), Gram matrix ([3]), and Energy ([5]). For the Mahalanobis baseline, we use the scores from the penultimate layer of the network to maintain uniformity.

To evaluate our proposed method, we systematically combine the following test statistics using our multiple testing approach as detailed in Algorithm 1:

1. 1.

   Mahalanobis distances from individual DNN layers ([1]): Let $g_{i}(X),i=1,\ldots,L$ denote the outputs of the intermediate layers of the neural network for an input $X$. We estimate $\mu^{c}_{i}$, the class-wise mean of $g_{i}(.)$, as the empirical class-wise mean from the training dataset:

   $$\mu^{c}_{i}=\frac{1}{n_{c}}\sum_{j:Y_{j}=c}g_{i}(X_{j}),$$

   (26)

   where $n_{c}$ is the number of points with label $c$. We estimate the common covariance $\Sigma$ for all classes as

   $$\Sigma=\frac{1}{n_{c}}\sum_{c}\sum_{j:Y_{j}=c}\left(g_{i}(X_{j})-\mu_{c}\right)\left(g_{i}(X_{j})-\mu_{c}\right)^{T}.$$

   (27)

   This is equivalent to fitting the class- conditional Gaussian distributions with a tied covariance. The Mahalanobis score for layer $i$ is calculated as:

   $$\max_{c}-\left(g_{i}(X_{j})-\mu_{c}\right)\Sigma^{-1}\left(g_{i}(X_{j})-\mu_{c}\right)^{T}.$$

   (28)

   We calculate 5 Mahalanobis scores from the intermediate layers for the ResNet34 architecture, and 4 scores for the DenseNet architecture.

2. 2.

   Gram matrix deviations from the individual DNN layers ([3]): For each intermediate layer $i$, the Gram matrix of order $p$ is calculated as:

   $$M^{p}_{i}(x)=\left(g^{p}_{i}{g^{p}_{i}}^{T}\right)^{\frac{1}{p}},$$

   (29)

   where the power is calculated element-wise. For each flattened upper triangular Gram matrix $\overline{M^{p}_{i}}$, there are $n_{i}$ correlations. The class-specific minimum and maximum values for the correlation $j$ (i.e., $j$-th element of $\overline{M^{p}_{i}}$), class $c$, layer $i$ and power $p$ are estimated from the training dataset as $\text{min}[c][i][p][j]$ and $\text{max}[c][i][p][j]$, respectively. For a new input $X$, the deviation for correlation $j$, layer $i$, power $p$ is calculated with respect to the predicted class $c_{X}$ as

   $$\delta_{X}(i,p,j)=\begin{cases}\frac{\overline{M^{p}_{i}(X)}[j]-\text{min}[c_{X}][i][p][j]}{|\text{min}[c_{X}][i][p][j]|}\;\;\;\text{if }\overline{M^{p}_{i}(X)}[j]>\text{min}[c_{X}][i][p][j]\\ \vskip 5.0pt\cr\frac{\text{max}[c_{X}][i][p][j]-\overline{M^{p}_{i}(X)}[j]}{|\text{max}[c_{X}][i][p][j]|}\;\;\;\text{if }\overline{M^{p}_{i}(X)}[j]<\text{max}[c_{X}][i][p][j]\\ \vskip 5.0pt\cr 0\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\text{otherwise}.\end{cases}$$

   (30)

   As proposed in [3], the Gram matrix score for layer $i$ is then calculated as the sum of $\delta_{X}(i,p,j)$ over values of $p$ from 1 to 10, and all values of $j$, and normalized by the empirical mean of $\delta_{X}(i,p,j)$. We calculate 5 Gram scores from the intermediate layers for the ResNet34 architecture, and 4 scores for the DenseNet architecture.

3. 3.

   Energy statistic ([5]): The energy score is a temperature scaled log-sum-exponent of the softmax scores

   $$-T\log\sum_{i=1}^{C}e^{\sigma_{i}(X)/T},$$

   (31)

   where $C$ is the number of classes, $\sigma_{i}(.)$ are the softmax scores, and $T$ is the temperature parameter. In our experiments, we set the temperature $T$ to 100 for all in-distribution datasets, DNN architectures and OOD datasets (as stated in [5], the energy score is not sensitive to the temperature parameter).

We use a subset of 45000 points from the training dataset (with no overlap with the calibration dataset) to calculate the class-wise empirical means and covariance for the Mahalanobis scores, and the minimum and maximum correlations for the Gram scores.

For CIFAR10 and SVHN as the in-distribution datasets, we use our proposed method in Algorithm 1 to combine the Mahalanobis scores and Gram scores across layers, and the energy score, to detect OOD samples. There are 11 scores (i.e., $K=11$) in total for the ResNet34 architecture, and 9 scores (i.e., $K=9$) for the DenseNet architecture. Recall that Algorithm 1 declares an input to be an OOD sample if any of the $K$ null hypotheses corresponding to the $K$ scores are rejected. For different OOD datasets, we empirically study the probability of each null hypotheses being rejected by Algorithm 1. In Figure 3, we plot the empirical probability of each score $i$ being rejected, i.e., the proportion of data points in each OOD dataset for which the corresponding null hypothesis $H_{0,i}$ was rejected. The Mahalanobis score and Gram score of layer $i$ are denoted by ‘Mahala i’ and ‘Gram i’ respectively, and the energy score is denoted by ‘Energy’. We observe that while the probability of a score being rejected is high for certain OOD datsets, there exists OOD instances for which it is quite low. For example, in the Resnet34 architecture with CIFAR10 as the in-distribution dataset, while the Mahalanobis scores of layers 2, 3 and 5, and the Gram scores of layer 5 are useful to detect OOD instances from the LSUN, Imagenet and iSUN datasets, they are not likely to be useful in detecting OOD instances from the SVHN dataset. On the other hand, the Mahalanobis and Gram scores from layer 4 of the network are more useful in detecting OOD instances from the SVHN dataset than LSUN, Imagenet and iSUN datasets. This study provides evidence that any single score may not be useful to detect all kinds of OOD instances that an ML model might encounter at inference time, and combining different scores systematically, as proposed in Algorithm 1, might lead to a more robust OOD detection method. We demonstrate an improvement in detection performance and the robustness of our proposed OOD detection method through extensive experiments presented further in this section.

The detection power performances for CIFAR10 and SVHN as in-distribution datasets are presented in Tables I and II, for the Mahalanobis, Gram and Energy baselines, and our proposed method of combining different statistics. We annotate our method with the number of statistics used, e.g., Mahalanobis, Gram and Energy (5/4+5/4+1) uses 5,4 layers in ResNet34, DenseNet architectures respectively, for both Mahalanobis and Gram, and the energy score. For each in-distribution dataset, we consider 8 cases, comprising of 4 OOD Datasets and 2 different DNN architectures.

1. 1.

   Improvement in probability of detection across OOD datasets and DNN architectures: The best probability of detection in all 8 cases with CIFAR10 as in-distribution correspond to our method of combining statistics. Similarly, with SVHN as in-distribution, our method of combining statistics gives the best probability of detection in all 8 cases.

   Thus, our approach leads to an improvement across OOD datasets and DNN architectures.

2. 2.

   Lower variation in detection probability across OOD datasets and DNN architectures: Detection probabilities of baselines Mahalanobis, Gram and Energy exhibit a much higher variation across different kinds of OOD samples as compared to the combination of all statistics.

   With CIFAR10 as the in-distribution dataset, for the ResNet34 architecture: the variation in $\mathrm{P_{D}}$ is $82.77-90.97$ for the Mahalanobis baseline, $92.34-96.04$ for the Gram baseline, and $73.21-81.16$ for the energy baseline. In contrast, our method of combining all statistics has a variation of $97.03-98.00$. For DenseNet, the variation in $\mathrm{P_{D}}$ is $82.81-92.98$ for the Mahalanobis baseline, $80.04-89.97$ for the Gram baseline, and $42.40-96.89$ for the energy baseline. Our method of combining all statistics has a variation of $94.57-97.78$.

   A similar trend is seen with SVHN as the in-distribution dataset. Our method reduces the variation across different kinds of OOD samples by almost 5X. This is a key improvement, as the kind of OOD samples encountered at inference time is unknown, and our proposed method shows very little variation across different OOD datasets.

3. 3.

   Impact of combining all the scores: For CIFAR10 as the in-distribution dataset, in all 8 cases, combining all the scores - Mahalanobis and gram from individual layers, and the energy score, is either the best method, or within $1\%$ of the best performance.

   Similarly, with SVHN as the in-distribution dataset, in 7 out of 8 cases, combining all the scores is either the best method, or within $1\%$ of the best performance (the gap is $2.41\%$ in the remaining case).

   Thus, in contrast to existing methods, combining all the statistics using Algorithm 1 is robust to different kinds of OOD samples across DNN architectures.

We further compare our proposed method of combining multiple scores in Algorithm 1 with a baseline that combines scores naively through an averaging rule. This naive OOD detection test maintains thresholds $\tau_{1},\ldots,\tau_{K}$ for the $K$ scores. Let $\gamma_{i}$ be the weight for the $i$-th score where

and let $\gamma$ be defined as

The naive averaging OOD detection rule declares an input to be an OOD sample if $\gamma\geq\frac{1}{2}$. The thresholds $\tau_{1},\ldots,\tau_{K}$ are set to ensure a false alarm probability of $0.1$. In Table III, we present a comparison of the detection powers of the naive averaging rule and our proposed method of combining scores, where both methods use the Mahalanobis and Gram scores from all the layers, and the energy score. We observe that the naive averaging method does not perform as well as our proposed method of combining statistics, and indeed has a high variability across different OOD datasets. Thus, we see that while it is imperative to combine multiple scores for effective and robust OOD detection, combining them in an adhoc manner such as uniform averaging does not yield good results.

In some of the prior work on OOD detection, the Area Under the Receiver Operating Characteristic (AUROC) metric has been used to compare different tests ([2, 5, 3, 1]). However, it is not clear that this measure is useful in such a comparison, especially when the ROC is being estimated through simulations. It is possible for a test (say, Test 1) to have a larger AUROC than another test (say, Test 2), with Test 2 having a larger detection power than Test 1 for all values of false alarm less than some threshold (equivalently, all values of TPR greater than some threshold). Nevertheless, we provide the AUROC numbers for our experimental setups below for completeness.

Table IV contains the AUROC numbers for CIFAR10 as the in-distribution dataset, and Table V contains the AUROC numbers for SVHN as the in-distribution dataset. We observe similar patterns in the AUROC numbers as the above observations on the detection power at a fixed false alarm probability. The Mahalanobis, Gram and Energy baselines have a high variability across different kinds of OOD samples and DNN architectures, whereas our proposed method of combining all statistics has a low variability. For instance, the Mahalanobis, Gram and Energy baselines for DenseNet architecture with CIFAR10 as the in-distribution dataset have an average variability of 10.5$\%$ in their AUROC, whereas our proposed method of combining all statistics has a variability of $0.5\%$ in the AUROC performance. Our proposed method of combining all statistics either has the best AUROC performance or within $1\%$ of the best performance in all 8 cases for CIFAR10 and SVHN as the in-distribution datasets.

While empirical methods for OOD detection have been studied extensively in recent literature, a formal characterization of OOD is lacking. We proposed a characterization for the notion of OOD that includes both the input distribution and the ML model. This provided insights for the construction of effective OOD detection tests. Our approach, inspired by multiple hypothesis testing, allows us to systematically combine any number of different statistics derived from the ML model with an arbitrary dependence structure.

Furthermore, our analysis allows us to set the test thresholds to meet given constraints on the probability of incorrectly classifying an in-distribution sample as OOD (false alarm probability). We provide strong theoretical guarantees on the probability of false alarm in OOD detection, conditioned on the dataset used for computing the conformal p-values.

In our experiments, we observe that no single score is useful for detecting different kinds of OOD instances. We demonstrated that our proposed method outperforms threshold-based tests for OOD detection proposed in prior work. Across different kinds of OOD examples, we observed that the state-of-the-art methods from prior work exhibit high variability across OOD instances and neural network architectures in their probability of detection of OOD samples. In contrast, our proposed method is robust and provides uniformly good performance (with respect to both detection power and AUROC) across different kinds of OOD samples and neural network architectures. This robustness is important, since a useful OOD detection algorithm should perform well regardless of the type of OOD instance encountered at inference time.