FOOD: Fast Out-Of-Distribution Detector

In a DNN classifier without OOD detection capability, there is an underlying assumption that a test sample $x$ is associated with one of the $C$ classes in the training dataset. By denoting the class that $x$ is associated with as $c(x)$, the classification can be expressed as:

OOD detection necessitates the option of classifying the input $x$ as none of the learned classes associated with the training dataset. In this case, the classification can be expressed as:

We propose a novel OOD detection method that meets the following requirements:

1. 1.

   Fast inference time - The method must be able to be applied in scenarios where inference time is limited. Therefore, no significant inference time overhead is permitted.

2. 2.

   In-distribution data only - The method must only use in-distribution data for training.

The multivariate Gaussian function is integrated in DNN methods and architectures in numerous ways [14, 15, 11]. For computational reasons, usually the log values of the likelihoods are calculated instead of the likelihoods’ values themselves. Given the parameters $\Sigma\in\mathcal{R}^{d\times d}$ and $\mu\in\mathcal{R}^{d}$, it is defined as in [16, chap. 3]:

A Gaussian layer was proposed to improve DNN classification accuracy [11] by employing the multivariate Gaussian distribution. Each of the Gaussian layer’s outputs corresponds to a density function with respect to the layer’s input. The Gaussian layer assumes a multivariate Gaussian distribution on its input, and each of its outputs match the log likelihood, as presented in Equation (3). In practice, it is based on a DNN architecture in which the classification layer (last fully connected layer) is replaced with a Guassian layer designed for classification. The Gaussian layer contains the trainable parameters that define the distribution of each class. For each class $c$, the layer assigns a score that corresponds to the log likelihood of the penultimate layer’s output, modeled as a Gaussian:

where $\mu_{c}$ and $\Sigma_{c}$ are the layer’s parameters for the $c^{th}$ class, and $f(x)$ is the penultimate layer’s output. Diagonal covariance matrices are used instead of full covariance matrices, resulting in significantly reduced computational complexity. To train the parameters of the modified DNN, the authors used softmax paired with cross-entropy loss ($\mathcal{L}_{ce}$) on the outputs of the Gaussian layer. In addition to $\mathcal{L}_{ce}$, a regularization term that maximizes the likelihood of the correct output is employed. Given a sample $\{x_{i},y_{i}\}_{i=1}^{m}$(where $m$ is the sample size), the regularization term is defined as:

The model optimization target $\mathcal{L}$ can be summarized as:

where $\lambda$ is a hyperparameter chosen using a validation set.

Max Softmax Probability (MSP) [17] - A baseline for OOD detection. It employs the predicted maximum class probability as a score to separate OOD samples from in-distribution samples.

Mahalanobis Distance (MD) [18] - Based on analysis of DNN hidden representations and input preprocessing. This method operates under the assumption that in each layer’s representation, the samples of each class are distributed as a Gaussian. Each Gaussian is parameterized by a center vector and a shared full covariance (all classes share one covariance matrix). The score for OOD detection is a weighted sum over the Mahalanobis distance calculated in each layer, whose weights are learned using OOD samples. The authors of [9] proposed a modification of this method, in which input preprocessing is not performed, and a simple sum is used instead of a weighted sum; we denote this method as MD*.

Outlier Exposer (OE) [7] - A detection method in which the DNN is fine-tuned using OOD samples that are not from the OOD test dataset. OE optimizes a loss function which includes an additional term, forcing the DNN to output equal predictions for all classes when encountering an OOD sample. On inference, the MSP score is used for OOD detection.

Generalized Odin (G.ODIN) [9] - A detection method built upon the Odin method [19], eliminating its need for OOD samples. Its architecture is comprised of temperature scaling and modification of the softmax function. During inference, G.ODIN uses input preprocessing to craft an adversarial sample from the input. The predicted maximum class probability of the output is used as the score for OOD detection.

Self-Supervised Learning for OOD detection (SSL) [6] - A detection method in which the last layer of the DNN is extended with several additional neurons that are trained using OOD samples (not from the OOD test dataset). The additional neurons are trained using an proprietary loss function, and their sum is used as a score for OOD detection.

Gram matrices (GM) [10] - A detection method in which the correlation between different features in each layer’s representation is extensively analyzed. Given a representation $F_{l}$ for layer $l$ and a maximal order $P$, the Gram matrix measure includes the calculation of the higher order Gram matrices: $(F_{l}^{p}F_{l}^{p^{T}})^{\frac{1}{p}}$, for $p\in\{1,\ldots,P\}$, where the power and root are element-wise operations. The deviation from the Gram matrix measure of the training data is used for OOD detection.

MALCOM [20] - A detection method that maps the internal representations to the string domain. In the string domain, a file compression technique is used to extract uncorrelated features from the DNN, which then serve as input to the most basic version of the MD method [18].

In Table I, we present a comparison of the various OOD methods with respect to our requirements, as specified in Section  II-A.

MSP [17] only relies on the DNN’s original output which was trained without any OOD data, so it meets the In-distribution data only requirement. The output is used directly, and therefore it complies with the Fast inference time requirement as well. MD [18] and G.ODIN [9] employ preprocessing to craft a perturbation of the input, which incurs significant inference time overhead, so they do not meet the Fast inference time requirement. MD also employs OOD samples to tune the OOD detection model, so it fails to comply with In-distribution data only. However, MD* does not require OOD samples or perform input preprocessing, meaning that this modified version of MD meets both of our requirements. The SSL [6] and OE [7] methods are trained on OOD samples (not from the OOD test data), meaning that they fail to meet the In-distribution data only requirement. The MALCOM [20] detection method is based on file compression techniques, which appear complicated but can be implemented efficiently, and since it does not rely on OOD samples, it complies with both of our requirements. The GM [10] method uses complex postprocessing calculations after the inference process has been performed. This postprocess increases inference time, as shown in the Results section (Figure 2); therefore, it does not comply with the Fast inference time requirement.

Adversarial samples are in-distribution samples that have been carefully perturbed to shift their DNN prediction from the correct class to an incorrect one [21]. There are different methods of crafting adversarial samples [22, 23, 24], each of which requires a different set of hyperparameters. Those hyperparameters guide the way in which the perturbation is crafted. The same techniques can be used to achieve the opposite effect (i.e., to strengthen the confidence of the DNN’s prediction in the correct class). In general, it is possible to perturb the sample towards any direction of a differentiable score calculated by the DNN. There are two main approaches in which OOD detection methods take advantage of adversarial methods:

In the first approach, it is assumed that adversarial perturbations have a different effect on OOD samples than on in-distribution samples. Based on this assumption, during inference, some OOD detection methods craft an adversarial sample out of the input sample in order to enhance OOD detection [18, 19, 9]. Each method crafts the perturbation according to its OOD detection score. In general, given a score $\mathcal{D}(\cdot)$, an input $x$, and a perturbation size $\epsilon$, the perturbed input $x_{per}$ is calculated as follows:

The perturbation is either added or subtracted from the original $x$. Finally, the value of $\mathcal{D}(x_{per})$ is used for OOD detection.

Another approach for using adversarial samples in OOD detection is to replace the OOD samples used in the training phase. In the paper proposing MD [18], the authors presented a version of MD that uses adversarial samples instead of OOD samples during the training phase. However, this MD version obtained suboptimal OOD detection results.

Because of the Fast inference time requirement, we developed a detection method that can be implemented efficiently as part of the DNN, without additional processing. In the typical DNN classifier architecture, the final layer usually consists of linear transformation (a fully connected layer) and a softmax activation. The softmax output can technically be employed as a density function, however studies have shown that the OOD detection capabilities of softmax are limited, and more sophisticated methods can surpass its OOD detection performance  [19, 17].

Instead of using the softmax output, in this research we focus on the internal representations of the DNN, particularly the penultimate representation. In prior research, representation analyses were performed to explain the relations in the geometric nature of DNN’s internal representations [25, 18]. Those analyses showed that the samples of each class are naturally grouped in the internal representations, creating a clustering effect. The deeper the layer is, the more obvious the clustering effect becomes, resulting in a complete separation between the classes in the penultimate representation. This finding indicates that the penultimate layer has the potential to serve as the input to a density function. We developed our architecture based on a method that employs a Gaussian layer as its final layer, instead of a fully connected one [11]. The Gaussian layer models each class ($c$) as a multivariate Gaussian, with its own trainable parameters - a positive semi-definite covariance matrix ($\Sigma_{c}\in\mathbb{R}^{d\times d}$) and a center vector ($\mu_{c}\in\mathbb{R}^{d}$). Since the likelihood term in the Gaussian layer (Equation 3) involves the calculation of $\log(|\Sigma_{c}|)$ and ${\Sigma^{-1}_{c}}$, diagonal covariance matrices are used to improve numerical stability. Moreover, it allows efficient paralleled computation of the Gaussian’s log likelihood. Note that in our implementation, the covariance matrix is learned during training, in contrast to some other methods that employ Gaussians [26].

For simplicity, and unlike the procedure proposed in the original paper that introduced the Gaussian layer [11], we do not train our DNN from scratch, but rather fine-tune an existing trained DNN model. We replace its last layer with a Gaussian one and then initialize the layer parameters manually, using their statistical estimations from the training data penultimate representation. The calculation is performed as follows:

where $S_{c}$ is the set which contains all of the samples from class $c$, $f(x)$ is the penultimate representation of $x$, and $\mu_{c}$ and $\Sigma_{c}$ are the layer parameters for class $c$. After the layer’s parameters are set, we fine-tune the model for 15 epochs using Equation (6) as the optimization target. We stop the fine-tuning process when the validation loss starts to increase.

The proposed DNN architecture outputs proper density values that correspond to the log likelihood of the classes’ Gaussians in the penultimate representation. Taking advantage of this property, we perform the $\mathcal{LLR}$ test on the Gaussians’ log likelihood values. We define two scores, the predicted class ($p_{pred}$) log likelihood and the log likelihood average over the other classes ($p_{other}$):

We define the $\mathcal{LLR}$ score as:

where $C$ is the number of classes, $x$ is the input sample, and $f(x)$ is the penultimate representation of $x$. We explain the $\mathcal{LLR}$ terms as follows: $p_{pred}$ is the log likelihood of a sample to be in the in-class distribution, and $p_{other}$ is the log likelihood of a sample to be out of the predicted class distribution. The $\mathcal{LLR}$ (on its own) can serve as a basic and efficient score for OOD detection (Table II) that does not require proprietary tuning. In-distribution samples have higher $\mathcal{LLR}$ values, while OOD samples have lower $\mathcal{LLR}$ values, allowing them to be differentiated from one another.

One of the challenges that arises when creating an OOD detector is tuning it. To address this challenge without additional data sources, we developed a novel method to generate artificial OOD samples that can be used to train an OOD detector.

We employ an adversarial crafting method to generate artificial OOD samples. Given the training set samples (in-distribtion data), one can observe that some training samples have higher $\mathcal{LLR}$ values than others (Figure 1). This implies that there are samples which are more likely to be associated with in-distribution samples than others. It is possible to use those values in order to estimate the boundaries of the in-distribution data. We estimate the boundary of the in-distribution data as the lower 95th percentile of the in-distribution data’s $\mathcal{LLR}$ values; we refer to it as $Thres$. We use an adversarial sample crafting method to alter the samples, such that their $\mathcal{LLR}$ values are lower than $Thres$, making those samples less likely to be associated with in-distribution data. We utilize the $LLR$ as the loss target when crafting the adversarial samples, iteratively altering the samples towards a lower $LLR$ value until $Thres$ is reached. The data used to perform this operation is a subset of the training data that was not used during training (a validation set); the algorithm is described in Algorithm 1.

$MaxIter$ was set at 10, because in most cases, the algorithm reaches its stopping criteria after 1-4 iterations. The $\epsilon$ was not tuned based on OOD samples and was set to $0.01$ in all of our evaluations. $Thres$ determines how large the perturbation will be for each sample, and it is calculated using only the in-distribution data. This eliminates the need to acquire OOD samples prior to tuning our detector. A sensitivity analysis is presented in Table III, showing that $\epsilon$ and $Thres$ are insensitive to changes.

The internal DNN representations, which are the outputs of each layer’s activation, are commonly used as input features for OOD detectors [18, 20, 10]. We were inspired by those studies and combined this approach with the maximum likelihood approach. We extract the likelihood of the most likely class in each of the internal representations of the DNN and use it as features. In order to perform the likelihood calculation in each representation, we assume that the samples of each class are distributed as a Gaussian. Under this assumption, we empirically estimate the parameters for the Gaussians (mean vector and diagonal covariance matrix) using the training data, as in Equation (3). To reduce the dimensionality, we apply a pooling function on the representations, instead of using the representations themselves. Unlike other OOD detection methods that only use average pooling over all representations, we use different pooling operations for different representations. Given a dataset, a sample $x$, and a DNN classifier of $C$ classes with $L+1$ hidden layers, for layers’ representations $l\in\{1,\ldots,\frac{L}{2}\}$, we perform max pooling, and for $l\in\{\frac{L}{2}+1,\ldots,L\}$, we use average pooling. This modification was made because the clustering effect is more prominent in the deeper representations [25], and thus we assume that less noise reduction is required to extract useful information from them (i.e., average pooling can be used instead of max pooling). Note that the feature extraction does not include the penultimate layer. Our feature extraction algorithm does the following

where $F(x,l)$ is the feature extracted from layer $l$, i.e., the maximum log likelihood (Equation 3) estimated for a sample $x$ to belong to a certain class in the representation of layer $l$, using the diagonal covariance matrix $\Sigma^{[l]}_{c}$ and the mean vector $\mu^{[l]}_{c}$. $f^{[l]}(\cdot)\in\mathbb{R}^{d}$ is the output of the $l^{th}$ layer, $l\in\{1,\ldots,L\}$.

The artificial OOD samples generated  (1) allow us to use a supervised learning classifier that differentiates between OOD samples and in-distribution samples. We train our classifier with artificial OOD samples labeled as OOD and a subset of the training data which was not used for the DNN training, which is labeled as in-distribution (the same samples that were used for the artificial OOD samples generation). To achieve minimal inference time overhead, we employ a logistic regression classifier, implemented as an output neuron in the DNN, as our OOD detector. Given a sample $x$, the input to the detector is the internal representation features, as presented in Equation (13), $\{F(x,l)\}_{l=1}^{L}$, and the $\mathcal{LLR}$ presented in Equation (12). The score function of the logistic regression classifier is:

where $\sigma$ is the sigmoid function, and the $w_{l}$ are the weights of the logistic regression classifier.

Figure 2, which provides a comparison of the inference times of state-of-the-art OOD detection methods, shows that there is a large variation between the methods. Unsurprisingly, methods that rely on MSP had the fastest inference time, as the only calculation required for them is the feedforward operation. FOOD’s inference time is similar to those methods and is also the fastest of all of the methods that do not require OOD samples for training. MD* is a little slower; this can be explained by the use of full covariance matrices in this method, as opposed to FOOD which uses diagonal covariance matrices. We observe that methods which require adversarial preprocessing (e.g., G.ODIN, MD) are more than four times slower than FOOD, as they require backpropagation in order to edit the sample before inference. GM requires complicated postprocessing, causing its run to take the longest of the methods examined.

Figure 3 shows the trade-off between OOD detection performance and inference time for different methods, evaluating the TNR95 metric against the speed. The figure shows that methods that are both fast and accurate tend to appear closer to the top right corner, whereas methods with suboptimal detection performance or a long inference time appear respectively more to the left and closer to the $Speed$ axis. As seen in the plot, FOOD’s inference time is faster than almost all the other methods, while maintaining high OOD detection capabilities. Methods that obtain higher TNR95 scores run significantly slower, and in the case of MD, use OOD samples for tuning. In addition, it is easy to see that methods which have a faster inference time than FOOD (i.e., OE, and SSL) have significantly lower OOD detection performance.

Table IV presents a comprehensive OOD detection performance comparison of methods that meet the Fast Inference Time requirement (Section II-A). As seen in the table, FOOD consistently obtains better results on the TNR95 metric, outperforming the other methods on all datasets on this metric. The smallest gap in performance was seen on the SVHN dataset, although the other metrics (AUROC and detection accuracy) indicate that there is a significant difference between the methods on that dataset as well. The most substantial performance gap between the methods examined is seen on the CIFAR100 dataset. DNN classification performance on CIFAR100 tend to be much lower than CIFAR10 and SVHN, this can affect the OOD detector’s performance. We observe that the results obtained by FOOD and MALCOM (the method achieving the second best OOD detection results) for the TNR95 metric on the CIFAR100 dataset are respectively $88.9$ (on average) and $75.2$. Resnet18-based models showed similar trends; on the CIFAR100 dataset, the results for the TNR95 metric are $91$ (on average) for FOOD and $76.9$ for MALCOM, which again has the second best OOD detection results. Due to space limitations, we cannot present the full evaluation of the methods based on Resnet18.