Evaluating Uncertainty Calibration for Open-Set Recognition

Deep neural networks (DNNs) have been very successful at vision-based tasks, which has led to their widespread deployment for real-world applications (e.g., object recognition, image segmentation, etc.). Nevertheless, a challenge of DNN-based vision systems lies in the network’s inclination to produce overconfident predictions during inference, especially when facing categories not seen during training (i.e., out-of-distribution (OOD) data) [1]. Within the realm of robotics, this motivates research questions such as how much trust can we put into the predictions of a DNN when misclassifications may have catastrophic consequences [2]? Methods have been proposed to mitigate the overconfidence problem by calibrating the predictive probabilities [3], or estimating the predictive uncertainties [4, 5], for various vision-based objectives (e.g., image classification [6], semantic segmentation [7], and object detection [8, 9]).

When deployed in real-world environments, we desire a DNN to have the discriminative ability to separate query inputs into either known (i.e., seen during training) or unknown (i.e., not yet seen during training) classes. This problem has previously been formalized and studied under the context of open-set recognition (OSR) [10]. OSR is the extension of object recognition from closed-set to open-set conditions, where classes outside of the training data can appear at inference time. Specifically, OSR trains a model on $K$ classes of an $N$ class training dataset. Then, at test time, the model is faced with $N$ different classes of which $N-K$ were not seen during training. An explicit none-of-the-above (or unknown) class placeholder (i.e., the $K+1$ class) is added during inference. OSR aims to assign correct labels from $K$ known classes to seen test-time samples, while detecting and placing unseen samples to the unknown class. Robotic applications of OSR include the division of semantically known places from semantically unknown [11], failure identification for self-driving cars [12], and entity detection [13].

Calibration refers to the issue of obtaining a model that has a predictive probability that reflects the true correctness likelihood. More formally, a model is well calibrated if outcomes predicted to occur with probability $p$ occur approximately $p$ fraction of the time [14]. In robotics, there is a clear need for uncertainty calibration of policies (e.g., autonomous driving [15]), especially for OSR. Previous work has evaluated calibration methods on both in distribution (ID) cases and under distributional shifts [16, 17]. The experimental settings in [16, 17] follow those in standard OOD detection works where a model trained on the ID set is evaluated for its reliability in identifying test images as either ID or OOD [18]. Although dealing with similar problems, OOD detection and OSR have been studied separately with different evaluation protocols. To the best of our knowledge, the evaluation of calibration methods for the OSR problem (i.e., open-set calibration) has not been explored. Hence, our work serves as a step toward this direction.

The first step of our evaluation is to enhance a model to perform OSR. To do this, we create a baseline by thresholding on the maximum probability which indicates the mostly likely predicted class. We also apply OpenMax [19], a well-known OSR method, for comparison. Following the standard experimental setup in OSR works, we use ResNet [20] as the backbone classification network and evaluate on the CIFAR10 benchmark dataset [21]. To test under OSR conditions, we follow the most common data partitioning protocol [22, 23, 24, 25, 26, 27]. Specifically, we split each dataset at random such that 6 classes are chosen to be known and the remaining 4 classes are unknown. We repeat the experiment over 5 runs and report the average score. Thus, the model is trained with 6 known classes and tested on all 10 classes. We map the randomly selected 6 known classes using indices 0-5 and set the other 4 unknown classes to start at index 6. A perfect OSR model should be able to assign correct labels to images belonging to any of the 6 known classes and identify images from all other classes as unknowns.

In the second step of our evaluation, we calibrate the trained model on the validation set in order to improve calibration during inference. Calibration is performed using temperature scaling [3], a simple and effective technique that makes adjustments to a model’s logits by a single temperature parameter $T$. Logits are probability vectors produced by a network before the softmax layer for a given input image. Additionally, $T$ is optimized on a held-out validation set and it can rescale a model’s confidence without affecting accuracy. After acquiring an OSR model, we apply temperature scaling on the model’s predictive confidence and evaluate the calibration performance in this novel problem setting. Concretely, to apply temperature scaling we reserve 10% of the training data as a validation set for optimizing $T$ and use the remaining 90% for training the model. To evaluate model calibration, we use the expected calibration error (ECE) [14] to measure the absolute difference between predictive confidence and accuracy. As a result of its simplicity and effectiveness, ECE is the most commonly used metric for model calibration. We also evaluate the Brier score within the framework of OSR as follows. If a trained classifier’s prediction $\hat{\bm{Y}}\in\mathbb{R}^{N\times K},$ where $N$ is the number of samples and $K$ is the number of known classes, then we can obtain a new prediction $\hat{\bm{Y}}_{\textrm{OSR}}\in\mathbb{R}^{N\times(K+1)}$ that takes into account the unknown class. For each prediction $\hat{\bm{y}}\in\mathbb{R}^{K}$ we estimate the probability of being unknown, i.e.,

Next, we calculate the Brier score by

where the predicted probability and ground truth of the $j$th class are $\hat{Y}^{ij}_{\textrm{OSR}}$ and $Y^{ij}$, respectively.

We visualize model calibration via reliability diagrams [28, 29], where the diagonal represents perfect calibration and any deviation from a perfect diagonal indicates miscalibration. The top rows of Fig. 1 and Table I present the reliability diagrams and scores for closed-set classification before and after calibration by temperature scaling. We see that temperature scaling significantly improves the model’s ability to match the true correctness likelihood in calibrating the predictions for this conventional 10-class closed-set classification problem. In the middle rows of Fig. 1 and Table I, the reliability diagrams and scores of a classifier operating in open-set conditions by thresholding on its predictive probabilities before and after calibration by temperature scaling are shown. In this scenario, temperature scaling still provides a clear improvement over uncalibrated models. Nonetheless, there exist clear gaps between temperature scaling calibrated confidence and perfectly calibrated confidence thus indicating that the calibration problem for open-set conditions is much more challenging. The bottom rows of Fig. 1 and Table I demonstrate the reliability diagrams and scores of a classifier operating in open-set conditions by applying OpenMax before and after calibration. Although OpenMax improves classification accuracy, it exhibits no obvious improvement in terms of calibration. In addition, we also conducted these experiments with disparate classification networks such as DenseNet [30] and EfficientNet [31] on the SVHN dataset [32] (see appendix). In all of these experiments, we observed the same findings as above.

In this paper, we assessed the efficacy of calibration methods for OSR. From the evaluation results, we observe that the calibration problem is much more challenging under open-set conditions in comparison to closed-set conditions. Previous techniques that calibrate confidence really well for closed-set classification only provide limited calibration performance for OSR. Moreover, traditional OSR approaches improve accuracy, yet they do not take calibration into consideration. Given these observations, we want to bring the community’s attention not only to the pursuit of more accurate OSR methods, but also well-calibrated OSR systems that can perform reliably and safely in open-set settings.