A Survey on Open-Set Image Recognition

Discriminative OSR models [21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55] directly learn discriminative feature representations or classifiers for recognizing both known-class samples and unknown-class samples. According to whether to introduce unknown-class information into the training procedure, the discriminative models can be further divided into two groups: (i) one group of models [21, 52, 53, 22, 36, 37, 38, 31, 32, 35, 42, 47, 24, 25, 26, 33, 28, 29, 30, 34, 39, 40, 41, 50, 51, 54, 55] that aim at learning known-class discriminative representations only depending on known-class training samples, and (ii) the other group of models [23, 27, 45, 46, 48, 49, 43, 44] that aim at introducing unknown-class information for reducing the discrepancy between the training classes and testing classes.

(i) As for the first group that learns known-class discriminative representations, the existing DNN-based OSR methods can be roughly divided into four types: score-based methods [21, 52, 53], distance-based methods [31, 35, 54, 55, 22, 42, 32, 36, 37, 38, 47], reconstruction-based methods [24, 25, 33, 26], and others [28, 34, 29, 30, 41, 50, 51, 39, 40].

Score-Based Methods. DNNs are more vulnerable to the unknown-class samples than traditional methods in the OSR task because of the closed-set assumption of the softmax layer, which is commonly used to obtain classification probabilities as indicating scores. The scores of each sample belonging to all known classes outputted from the softmax layer are sum to 1, and the traditional classification methods take the index where the maximum score occurs as the predicted label, thus taking no account of unknown classes that are left out of the known classes.

To solve this problem, Bendale and Boult [21] introduced the Extreme Value Theory (EVT) into modeling the distribution of the distances between the activation vectors of training samples outputted from the penultimate layer of the network and the mean activation vector of each known class. Specifically, the aforementioned distances of each known class were firstly calculated, partial distances which had the most largest values were selected to fit the Weibull distribution as the extreme value distribution of the corresponding known class. At the testing stage, the probability of a testing sample belonging to each known class was calculated according to the distance between the activation vector and the mean activation vector of each known class falling into the modeled extreme value distribution of the corresponding class, thus obtaining the known-class scores. Besides, they also calculated the unknown-class score based on the weighted combination of known-class scores and the rectified weights. Finally, the testing sample is identified as one of the known classes if the maximum score occurred at the known-class scores and was larger than a threshold; otherwise, it would be classified as the unknown classes. The Weibull distribution is commonly used to model the extreme value distributions, and this method paves an early way for later OSR methods to fit the distribution of known-class features and use threshold to distinguish unknown-class samples from known-class samples.

However, due to the calculation complexity, EVT-based scores are gradually abandoned in many OSR methods, and the simple softmax scores combined with a comparing threshold are still widely used. Recently, Dai et al. [52] found that the original logit vectors perform better than the probabilistic scores additionally calculated by the softmax layer in the OSR task, since the translation invariance for the input logit vectors weakens the ability of the scores for capturing fine-grained information. Subsequently, Vaze et al. [53] also emphasized the effectiveness of the logit-based scores for boosting the model performance.

The EVT-based scores can also be regarded as one of the distance-based scores since the scores are calculated based on the distances between the instance-specific features and the class-specific prototypes. Such distance-based scores can be used for identifying different classes because the models are trained to shrink the intra-class distance and enlarge the inter-class distance, which will be described in detail in the following paragraphs.

Distance-Based Methods. Similar to the developing process of traditional closed-set classification methods, a research branch is extended from the study of the classification loss functions, which studies the distance-based loss functions for imposing constraints on the features for learning more compact and discriminative features. This research branch is reasonable for OSR because one of the main reasons that cause the recognition difficulty on unknown-class samples is that the known-class features over-occupy the space that should be reserved for unknown-class features, and cause the confusion between known-class and unknown-class features, as indicated in [76].

The distance-based loss functions are inspired by Fisher’s criterion, which aims at minimizing intra-class variance and maximizing inter-class difference. Such idea that constrains feature representations has now formed a research direction, representation learning. Hassen and Chan [31] applied a simple representation learning method for handling the OSR task. They took the logit vectors as the feature representations of the input images in a different space (i.e., the logit space), and updated the mean vector of each known class by each training batch. Then, an instance-specific loss term that constrained the $L_{2}$ distance (i.e., the Euclidean distance) between each training sample and the corresponding class-specific mean vector to be smaller, as well as a pair-wise loss term that constrained the Euclidean distance between different class-specific mean vectors to be larger, were combined with the cross-entropy classification loss (which was the commonly-used loss in the traditional closed-set classification task) for training the network. Jia and Chan [35] added a loss extension to a representation loss, which extracted activation vectors outputted from the penultimate layer of the network and formed a representation matrix for emphasizing the features that had the largest and the features that had the smallest magnitudes at the training stage, thus learning more discriminative feature representations.

Since the distance-based OSR methods emphasize the feature compactness within each known class as well as the feature discrimination between different known classes, the anchors/prototypes of these known classes are the key to the model performance. Yang et al. [54] proposed a convolutional prototype learning system, which utilized a CNN for extracting image features. To boost the intra-class compactness, the known-class features were constrained by a prototype loss to be close to class-discriminative prototypes, which were also learnable during the training stage. In [55], they added a one-versus-all loss for further boosting the feature discriminability, and analyzed the influence of both different unknown rejection criteria and different discriminative losses on the model performance under different open-set recognition deployments. Miller et al. [22] constrained known-class logit vectors to be clustered at the anchors of the corresponding classes, where each known-class anchor was pre-set at its class coordinate axis thus being equal to a scaled one-hot vector. Compared with general prototype-clustering methods where class anchors were learnable without additional constraints, this method where the anchors were fixed achieved better performance in the OSR task, and the constraints in the logit space reduced unnecessary noises caused by visual variations within each known class.

However, fixed prototypes play a limited role in constraining the feature discrimination. Hence, Xia et al. [42] proposed a constraint loss term for controlling the spatial location of these feature prototypes to be more discriminative. Compared with previous methods whose known-class features tended to occupy the same center part of the feature space as the unknown-class features did, this method limited the known-class prototypes to be located at the edge region of the feature space, thus alleviating the confusion between known-class features and the unknown-class features. This constraint was performed by constraining the variance of the distances between the prototypes and the center of the feature space. Considering that there was single prototype representing for each known class in previous methods which ignored the feature diversity within each class, Lu et al. [32] designed a prototype mining strategy before optimizing the feature space, which mined high-quality and diverse prototypes for each known class.

As an effective tool for constraining the distances between pairs of features that are from the same image or different images via data augmentation in the self-supervised tasks, contrastive loss has received more and more attention in recent years. Kodama et al. [36] applied a supervised contrastive loss for constraining the feature pairs from the same known class or different known classes, which was convenient for learning latent features that were with intra-class compactness and inter-class discrepancy. Similarly, Xu et al. [37] also utilized supervised contrastive learning for improving the quality of the learned feature representations.

Additionally, some OSR methods aim at designing feature representations or classifiers with angles for improving inter-class similarity and inter-class difference. Park et al. [38] proposed to learn divergent angular representations, which improved the global directional feature variation. Cevikalp and Saglamlar [47] introduced a quasi-linear polyhedral conic classifier, which constrained the known-class regions to be L1 or L2 balls.

Reconstruction-Based Methods. There is a view exists in the OSR task that reconstruction is valuable for the model discriminability since known-class samples generally have smaller reconstruction errors than unknown-class samples which the model has never seen. Following this point, some OSR methods make use of the latent feaures in a reconstruction network, reconstructed samples, or reconstruction errors for improving the model discriminability.

Yoshihashi et al. [24] used the latent feature representations in a reconstruction network in addition to the network predictions for classification. They proposed a special autoencoder, deep hierarchical reconstruction network, for extracting latent features at each layer. At the training stage, the network was trained by jointly using the classification loss and the reconstruction loss, where the classification loss used the EVT-based scores [21]. At the testing stage, the classification scores were used for identifying known- and unknown-class samples.

Oza and Patel [25] split the whole training process of a autoencoder into two steps. Firstly, they trained the encoder and a classifier which was connected to the latent features by traditional cross-entropy classification loss. Then, the weights of the encoder and the classifier were fixed, and the decoder was trained by a well-designed pair-wise reconstruction loss. At this step, the feature maps input to the decoder were controlled by two parts: the encoded features and conditional vectors via linear modulations, thus obtaining pairs of original/reconstructed images. The decoder was trained to maximize the reconstruction errors of label-unmatched pairs while minimize the reconstruction errors of label-matched pairs. At the testing stage, the encoded feature of the testing image was linearly modulated by all-class conditional vectors, and the corresponding reconstruction errors of all known classes could be obtained, which were compared with a threshold for determining the predicted label.

Huang et al. [33] integrated prototype learning and reconstruction, they proposed to reconstruct class-specific semantic feature maps rather than instance-specific images for boosting the semantic discriminability of the model. They modeled an autoencoder for each known class in the latent space, which was used to reconstruct the feature maps extracted by the backbone encoder from the input images. The class-specific pixel-wise reconstruction error maps were taken as logits and converted to traditional logit vectors by a softmax layer and pooling layer. At the training stage, the model is trained by a classification loss constraining the logit vectors. At the testing stage, the reconstruction errors corresponding to the all-class autoencoders were used for identification.

Perera et al. [26] found a new path to make use of reconstruction information, where reconstructed images were used. They firstly trained a generative model (e.g. vanilla autoencoder, conditional autoencoder, etc.) for obtaining the reconstructed images for known classes, then they extended the original images by taking the reconstructed ones as new dimensions for learning a classifier.

Others. Some methods aim at mining discriminative information from data augmentation, collective decision, multi-task learning, gradients, space transformation, hierarchical attentions, and even with the help of visual psychophysics.

Perera and Patel [28] found that extreme geometric transformations could cause different feature representations, hence, they passed the features corresponding to the transformed images through parallel network branches and used majority voting for final prediction.

With the development of network backbones, vision transformers have been more and more widely applied due to their better performance. Azizmalayeri and Rohban [34] empirically observed that taking a vision transformer as the backbone and using the softmax probabilities for classification could achieve better performance than other OSR methods. Besides, they also found that delicately selected data augmentations rather than standard training augmentations were helpful for boosting the model discriminability.

Besides, Jang and Kim [29] proposed to integrate multiple one-vs-rest networks as the feature extractor, and combined the multiple decisions for modeling the final decision score for the input image. Oza and Patel [30] proposed a multi-task learning method for OSR, which simultaneously trained a classification loss in the latent feature space and a reconstruction loss at the end of an autoencoder, which was similar to [24]. Lee and AlRegib [41] utilized gradient-based feature representations for recognition, since gradients contained information about how much the model need to be updated for properly representing known-class samples. Baghbaderani et al. [50] proposed to utilized the transformations among three spaces (i.e., the original image space, the latent feature space, and an abundance space) for exploiting more discriminative information. Liu et al. [51] proposed an orientational optimization strategy for constraining the feature space and a hierarchical spatial attention mechanism for capturing the global dependencies in the feature space, which further improved the feature discriminability. Sun et al. [39] proposed an hierarchical attention network for learning fine-grained known-class features, which progressively aggregated both the hierarchical attentioned features and the contextual features in each hierarchical attentioned feature map. Huang et al. [40] were inspired by the fact that the OSR task is challenging for computer vision algorithms while is easy for human, they incorporated a psychophysical loss and the corresponding network architecture into deep learning, which could support reaction time measurements to simulate human perception.

(ii) As for the second group that introduces unknown-class information into model training, the existing DNN-based OSR methods can be roughly divided into two types according to different sources of the unknown-class information: methods that exploit unknown-class information from known-class samples [23, 48, 49, 45, 46, 27], and methods that introduce unknown-class information from outlier-class samples [44, 43].

Known-Class Exploitation. The first group of OSR methods fall into a bottleneck since only aiming at learning more discriminative known-class features or classifiers is not enough to handle the unknown classes that the model will encounter in the testing set. Addressing this problem, some methods aim at exploiting unknown-class information based on known-class images or features via mixup [77], augmentation, splitting, etc.

Mixup is a data augmentation strategy, which linearly mixes images and the corresponding labels from two different classes. Vanilla mixup [77] produces new samples in the input space by using linear interpolations, while manifold mixup [78] produces new features in the latent feature space. Zhou et al. [23] learned data placeholders for unknown classes by producing new features via manifold mixup, which were taken as an additional class and used to train the model along with the known-class samples. By generating unknown-class features, the known-class features were constrained to be more compact and discriminative, thus alleviating the over-occupying problem for the OSR task. Besides, they also proposed to learn classifier placeholders, which represented a class-specific threshold for distinguishing unknown-class samples from known-class ones. Specifically, they gave an extra dimension for unknown classes in the output vector of the classifier as the learnable threshold for discriminating between known classes and unknown classes, which was conducted by constraining the value of the extra dimension to be the second largest among all of the dimensions, since the learnable threshold that could distinguish the target class from non-targeted classes was considered to have the ability of distinguishing known classes from unknown classes.

Besides, Wang et al. [48] proposed a new metric that couples both closed-set and open-set performance by pair-wise formulation about known-class and unknown-class features, and transformed this formulation into an optimization problem that minimized the corresponding risk. Similar to [23], they generated unknown-class features by manifold mixup, while the known-class features could be extracted from the training images. Jiang et al. [49] generated high-quality negative images by mixing them, which was proved to reduce both closed space structural risk and open space risk.

In addition to mixup, some methods use augmentation-based similarity learning, intra-class splitting, or multi-class interaction for mining unknown-class information based on known-class data.

Esmaeilpour et al. [45] adopted similarity loss for encouraging the model to learn how to discriminate between known-class and unknown-class samples, where unknown-class images were generated by distribution-shifting data augmentation. Specifically, they conducted two steps for the model training. At the first step, unknown-class images were generated via randomly rotating original images by multiples of $90^{\circ}$. At the second step, a classification model was trained with both known-class training images and generated unknown-class images, and two losses are imposed: a cross-entropy loss for classifying known-class images and a binary cross entropy loss for learning clusters for both known-class and unknown-class images via similarity supervision.

Schlachter et al. [46] exploited unknown-class information from known-class training images. Specifically, they split training images into typical subsets and atypical subsets by a closed-set classifier. Then, the atypical subsets were served as unknown-class data and were added into the training set as the ($C+1$)-th (where $C$ is the number of known classes) class for training a ($C+1$)-class classification model. These two methods are relatively straightforward and simple for building unknown-class data. To capture more precise unknown-class information, some researchers explore learnable strategies for learning unknown-class features based on known-class features.

Chen et al. [27] defined reciprocal points for capturing unknown-class information for each known class, containing both extracted image features from other known classes and a set of learnable features served as unknown-class features. During training, the extracted features from one known class were pushed to be far from the reciprocal points of the corresponding known class, by which the learned known-class features were located in the periphery of the feature space, while the unknown-class features were limited within a bounded region at the center of the feature space. In this case, the unknown space was shrunk and bounded, which could prevent the classification network producing high confidence for unknown-class test samples.

Outlier-Class Introduction. The methods that exploit unknown-class information from known classes are still limited by the training set. Sometimes, when the distribution of the training set significantly deviates from the distribution of the testing set, or when the data volume is small, the original training set is insufficient to support the model in exploring effective unknown-class information. In this case, some methods try to introduce outlier data into the model training.

The usage of outlier data can be traced to the object detection task, where the classification network is also trained by a background class indicating that there is no object of interest in the proposal. The introduction of outlier-class samples can prevent the network from outputting over-confident erroneous prediction for unknown-class samples. Dhamija et al. [44] utilized this idea for reference, and they took the digital images from some classes of the NIST Letters dataset [79] as the known-class samples, while taking the rest-class images from NIST Letters as the unknown-class samples for testing and taking the images from CIFAR10 [80] and MNIST [20] as the outlier samples. They firstly found that unknown-class features usually presented lower feature magnitude as well as higher SoftMax entropy than known-class features. Based on this default observation, they designed an entropy-based loss and a magnitude-based loss to constrain both the known-class samples and outlier samples in the model training for increasing such separation, which boosted the model robustness to the unknown-class samples that are unavailable in training.

Such operations that introduce outlier-class samples in the model training were also adopted (called outlier exposure) in the out-of-distribution (OOD) detection task [81, 82], where the model ability of rejecting unknown-class samples were more focused on. However, the known-class samples were categorized into one group in the outlier-class related loss, either in the open-set method [44] or in the OOD detection methods [81, 82], which could affect the closed-set classification performance in the OSR task. Besides, these methods adopted a same margin between different known classes and outlier classes on different datasets, which limited the open-set recognition performance.

To address the problems mentioned above, Cho and Choo [43] preferred multiple distance-based classifiers based on the principle of linear discriminant analysis (LDA) rather than the commonly-used SoftMax classifier. Besides, they designed a class-inclusion loss which encouraged each outlier sample to be far away from the nearest class-wise hypersphere classifier, which further boosted the model discriminability. Suggested by [83], they chose images from the ImageNet dataset as the outlier-class samples. In addition to the discriminative methods introduced above, a generative OSR method [64] also adopts outlier exposure to augment the training set, which will be introduced in the following paragraphs.

With the development of generative models, more and more OSR methods pay attention to adopt generative learning techniques for boosting the model discriminability. Generative OSR models [56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72] mainly learn the distributions from known-class samples, based on which the discrimination criterion of how to both identify unknown-class samples and classify known-class samples is built. According to the specific generative models utilized, these methods can be further divided into three groups: Generative Adversarial Network (GAN)-based methods [64, 65, 66, 67, 69, 70, 71], Auto-Encoder (AE)-based methods [56, 57, 59, 60, 62, 63, 72], and others [58, 61, 68].

GAN-Based Methods. Considering that GANs are able to generate various new samples, some methods make use of GANs for generating unknown-class samples to fill the missing information about the open space. They model the data distributions implicitly by adversarial training rather than explicitly by fitting a specific parameterized distribution. Most of GAN-based methods generate unknown-class samples or features only based on the known-class training samples, and seek for different assumptions about the position of the unknown-class samples/features as well as different strategies to make use of the generated unknown-class samples/features.

Ge et al. [66] assumed that unknown-class samples lay in the mixed distribution of known-class distributions. They modified the training process of the conditional GAN, where several prior known-class distributions were mixed. Based on the generated unknown-class samples, they extended OpenMax [21] by providing the explicit probability estimation about unknown classes.

Neal et al. [67] assumed that unknown-class samples lay outside of any known-class distribution in the feature space, but simultaneously resembled known-class images in the pixel space. Each unknown-class sample was generated from a known-class training image based on an encoder-decoder-GAN. Specifically, they minimized the reconstruction errors between the input known-class image and the generated image for guaranteeing the similarity in the pixel space, and simultaneously minimized the classification probability of the generated image to be classified into the corresponding known class. Then, the generated images were taken as one additional class to augment the training set and used to train a ($C+1$)-class (where $C$ is the number of the known classes) classifier.

Jo et al. [69] assumed that the unknown-class features lay in adjacent feature space to the known classes. Hence, they combined the generative in the GAN with a marginal denoising autoencoder so as to model the distribution away from each known class. With the unknown-class features generated, they trained a $C$-class classification model with an additional entropy-based regularization loss for encouraging the model to have high uncertainty on unknown-class features, which tightened the decision boundary of known classes.

Chen et al. [65] extended their previous discriminative OSR method [27] by generating confusing samples based on an adversarial training strategy for boosting the model discriminability. Specifically, the generated features were constrained to cheat the discriminator, while they were also constrained to be close to the reciprocal points by maximizing the output entropy from the classifier. At this time, the generated confusing features were located at the boundary between known-class features and reciprocal points. After feature generation, they designed three losses for training the classification model, including two adversarial losses that both maximizing the distance each reciprocal point to its corresponding known-class prototype and limiting the distance within a learnable margin for both reducing the empirical classification risk and bounding the open space risk, and a distance-based entropy loss constraining the generated confusing features to be located near to the reciprocal points for further enhancing the distinguishment between known classes and unknown classes.

Inspired by [27, 65] which pushed known-class features to the periphery of the feature space while constrained the unknown-class features within the center of the feature space, Xia et al. [71] generated unknown-class features that were located near known-class prototypes while outside a learnable distance to the center of the known-class prototypes. They also introduced adversarial motion properties that enabled the adversarial motion of the distance margin, which further reduced both the empirical risk and the open space risk.

However, the above methods usually generated unknown-class samples outside the decision boundaries that were easy to be discriminated, and neglected the “hard negative” samples that mattered more to the open-set recognition performance. To address this problem, Moon et al. [70] took diverse difficulty levels of generated unknown-class samples into consideration from the classifier’s perspective. They trained a multi-group convolutional classification network and a copied counterpart whose layers were separated by the same predefined criteria based on multi-level knowledge distillation for generating features in hard- or easy-difficulty level, and concatenated with a GAN for generating multiple features at different difficulty levels. As done in [65], the generated unknown-class features were given uniform probabilities as labels when finally training the classifier.

Considering that sometimes outlier datasets could be available, some discriminative OSR methods [44, 43] have introduced outlier samples for simulating unknown-class samples. However, such models perform poor generalization ability on diverse unknown-class samples, since the outlier samples utilized in training could not cover the open space exhaustively. To address this problem, Kong and Ramanan [64] proposed OpenGAN, which trained a GAN combined with a $C$-way classifier based on both the original known-class training samples and the introduced outlier-class samples, and took the discriminator in the GAN for distinguishing unknown-class samples from known-class ones. The model selection was also operated based on outlier validation samples, which was found effective even the outlier validation samples were sparse or biased.

AE-Based Methods. As mentioned before, some reconstruction-based discriminative OSR methods distinguish unknown-class samples from known-class ones based on the discrepant reconstruction errors outputted from the autoencoder. In recent years, some generative OSR methods have utilized autoencoders to model the known-class distributions explicitly, such that unknown-class samples could be rejected if it does not belong to one of the modeled known-class distributions, and known-class samples could also be classified according to which known-class distribution it belongs to.

As a typical autoencoder, variational autoencoder (VAE) [84] has been widely used in many visual tasks, which models the known-class samples as a standard Gaussian distribution. However, using VAE to model feature distribution of all known classes as a Gaussian distribution would damage the discriminability between two different known classes. Sun et al. [59] extended the VAE into a class-discriminative autoencoder based on the probabilistic ladder architecture [85]. Specifically, the encoder encoded each image into two distribution parameters in a Gaussian (i.e., mean and standard deviation). The latent feature sampled by the learned distribution was forced to approximate the Gaussian distribution of the corresponding known class, whose standard deviation was an identity matrix and mean was mapped from the one-hot label. In the inference stage, an unknown-class image could be detected not only based on its deviated distribution, but also according to its high reconstruction error. Subsequently, they also extended the adversarial autoencoder (AAE) [86] to another class-discriminative autoencoder in a similar manner [62].

Inspired by the above method, Guo et al. [60] replaced the CNN backbone with the capsule network [87], where each latent neuron in the network outputted a vector instead of a scalar, such that the encoded latent features could represent more diverse information. Rather than using an additional classifier for enabling the latent features to be discriminative as done in [59, 62], they utilized a contrastive loss with a margin for forcing the encoded features to be located at the corresponding known-class region and keeping different known-class features to be far from each other.

However, modeling each known-class-feature distribution as a single Gaussian could not well represent the intra-class discrepancy. To address this problem, Li and Yang [72] assumed that each known-class-feature distribution followed a Gaussian mixture distribution, which could represent the intra-class discrepancy by different Gaussian components. They embedded the neural Gaussian Mixture Model [88] into the autoencoder to mapping the latent feature into the marginal distribution, which was called a dual distribution since two opposite probabilities could be derived from it: i) the probability of the latent feature belonging to known classes, and ii) the probability of the latent feature belonging to unknown classes. Either known-class features or unknown-class features encoded by the trained model could form a distribution peak, hence, a testing sample could be recognized according to the distribution peaks in the latent space. Once it lay close to the known-class peak, it would be further classified by the known-class classifier in the latent space.

Similarly, Cao et al. [56] also modeled the feature distribution of each known class as a Gaussian Mixture, but they directly modified the Gaussian Mixture VAE [89] rather than embedding the neural Gaussian Mixture Model. Liu et al. [57] combined conditional VAE [90] and prototype learning to constrain the feature distribution of each known class as multiple Gaussians, which could also be regarded as a Gaussian Mixture. Considering that some complex feature distribution could not be represent by Single Gaussian or Gaussian Mixture (e.g., sub-Gaussian and super-Gaussian), Sun et al. [63] introduced the mixture of exponential power distribution into the network based on a new re-parameterization strategy, which modeled feature distributions from different known classes by discrepant mixtures of exponential power distributions.

Generally, the main research trend of this group of OSR methods aim to seek for more effective explicit representation about the known-class feature distributions, as well as investigate how to make better use of the reconstruction errors for boosting the feature discriminability, where the second motivation coincides with that of reconstruction-based discriminative OSR methods.

Others. In addition to the GAN-based and AE-based methods, there also exist some OSR methods that adopt other generative models (e.g., flow-based models) [58], or only take the encoder for modeling feature distributions [61], or generate instance-weights for the transfer learning [68].

Flow-based generative models [91, 92, 93] generate images or features of comparable quality to GANs, but can model the training distributions explicitly due to its invertible architecture, which also model the training-class feature distribution in the latent space as a standard Gaussian as done in VAEs. Zhang et al. [58] combined a typical flow network Resflow [93] with a known-class classifier in the latent space. Resflow was used as a density estimator for detecting unknown classes, while the latent classifier was used for maintaining the known-class classification accuracy.

Considering that image-level reconstruction in AE-based methods would take all image pixels into account where many pixels were irrelated to the class or even easy to cause misleading, Sun et al. [61] removed the decoder in an autoencoder, and used a Kullback-Leibler (KL) divergence loss to constrain the latent features such that the feature distribution of each known class was modeled as a single Gaussian. Besides, they designed a multi-scaled mutual information maximization strategy for establishing the dependence between the input image and its latent features, which further improved the feature discriminability.

Another innovative generative OSR method was to learn intermediate vectors used for generating new samples rather than directly generate images or features, which was proposed by Fang et al. [68]. Motivated by both the transfer learning theory and probably approximate correct theory, they aligned the known-class samples to the samples in an auxiliary domain (representing unknown-class domain) by learnable instance-weights. Through such instance-weighting strategy, the unknown-class samples can be detected by the instance-weights.

No matter the discriminative models or the generative models mentioned above, both of them are easy to fall into the trap of lazy learning, since once a model has sought a group of weight parameters that could minimize the loss function, it would no longer update these weight parameters. Such lazy learning would inevitably lead the model to learn the correlation relationships which are easy to learn but are relatively fragile. To address this problem, some causal models have been proposed [75, 73, 74], which aim to seek for causal relationships from the training data, which alleviate the confounders in the features learned by the non-causal OSR methods in two manners: disentangling the robust representations from the highly-coupled correlated features [75], and generating more reliable known-class samples based on counterfactual generation [73, 74].

Causal Disentanglement. Yang [75] proposed an invertible causal model for open-set recognition, which consisted of an invertible encoder (i-RevNet [94] was used here) for encoding images into features and class functions for providing the distribution prior of the encoded features belonging to each known class. Different from AE-based generative OSR methods that modeled the feature distributions as explicit fixed distributions, the class functions here were built in the form of Structural Causal Models (SCMs) [95] each of which was a directed acyclic graph. At the training stage, the encoded features were constrained to match the corresponding-class prior class functions. At the inference stage, similar to the inference strategy in most of AE-based generative OSR methods, the samples were classified/detected according to their probabilities belonging to these class functions.

Counterfactual Generation. Since the generative models have the strong ability of modeling the known-class feature distributions and training samples may be insufficient in some cases, generative OSR methods have become the main stream in handling the OSR task. However, most of these methods generate samples or features based on the one-hot class labels directly, which neglect the non-causal confounders infiltrating into different known classes, leading to the bias of the generated samples or features. To address this problem, some methods adopt counterfactual generation for generating more reliable samples or features [73, 74].

Yue et al. [73] proposed a counterfactual generation method for handling the OSR task based on TF-VAEGAN [96], which consisted of an encoder for encoding the images into the latent features, a decoder/generator for reconstructing/generating counterfactual images based on both the latent features and the provided one-hot labels, and a discriminator for distinguishing the real images (True) from the counterfactual images (False). They trained the network with three losses: a beta-VAE loss for modeling the latent feature distribution as the isotropic Gaussian distribution, a contrastive loss for minimizing (or maximizing) the reconstruction errors between an input image and its reconstructed image based on the matched (or mismatched) labels, and a GAN loss for encouraging the generated image to deceive the discriminator. At the inference stage, similar to [25], the minimal distance between the testing image and its $C$ counterfactual images generated based on $C$ one-hot known-class labels was compared with a threshold for detecting unknown-class images. If the distance is smaller than the threshold, the testing images was predicted as the known class corresponding to the minimal distance.

Zhou et al. [74] applied the similar method in handling the open-set Synthetic Aperture Radar (SAR) image target recognition task, which still outperformed other OSR methods.

In the common scenario, more than one known class exists in the training set, and training the model to be able to distinguish among different known classes also improves the model ability of distinguishing unknown classes to some extent. In an extreme scenario, only a single known class is available, called one-class classification [106, 107, 108], where some discriminative OSR methods that rely on boosting the known-class feature discriminability to improve the OSR performance are likely to fail.

To address this problem, some discriminative methods embedded the one-class support vector machine into the loss functions [109, 110], or applied specific data transformations for capturing the unique geometric structural information of the known class [111, 112]. However, since the negative samples were unavailable in the model training, bias would exist in the learned decision boundary. Considering that generative models could model the distribution of the known-class data, generative methods were widely used in the one-class classification task, which recognized the unknown-class samples based on their deviated distributions from the known-class distribution [113, 114]. Besides, the reconstruction errors could be still utilized to distinguish unknown-class samples from known-class samples.

In the common scenario, the dataset is generally static and fixed, and OSR models learn from the existing dataset only at one time. However, data in the realistic scenarios usually presents dynamically, and new data can be obtained periodically or even continuously. It is costly to re-train the model every time. Under such application demand, a series of open-world recognition methods have been proposed, which aim to continuously detect and add new classes encountered [115, 116, 117].

Bendale and Boult [115] firstly proposed the open-world recognition concept, and also extended the nearest class mean classifiers to the open-world recognition task. Cao et al. [116] proposed a progressive transductive method, which selected unlabeled new samples and provided them pseudo labels based on clustering results for updating the feature prototypes. Wu et al. [117] replied on graph representation and learning for predicting and utilizing new-class samples, where a graph network was used for extrapolate embeddings for features extracted from new data based on a feature-level graph and a prediction network was used for predicting pseudo labels for the new features. Generally, clustering and transductive learning are two common tools for handling the open-world recognition task. The nearest class mean can be considered as a clustering strategy since it pulls the samples to be close to its nearest neighbors. The graph network is also a common tool for transductive learning. With transductive learning, a model can progressively make use of new samples, even if their labels are unavailable. With clustering, the model can be updated with relatively smaller cost.

Another problem that an OSR model would encounter in the realistic scenarios is that the data has a long-tailed distribution, and the model would prefer the majority classes where the number of samples is dominant while neglect the minority classes whose samples are significantly insufficient. The long-tailed problem is an extreme case of the class imbalance problem. There are some simple strategies for alleviating the model bias towards majority classes, e.g., data re-sampling technique (including down-sampling from the majority classes and over-sampling from the minority classes) and loss re-weighting (increasing/decreasing the loss weights for the minority-/majority-class samples).

Recently, the open-set long-tailed recognition task has received more and more attention, how to mine effective information in the minority classes in an open-set environment becomes a key problem in this task. Liu et al. [118, 119] firstly formally defined the open-set long-tailed recognition task, and handled it based on a dynamic meta-embedding mechanism. The meta-embedding mechanism associates the majority-class visual features with the minority-class visual features for enabling the model to be robust to the minority classes, and dynamically calibrated the feature norm based on the visual features in the memory bank for supporting open-world recognition. Cai et al. [120] proposed a distribution-sensitive loss, which provided larger weights to minority-class samples when constraining the intra-class distances to be minimized. Besides, they designed a distance-based metric for recognition according to the feature distance to the clusters.

As mentioned in Sec. I, the distribution shift contains both semantic shift that exists in the common OSR task and covariate shift. In the open-set domain adaptation task which was firstly proposed in [121], both semantic shift and covariate shift exist. In other words, unknown-class samples would exist in the testing set, besides, known-class samples in the training set and testing set are in different domains (i.e., source domain and target domain, respectively). Most of existing closed-set domain adaptation methods aim to align the whole target domain with the source domain according to the labeled known-class samples in the source domain as well as the unlabeled (or partly labeled) samples in the target domain. However, unknown-class samples in the target domain would be improperly aligned with the known-class samples in the source domain in the open-set domain adaptation task, which would harm the distinguishability between known classes and unknown classes.

To address this problem, Busto and Gall [121] added an implicit outlier detection mechanism when assigning the images in the target domain to some classes in the source domain, thus the images that didn’t belong to known classes could be discard in the assignment. To separate unknown-class samples in the target domain from known-class samples in the target domain, Saito et al. [122] adopted adversarial training between a feature extractor and a ($C+1$)-class classifier, where the classifier was trained to not only classify the known-class source features but also distinguish between known classes and unknown classes according to the probability of the ($C+1$)-th class, while the feature extractor was trained to cheat the classifier. Liu et al. [123] were inspired by the observation that the gap between known-class samples in the target domain and known-class samples in the source domain is much smaller than the gap between unknown-class samples in the target domain and known-class samples in the source domain. They designed a coarse-to-fine weighting mechanism, which iteratively operated two steps: multi-binary classifier training step that measured the similarity of an target image to each source known class, and the binary classifier learning step that trained a binary classifier based on both known-/unknown-class target samples selected by their high/low similarities to the source classes.

The common OSR task is deployed in large-scale datasets. However, in some extreme realistic scenarios, the number of training samples in each known class is extremely small, and such task is called few-shot open-set recognition task [124].

To handle this task, Liu et al. [124] extended a closed-set few-shot learning model into the open-set environment. They added some pseudo-unknown samples into the model training via an open-set distance based loss term. Jeong et al. [125] recognized unknown-class samples based on their larger difference from the transformed prototypes. Wang et al. [126] proposed an energy-based model, where the samples that deviated from either the class-wise features or the pixel-wise features of the few-show known-class samples were assigned larger energy scores.

The open-set adversarial defense task combines the open-set recognition task that aims to both classify known classes and identify unknown classes during testing and the adversarial defense task that aims to enable the network to defend against the imperceptibly adversarially-perturbed images, and was firstly proposed in [127].

To handle this task, Shao et al. [127] proposed a open-set defense network consisting of an encoder with denoising layers and a classifier for learning noise-free features. Further, they incorporated a decoder for reconstructing clean images, added a self-supervision loss for improve the feature discriminability, and a clean-adversarial mutual learning mechanism where another classifier (which handled clean images) was mutually learned with the original classifier (which handled adversarial images) for facilitating the feature denoising [128].

In the common OSR scenario, the model relies heavily on clean labels. However, providing labels to large-scale dataset is very costly and error-prone, and realistic data inevitably contains noisy/incorrect labels. The OSR with label noise task was firstly proposed in [129], following which some researchers have studied in this task.

Wang et al. [129] proposed an iterative learning framework, which iteratively detected noisy labels, enlarged the difference between clean labels and noise labels, and applied a re-weighting module for encouraging the model to learn more from clean labels rather than noisy labels. Sachideva et al. [130] made use of the subjective logic loss [131], which could yield higher losses on closed-set noisy samples while lower losses on open-set samples. Unlike the above methods, Wei et al. [132] empirically demonstrated that open-set noisy labels would even help boost the model robustness against noisy labels, and introduced open-set samples with dynamic noisy labels into the model training as a regularization term.

Under the standard-dataset setting, five datasets are used in [59, 21, 22, 24, 25, 26, 27, 65, 67, 58, 23, 63, 105, 39]:

1. -

   MNIST: In this dataset, the images are taken from the Mixed National Institute of Standards and Technology (MNIST) datasets [20], which contains 70000 10-class hand-written digit images (28$\times$28), including 60000 training images and 10000 testing images. 6 classes are randomly selected as the known classes, while the rest 4 classes are taken as the unknown classes.

2. -

   SVHN: In this dataset, the images are taken from the Street View House Numbers (SVHN) dataset [133], which contains 99289 10-class street-view house digit numbers (32$\times$32), including 73257 training images and 26032 testing images. Similarly, 6 classes are selected as known classes, while the rest 4 classes are taken as the unknown classes.

3. -

   CIFAR10: In this dataset, the images are taken from the CIFAR10 dataset [80], which contains 60000 10-class natural object images (32$\times$32), including 50000 training images and 10000 testing images. Similarly, 6 classes are selected as known classes, while the rest 4 classes are taken as the unknown classes.

4. -

   CIFAR+10/+50: In this dataset, the images are taken from both the CIFAR10 and CIFAR100 [134] datasets. Similar to CIFAR10, CIFAR100 contains 60000 100-class natural object images (32$\times$32), including 50000 training images and 10000 testing images. The 10 known classes are fixed to the 10 classes in the CIFAR10 dataset, and 10 or 50 classes are randomly selected from the CIFAR100 dataset for CIFAR+10 or CIFAR+50 as the unknown classes.

5. -

   TinyImageNet: In this dataset, the images are taken from the Tiny-ImageNet dataset [135], which is a 200-class subset of the ImageNet dataset [136] that contains 120000 natural object images (64$\times$64), including 100000 training images, 10000 evaluating images, and 10000 testing images. 20 classes are selected as known classes, while the rest 180 classes are taken as the unknown classes.

Under the cross-dataset setting, the 10-class CIFAR10 dataset is taken as the known-class dataset, while the four datasets collected by [137] are taken as the four unknown-class datasets respectively: ImageNet-crop, ImageNet-resize, LSUN-crop, and LSUN-resize, which are cropped or resized images from 200-class TingImageNet and 10-class LSUN [138].

Under the cross-dataset setting, three semantic shift datasets which contain higher-resolution images from different subclasses of birds, cars, and aircrafts are used in [52, 53, 39]:

1. -

   CUB: In this dataset, the images are taken from the Caltech-UCSD Birds (CUB) dataset (CUB-200-2011) [139], which contains 11788 200-class bird images with both labels and attribute tags, including 5994 training images and 5794 testing images with various image sizes. 100 classes are randomly selected as the known classes, while the rest 100 classes are further divided into three groups of unknown classes according to the attribute similarity between each unknown class and the whole known classes: ‘Easy’ group that contains 32 classes more distinguished from the known classes, ‘Hard’ group that contains 34 classes more similar to the known classes, and ‘Medium’ group that contains the rest 34 classes.

2. -

   FGVC-Aircraft: In this dataset, the images are taken from the FGVC-Aircraft-2013b dataset [140], which likewise contains 10000 100-class car images with both labels and attribute tags, including 6667 training images and 3333 testing images with different image sizes. 50 classes are randomly selected as the known classes, while the rest 50 classes are also divided into three difficulty groups similar to CUB: 20-class ‘Easy’ group, 13-class ‘Hard’ group, and 17-class ‘Medium’ group.

3. -

   Stanford-Cars: In this dataset, the images are taken from the Stanford-Cars dataset [141], which contains 16185 196-class aircraft images with labels, including 8144 training images and 8041 testing images (360$\times$240). In [39], the first 98 classes are selected as the known classes, while the rest 98 classes are taken as the unknown classes.

Under the cross-dataset setting, the subset of FGVC-Aircraft that contains above selected 50 known classes is taken as the known-class dataset, while the 200-class CUB and 196-class Stanford-Cars datasets are taken as the two unknown-class datasets respectively: Aircraft-CUB, Aircraft-Stanford-Cars.

The ACC and AUROC results of 34 representative OSR methods that have been evaluated on the coarse-grained datasets under the standard-dataset setting are reported in Tables I and II, where the results of the comparative methods are cited from their original papers or the papers citing them and sorted under both the year and the AUROC metric on TinyImageNet. Besides, the corresponding macro-F1 scores under the cross-dataset setting are reported in Table III, where the results of some methods are absent since they are not reported anywhere. For maintaining the consistency, the macro-F1 results are also sorted following the above strategy.

The backbones and the network architectures are also listed for better comparison. “Convs”, “ResNet”, “Wide-ResNet”, and “Transformer” denote that the corresponding methods are developed based on plain CNNs, ResNets, Wide-ResNets, and vision transformers, respectively. Since different methods usually adopt different layer configurations, in spite of using the same group of backbones, we take “*” for covering different configurations. Besides, “+” and “/” in Backbone item denote that the method combines two networks as its backbone and adopts different backbones on different datasets, respectively. “E”, “G”, “D”, and “D*” in Architecture item denote encoder, generator, decoder, and discriminator, respectively.

The ACC, AUROC, and OSCR results of some OSR methods that have been reported on CUB, FGVC-Aircraft, and Stanford-Cars, under the standard-dataset setting are reported in Tables IV and V. Besides, the corresponding macro-F1 scores under the cross-dataset setting are reported in Table VI. The results in these tables are sorted by both the year and the OSCR metric on CUB. “E”, “M”, and “H” denote the ‘Easy’, ‘Medium’, and ‘Hard’ difficulty levels, respectively. As done in [53], the ‘Medium’ and ‘Hard’ groups are reported jointly rather than separately on CUB.

In summary, according to the above comparisons, 8 points can be seen from these tables:

1. -

   As seen from Tables I and II, most of the comparative methods both with more powerful backbones and even with simpler architectures outperform the methods with more lightweight backbones in most cases, demonstrating that a powerful backbone is helpful for boosting the model discriminability. Although a more powerful backbone generally leads to higher model complexity, which limits the architecture complexity, it can still combine with some elaborate-designed modules such as attention mechanisms in HAN-OSFGR [39], comprehensive data augmentation in Cross-Entropy+ [53], lightweight reconstruction in CSSR [33] and MoEP-AE [63], prototype constraints in PMAL [32], outlier exposure in Class-inclusion [43], feature generation in OpenGAN [64] and IT-OSR-TransP [105], etc., for further improving the model performance.

2. -

   As seen from Tables I and II, if a method yields a higher ACC, it generally also achieves a higher AUROC. In other words, in most of the comparative methods, a better closed-set classification performance roughly brings with a better open-set detection performance, which is consistent with the observation in [53]. However, there are also lots of counter-examples when comparing in pairs, where a significant higher AUROC corresponds to a similar or even significantly lower ACC and a better ACC corresponds to a poorer AUROC. This is mainly because that some unknown-class samples are similar to some known-class samples, separating these unknown-class samples from known classes would harm the known-class classification accuracy. Hence, How to achieve a better balance between open-set detection performance and closed-set classification performance is an open issue in the OSR community.

3. -

   As seen from Tables I-III, the results under the cross-dataset setting are generally lower than those under the standard-dataset setting, mainly due to two reasons: (i) there also exists covariate shift between the training samples and unknown-class testing samples in addition to the semantic shift, which may cause the class confusion more easily. (ii) the macro-F1 metric simultaneously considers the closed-set classification performance and open-set detection performance, a model needs to not only correctly classify known-class samples, but also identify them as known classes based on a threshold.

4. -

   As seen from Tables I-III, the recent OSR methods have achieved nearly saturated performance, i.e., approaching or higher than 95%, especially with the transformer backbones. Hence, it is necessary to conduct evaluations on larger-scale, higher-resolution, and more difficult datasets.

5. -

   As seen from Table IV, the results in the ‘Medium’ and ‘Hard’ groups are generally lower than those in the ‘Easy’ group, mainly because that the unknown-class images in harder groups usually have similar appearance to known-class images and they are only different in some fine-grained properties. Besides, the recent model performances also achieve or approach saturation in the ‘Easy’ group, and identifying more difficult unknown-class samples mainly affects the overall performances. Hence, how to distinguish and make use of the harder unknown-class samples becomes a key to improving the model discriminability on fine-grained datasets. More effective acquisition and detailed processing of fine-grained information maybe helpful for distinguishing the samples in harder classes, which can then be utilized for model training based on some unsupervised techniques.

6. -

   As seen from Tables IV and V, the methods with ResNet* backbones usually perform better than those with Convs* backbones, and the methods with Transformer* backbones usually perform better than those with ResNet* backbones. This is mainly because that the residual connections in ResNets deepen the network thus enhancing the network’s ability to learn more complex features, and the multiple self-attention operations in transformers are helpful for capturing more fine-grained attentions on the semantic objects in images.

7. -

   As seen from Table VI, the results on Aircraft-CUB are significantly lower than those on Aircraft-Cars. This is mainly because that the distribution shift between FGVC-Aircraft and CUB is larger than that between FGVC-Aircraft and Stanford-Cars. Combining with the previous observations, we can find that both excessively larger distribution shift (e.g., the distribution between unknown-class samples on CUB and known-class samples on FGVC-Aircraft) and excessively smaller distribution shift (e.g., minor fine-grained differences between ‘Medium+Hard’ group of unknown-class samples and known-class samples) would cause the class confusion. Since the distribution shift problem is an inherent issue in the OSR community, transductive learning may be helpful for alleviating the distribution shift.

8. -

   As seen from Tables I-VI, generative OSR methods generally perform better than discriminator OSR methods under the standard-dataset setting and on coarse-grained datasets, mainly because that generative models are learned based on not only the relationships between inputs and outputs, but also the internal distribution structure of the data. However, such phenomenon is difficult to be observed either under cross-dataset setting or on fine-grained datasets, mainly because that the learned semantic-based distribution couldn’t adapt to the covariate shift under the cross-dataset setting, and the distributions learned from similar samples would also be similar on fine-grained datasets. Hence, the discriminative methods that adopt various strategies for enhancing the feature discriminability would be more effective either under the cross-dataset setting or on the fine-grained datasets.

The inherent issue in the OSR task is the semantic shift problem, where some new-class images would be encountered in the testing set. Since the deep learning models are data-driven models, only training the models based on known classes would make the models prefer the known classes, i.e., the models would wrongly predict the unknown-class samples as one of the known classes. Most of existing OSR methods are inductive methods, which assume that only known-class samples are available in the model training. Although they pursue more discriminative representations for known classes, hoping that unknown-class samples could be identified based on their deviation from known classes, the bias of the decision boundary between known classes and unknown classes still exists due to the absence of the real unknown classes. Transductive learning, however, has demonstrated its effectiveness in both other tasks and the OSR task. Up to now, there only exist two transductive OSR methods. Besides, some open sub-problems as mentioned in Sec. II.B also exist in the existing transductive OSR methods. Hence, how to effectively make use of the unlabeled testing samples is still worth studying, extremely the testing samples which are difficult to be identified.

The OSR task aims to simultaneously classify known-class samples as well as identify unknown-class samples. A good OSR model is desired to produce both high closed-set classification accuracy and high discrepancy between known classes and unknown classes. However, the two goals maybe sometimes inconsistent, when some hard unknown-class samples are confused with known-class samples in the feature space. In this case, separating these difficult unknown-class samples from the known-class ones is likely to harm the discriminability among different known classes. Such phenomenon can be also observed in the closed-set recognition task, where the accurate discrepancy among a part of the classes may lead to the confusion among some other classes. Hence, another open issue is raised: How to achieve consistently better performance or achieve a better balance between known-class classification and unknown-class identification? Mining and making use of the confused samples that are difficult to be identified may be a way to address this issue.

Since the unknown-class samples are generally unavailable in the model training, most of existing OSR methods firstly train a $C$-way classifier and identify the unknown-class samples by comparing the $C$-class based identification score with a threshold. Hence, the choice of the threshold become essential to the open-set recognition performance. Most of existing OSR methods empirically choose a threshold for identification, which may not be suitable for all classes in all datasets. Zhou et al. [23] proposed a class-specific threshold by a learnable strategy, which has paved a way for further researches about the threshold.

In the neuroscience field, human brains and animal brains are proved to have the ability of quickly recognizing new categories [142, 143, 144]. Some recognition mechanisms can inspire the future OSR methods. Some existing OSR methods have already provided examples. Yang et al. [55] designed the class-specific feature prototypes inspired by the abstract memories of different classes in human brains. Huang et al. [40] enabled the model to produce different reaction time for different images, aiming to improve the consistency with human behavior. Sun et al. [39] temporally aggregated the hierarchical attentioned features, inspired by the temporal attention mechanism in brains. Hence, it’s a promising future research direction to refer from recognition mechanisms in brains.

With the rapid expansion of data volume and the improvement of hardware performance, deep neural networks are entering the era of multi-modal large models. Recently, a lot of multi-modal large models pretrained on multi-modal large-scale datasets have demonstrated their generalization ability in assisting many visual tasks [145, 146, 147, 148, 149, 150], such as few-shot and zero-shot image recognition tasks. The pretrained large models store abundant prior information about the open world, which is a promising auxiliary tool for handling the OSR task. A simple approach to make use of the large models is to tune their prompts, which provide context or parameter information about the input data for helping the large models to better understand and handle the specific tasks.

Encouraged by the impressive generation ability of the pretrained large models, Qu et al. [151] collaborated several large models (ChatGPT [152], DALL-E [153], CLIP [154], and DINO [155]) to make use of the rich implicit knowledge in a training-free manner to reduce the reliance on spurious-discriminative features. Their method was operated at two stages. At one stage, virtual unknown-class images and simulated names were generated by ChatGPT and DALL-E. At the other stage, a testing image was inferred based on two alignments with the generated images and the expanded list of both known classes and virtual unknown classes by CLIP and DINO. Although this method was intuitive and didn’t need training, it was effective in handling the OSR task. To further make use of large models, Liao et al. [156] combined the open words and prompt tuning on large models for handling the OSR task. Different from [151], the open words were taken from WordNet [157] rather than generated by asking ChatGPT, and the learnable prompts improved the model adaptability to the downstream tasks. Besides, to XXX on larger-scale datasets, they firstly performed multiple independent group-wise prompt tuning on groups of fewer classes, and then predicted based on optimal sub-prompt. Furthermore, they proposed new baselines for fair comparison with large-model based OSR methods, which paved a way for future OSR methods based on prompt-tuning. How to make better use of the open-world pretrained large models based on new prompts and tuning strategies for handling the OSR task is still worth studying.