Cluster-aware Contrastive Learning for Unsupervised Out-of-distribution Detection

On account of the massive amount of image data and the high cost of labeling data, semi-supervised learning or unsupervised learning in OOD detection have been received great attention. For example, ODIN Liang et al. (2017) uses a basic pre-trained model and separates the softmax score distributions between ID and OOD images by a temperature scaling adding perturbation. Yang et al. (2021) designs a unsupervised dual grouping (UDG) to solve the semantically coherent unsupervised OOD. In particular, as one of unsupervised learning methods, self-supervised method is widely used in OOD detection. For examples, Madan et al. (2022) presents a self-supervised masked convolutional transformer block (SSMCTB). Zhou (2022) designs a hierarchical semantic reconstruction framework, which maximizes the compression of the auto-encode latent space. SSD Sehwag et al. (2020) uses self-supervised representation learning and Mahalanobis distance score to identify OOD samples on the pre-trained softmax neural classifier. Different from some methods based on pre-trained models, we solve the completely unsupervised OOD problem which does not introduce any prior semantic label information.

In self-supervised learning, contrastive learning methods have been widely used because of excellent representation ability via instance discrimination, such as SimCLR Chen et al. (2020) and MoCo He et al. (2020). Contrastive learning considers two augmented views of the same instance as the positive to be pulled closer, and the rest samples are considered as the negative to be pushed farther apart. In OOD detection, CSI Tack et al. (2020) uses data enhancement in contrastive learning to generate OOD negative samples to improve the sensitivity to OOD samples. Cho et al. (2021) proposes masked contrastive learning for OOD detection, performing a designed auxiliary task class-conditional mask in contrastive learning. To introduce semantic information, some researches try to add clustering as an auxiliary task to contrastive learning, such as PCL Li et al. (2020) and SWAV Caron et al. (2020). However, it is difficult to integrate instance and semantic representation ability at the same time and there are few studies on combining clustering and contrastive learning to solve OOD problems. We design a novel framework to introduce clustering, which has a wider vision to extract semantic information from sample relationship.

Contrastive clustering is essentially a deep clustering method based on contrastive learning and has been widely used in images Sharma et al. (2020), contexts Zhang et al. (2021) and graphs Wang et al. (2022). For example, Li et al. (2021) proposes a basic framework of contrastive clustering. The instance- and semantic-level contrastive learning are respectively conducted in the row and column feature space by a similarity loss function. And Graph Contrastive Clustering (GCC) Zhong et al. (2021) proposes a novel graph-based contrastive learning strategy to learn more compact clustering assignments. The core of contrastive clustering is performing the instance- and semantic-level contrastive learning to minimize the intra-cluster variance and maximize the inter-cluster variance, which gives a lot of inspiration for OOD detection. In OOD detection, Albert et al. (2022) applies an outlier sensitive clustering at the semantic level to detect the OOD clusters and ID noisy outliers. Lehmann and Ebner (2021) performs a layer-wise cluster analysis at inference to identify OOD samples. Since contrastive clustering will increase the inter-class variance for classification, it is inappropriate for direct application in OOD task. However, we could refer to its idea of combining clustering and contrast learning in OOD detection.

As shown in Figure 1, the proposed CCL framework contains two modules, a basic contrastive learning module and a clustering module. Considering a training set batch of $N$ unlabeled images $X=\{{x_{1}},{x_{2}},...,{x_{N}}\}$, image pairs $({{\tilde{x}}_{2k-1}},{{\tilde{x}}_{2k}})$, are obtained by data augmentation including random cropping, random color distortions, random Gaussian blur same with SimCLR Chen et al. (2020). The augmented image dataset is $\tilde{X}=\{({{\tilde{x}}_{1}},{{\tilde{x}}_{2}}),({{\tilde{x}}_{3}},{{\tilde{x}}_{4}})...,({{\tilde{x}}_{2N-1}},{{\tilde{x}}_{2N}})\}$. For the sake of description, the last layer of backbone encoder $F(\cdot)$ is called embedding layer and the last layer of MLP $G(\cdot)$ is called projection layer. The augmented image ${\tilde{x}}_{i}$ is input into the backbone encoder $F(\cdot)$ to obtain high-dimensional feature vector $h_{i}$ on the embedding layer. And $h_{i}$ is input into MLP $G(\cdot)$ to obtain ${z}_{i}$ on the projection layer.

In the basic contrastive learning module, a classic self-supervised contrastive loss $\mathcal{L}_{self}$ is calculated on the projection layer,which focuses on the relationship between samples and their enhancement samples for instance-level representation. In the clustering module, for each $h_{i}$ on the embedding layer, the corresponding cluster center $c_{i}$, $i=1,...,2N$, is obtained by K-means to calculate the cluster-aware contrastive loss $\mathcal{L}_{cluster}$. It includes two parts: a cluster center loss $\mathcal{L}_{CCL}$ and a cluster instance loss $\mathcal{L}_{CIL}$. The self-supervised contrastive loss updates the MLP and are simultaneously combined with the cluster-aware contrastive loss to update the backbone encoder. Finally, OOD detection needs a score function $score$ to identify whether a new input image is OOD sample according to its score.

The loss function plays a decisive role in contrastive learning. Although traditional contrastive learning has strong individual discrimination capacity, it ignores the relationship among instances with similar semantic. On the contrary, Supervised contrastive learning (SupCon) Khosla et al. (2020), uses label information to extract the relationship between samples. Moreover, it is a natural and common way to introduce semantic information in unsupervised tasks utilizing cluster centers obtained by clustering as pseudo labels. Combining the above two advantages of using semantic information, we design a novel loss function as an auxiliary task in contrastive learning.

Though the idea that combining contrastive learning and clustering is not proposed for the first time, how to make it appropriate for OOD detection needs further discussion. Recent research Sinhamahapatra et al. (2022) concludes that the data clustering performance of the self-supervised contrastive learning is not as good as supervised contrastive learning, which means it is potential to improve clustering ability without label information. Given that most works appreciate the output feature of embedding layer rather than projection layer, we first propose an assumption that the performance of clustering on the embedding layer is better than that on the projection layer. Further, to guarantee clustering more stably, we think it is necessary to commit a warm-up operation and update the cluster centers periodically in training phase. Detailed discussions of the above ideas are as follows:

In OOD detection, the score function is finally used to identify whether a sample is an OOD sample. The commonly using Mahalanobis distance as the score function is not suitable when the number of training samples is small. The another common score function proposed in CSI Tack et al. (2020) uses L2-norm and cosine similarity, formulated as follows:

Based on the score function ${score_{cos}}$, we design a novel score function by combining cosine similarity and ID sample variance, described as follows:

where the denominator is the standard deviation of ID set $V$. $V\equiv\{{{\tilde{x}}_{v}}\in\{{{\tilde{x}}_{m}}\},v=1,...,K\}$ is the first $K$ ID samples with highest score $score_{cos}$. ${\bar{Z}({\tilde{x}}_{v})}$ is the sample mean.

The main experiment results are shown in Table 1. We can make the following observations from it:

• •

  Compared with SimCLR that only considers instance-level representation, CCL outperforms on all OOD datasets, which indicates that it is beneficial to take semantic information into account for unsupervised OOD detection and our framework could take advantage of semantic information to improve representation ability of the backbone encoder.

• •

  Among the methods, CCL achieves the best performance on four OOD datasets including LSUN(98.5%), LSUN-Fix(97.8%), ImageNet-Fix(96.0%) and Interp(87.2%). Specially, CCL with $score_{var}$ surpasses the CSI in Tack et al. (2020) by 7.9% on Interp, by 7.5% on LSUN-Fix, by 2.7% on ImageNet-Fix and by 2% on CIFAR-100. On the other two datasets, SVHN and ImageNet, CCL is slightly inferior to CSI. Although CCL does not perform best on all datasets, it shows greater robustness. Compare to CSI that needs designed specific OOD augmentation approaches, CCL is more general and can perform unsupervised OOD detection on more types of datasets.

• •

  From the comparison of scoring functions, it can be seen that the designed variance scoring function can improve performance to a certain extent, which is helpful for unsupervised OOD detection.

In this section, we carry out some targeted ablation experiments to verify the effectiveness of our framework and the rationality of our assumptions in Section 3, including four parts as follows: