Dual-Image Enhanced CLIP for Zero-Shot Anomaly Detection

Advancements in data scale have led to significant strides in pretrained visual language models [15, 6, 34, 3], which demonstrate remarkable proficiency in a variety of downstream tasks [47, 46, 23, 24, 33, 31, 41]. A prime example is the CLIP model, through contrastive vision-language pre-training on a diverse array of internet-sourced image-text pairs, it exhibits exceptional generality and adaptability. This model is particularly adept at zero-shot inference, displaying a superior capacity for recognizing images beyond its training data. Recent explorations have extended the zero-shot capabilities of CLIP models to tasks like open-vocabulary semantic segmentation, achieved by harnessing intrinsic dense features [45, 16, 30]. Additionally, efforts to optimize CLIP’s recognition performance have been fruitful, focusing on areas such as prompt engineering [47, 46], adapter modules [44, 18], and additional training for enhanced vision-language alignment [23, 24]. Importantly, CLIP’s inherent ability to detect out-of-distribution data without additional training has catalyzed its application in zero-shot anomaly classification and localization.

The essence of most existing AD methods lies in modeling the distribution of normal samples to detect anomalies [37, 36, 10, 43]. Particularly, reconstruction-based approaches, such as those using autoencoders [1, 2, 19] and GANs [37], have been widely adopted. These methods leverage the reconstruction capabilities of anomaly-free images, hypothesizing that abnormalities will manifest as discrepancies in the reconstructed output. Additionally, discriminative models have been developed, often with the aid of synthetic anomalies [43, 28]. Moreover, large pretrained models like ImageNet [14] have been found to be robust in anomaly detection, by comparing features embedded from normal images [9, 35, 8]. To bridge the gap between natural image pretraining and industrial application, further adaptations have been employed, such as student-teacher knowledge distillation [36, 10] and normalizing flows [26, 20].

A significant challenge in this field is the development of unified models capable of detecting anomalies across multiple classes [42, 38, 11]. In the context of zero-shot settings, MuSc [29] innovatively utilizes unlabelled images from the test set as references. Alternatively, the WinCLIP model [22] utilizes CLIP [34] for prompt-guided AD. Building on this, AnoCLIP [12] enhances localization representation and implements V-V attention introduced in [30]. However, vision-language models such as CLIP are primarily trained to align with the class semantics of foreground objects rather than the abnormality/normality in the images, and as a result, their generalization in understanding the visual anomalies is restricted, leading to weak ZSAD performance. Existing zero-shot prompt-guided AD models often lack robust visual representation as a basis for detecting anomalies. Addressing this, methods such as [5, 48] propose fine-tuning the pretrained CLIP model with auxiliary images for cross-set training/validation. In light of the limitations inherent in current CLIP-based anomaly detection models and the essential role of visual references, we propose an innovative, training-free method that inference on pairs of images. In this approach each image acts as a visual reference for the other, significantly enhancing the accuracy and effectiveness of anomaly detection.

CLIP’s zero-shot visual recognition, trained on a multi-million image-text pair dataset, aligns images with textual descriptions through a visual encoder and a text encoder. These encoders respectively transform images and text prompts (e.g., a photo of a [class]) into visual and text tokens in a shared feature space. The model’s ability to compare these tokens via cosine similarity allows it to identify class concepts within images.

For anomaly detection, CLIP utilizes semantic concepts of “normal” and “anomalous” states. Multiple prompts with varied descriptors (like “perfect”, “broken”, etc.) are used to create averaged text tokens representing these states, $t_{n}$ and $t_{a}$ for normal and anomalous text tokens, respectively. Anomaly score for an image is computed based on the similarity between its visual token and these averaged text tokens. Specifically, given a text prompt and the corresponding class token $v$, the sample-level anomaly score $A^{L}_{cls}$ is computed as:

where $\tau$ is the temperature hyperparameter. Note that no visual information is injected into the model, but rather unknown anomalies are detected through the powerful open-world generalization of CLIP.

The computation is extended from global visual embeddings to patch-level visual embeddings to derive the corresponding segmentation maps $A^{L}_{loc}\in\mathbb{R}^{H\times W}$, the final layer of the visual encoder has a set of patch tokens $p_{(j,k)}\in\mathbb{R}$ that potentially contain image local information in the patch level. For a patch token $p_{(j,k)}$, the local anomaly score is computed as:

However, since CLIP was primarily trained to align the class tokens with the text token for global classification, there’s a lack of alignment between local patch tokens and text embeddings that leads to limited performance in segmenting anomalous regions. Hence, after iterative explorations [30, 12, 48], V-V attention was adopted to produce the local-aware patch tokens.

In original Q-K-V attention, the attention score can be disproportionately influenced by specific tokens, leading to a representation that is disturbed by unrelated local features, which can weaken the model’s localization ability to detect anomalies. The V-V attention mechanism is proposed as an alternative that enhances the local features without additional training. This novel attention mechanism replaces the queries and keys with values.

By focusing on self-attention within the values themselves, V-V attention avoids bias introduced by the query and key interactions in Q-K-V attention. It reduces the disturbance caused by other tokens, ensuring that each value contributes significantly to its own representation. As a result, attention maps produced by V-V attention exhibit a pronounced diagonal pattern, indicating that each token predominantly attends to itself, thereby preserving its local information.

In our model, the architecture remains the same with AnoCLIP [12], which follows a 2-way forward path. The original Q-K-V attention path was kept to produce the class token, which was used to calculate sample level anomaly score $A^{L}_{cls}$. The patch tokens used for localization score $A^{L}_{loc}$ are all computed by the V-V attention path.

As shown in Fig. 2, we proposed a novel approach that inputs a pair of images in test-time. Unlike previous CLIP-based AD works [22, 12, 48, 39] which predominantly rely on text prompts for inference, we incorporate additional visual information to facilitate a more comprehensive joint vision-language prediction. To highlight the effectiveness of our approach, we provide a comparative analysis in Fig. 3.

Fig. 3 demonstrates a significant observation: the exclusive dependence on either textual or visual information alone proves inadequate for the accurate detection of certain anomalies. The limitation in leveraging text stems from the constraints inherent in utilizing state descriptions within prompts, with terms like “broken” or “damaged” falling short of encapsulating the full spectrum of potential anomalies. Making inferences on a single image invites biases and misinterpretations, emphasizing a more extensive visual intra-class diversity to form a baseline for normalcy. For instance, consider the “PCB” example in column 3 of Fig. 3, where a misplaced LED is an anomaly; however, using text description alone is insufficient for its detection. Moreover, logical anomalies, global distortions, rare objects, or complicated scenes are more challenging to discern from by text-based method. This emphasizes the importance of incorporating additional visual context for a varied example to enhance the detection accuracy. Conversely, relying solely on pairwise visual comparisons also presents limitations, as the reference image could itself be anomalous. Consequently, this paves the way for an integrated approach that combines textual and visual data to overcome these challenges. As depicted in Fig. 3, employing dual-image inputs within the CLIP-based framework mitigates these issues, contributing to improved anomaly localization accuracy.

In response to these findings, our framework introduces a novel strategy that capitalizes on both textual and visual features. This is achieved by a unique process of randomly selecting pairs of test images to serve as the query and reference images. For these image pairs, we extract patch tokens, denoted as $q_{(j,k)}$ for the query and $r_{(m,n)}$ for the reference. These patch tokens form the basis for the pairwise visual feature comparison.

In this pairwise feature comparison strategy, each patch token $q_{(j,k)}$ from the query image undergoes a nearest neighbour search with the patch tokens from the reference image, effectively using the latter as a memory repository. The anomaly score for each query patch token $q_{(j,k)}$ is determined by calculating its cosine similarity with all reference patch tokens set $\mathcal{S}_{r}=\{r_{(m,n)}\mid m\in\{1,\ldots,H\},n\in\{1,\ldots,W\}\}$. The maximum similarity score, indicating the minimum deviation, is then designated as the anomaly score for the query patch:

In the equation above, $A^{V}_{(j,k)}$ represents the visual reference anomaly score of the query patch $q_{(j,k)}$, the overall anomaly score $A^{V}\in\mathbb{R}^{H\times W}$ for the query image, is a composition of all the patch scores across the entire image. sim represents the cosine similarity between the patch tokens of the two samples.

As a result, the vision-language joint anomaly score can be computed as:

As the visual-language alignment needed to be refined for AD, we proposed a test-time adaptation module to boost the CLIP-based model’s AD capabilities. Our TTA module is achieved through a linear adapter, as depicted in Fig. 4. For the original image, we utilize the pseudo anomaly synthesis technique from DRAEM [43] to introduce image corruptions. DRAEM creates random-shaped pseudo anomaly masks using Perlin noise [32] and overlays textures from [7] onto the original image at masked locations. The resultant pseudo-anomalous patch tokens, denoted as $q^{\prime}_{(j,k)}\in\mathbb{R}$, encapsulate pseudo-anomalous features.

The online adaptation of the original and synthesized patch tokens is mathematically represented as:

Here $G(\cdot)$ denotes the linear computation. Subsequently, these adapted patch tokens are aligned with text tokens to compute the anomaly score:

To optimize the weights of the linear layer, we establish self-supervised tasks using pseudo anomalies. For the original and adapted patch tokens $q^{T}_{(j,k)}$, $q^{\prime T}_{(j,k)}$ from queries, we design two discriminative self-supervised tasks for TTA:

(1) For predicting pseudo anomaly masks $M_{a}$, we define the $L_{\text{pseudo}}$ loss as:

Here, $\mathcal{S}_{a}$ represents the set of indices $(j,k)$ where $M_{a}\neq 0$, indicating regions augmented by the pseudo mask $M_{a}$. $L_{\text{pseudo}}$ prompts the adapter to retain abnormal features and recognize pseudo anomalies, aiding in the subtle detection of real anomalies.

(2) To encourage the adapter to preserve normal features and uphold general anomaly detection capabilities, we utilize the similarity loss $L_{\text{sim}}$ to ensure that adapted anomaly scores $A^{T}$ are consistent with the zero-shot vision-language joint localization:

The aggregate learning objective to train our adapter is $L=L_{\text{pseudo}}+\beta L_{\text{sim}}$. This TTA process is efficient and does not require any training data or annotation. Finally, the overall anomaly classification and localization score for the query image should be computed as:

Tab. 1 presents the performance of zero-shot anomaly detection on MVTecAD and VisA datasets. Our proposed method is compared with prior ZSAD based works, including CLIP [34], WinCLIP [22], AnoCLIP [12], and MuSc [29]. Notely, MuSc utilized the entire test set for visual reference, aligning with transductive rather than inductive learning. For a fair comparison, we adapted MuSc to our pairwise image setting and used ViT-B-16+ as the backbone, denoted as MuSc-2. From the table, our proposed methods exhibit exceptional performance, significantly outperforming AnoCLIP by margins of $2.2\%,6.0\%,6.2\%$ in AUROC, F1Max, and PRO for anomaly localization. We also achieve advancements in anomaly classification, surpassing other methods by substantial margins. This trend of exceptional performance is consistent on the VisA dataset. Qualitative results for ZSAD are further detailed in Fig. 3, illustrating our model’s capacity to effectively classify and localize the anomalies across varied samples.

Additionally, Tab. 2 presents a comparison of our method against other AD models by AUROC scores on MVTecAD. Here, we categorize current zero-shot anomaly detection methodologies into three paradigms: Auxiliary Data approaches that utilize additional anomalous data for training, Transductive Learning Methods that infer from the extensive portion of the test set, and Inductive Learning Approaches that assess each query independently, without prior knowledge of the overall distribution. Our method, an exemplar of the inductive approach, outperforms Auxiliary Data Approaches without requiring exposure to anomalies during training.

Contrasting our method with MuSc, the state-of-the-art transductive learning setting of ZSAD, MuSc requires accessing the entire test set distribution before inference, making it highly dependent on the data distribution, and potentially limiting in real-world applications like online real-time inference. In Tab. 1, our method surpasses MuSc’s performance on the pair-image setting, especially in anomaly classification. This derives advantages from the utilization of the class token in CLIP embeddings, and emphasizes the efficiency and robustness of our joint language-vision prediction.

From Fig. 5(a), a clear trend is observed: with the increasing number of reference images, the anomaly detection performance improves, reflected from both pixel-level and sample-level AUROC. This supports the hypothesis that unlabelled images can still offer valuable comparative references for anomaly identification.

Significantly, the most pronounced performance leap occurs when the number of reference images is increased from $0$ to $1$, indicating that even a single reference image can substantially improve the model’s AD ability. However, the subsequent performance gains from $1$ to $7$ reference images are present but exhibit diminishing returns. This trend implies that considering the factors of inference time and memory efficiency, utilizing a large pool of reference images might not always be feasible or optimal, particularly in real-world applications where access to a wide array of suitable samples is often limited. In this context, a pairwise approach emerges as a balanced solution, optimizing the trade-off between improved detection performance and computational resource efficiency. Moreover, the observed differences in the degree of improvement between pixel AUROC and sample AUROC point that while visual details are paramount in pinpointing anomalies, they may be less influential in the broader context of classifying an entire sample as anomalous.

Since the reference images are randomly sampled from unlabelled images, they can either be normal or anomalous. This leads to the question: how does an anomalous reference image affect the precision of anomaly detection?

Fig. 5(b) illustrates how different reference images can significantly influence the anomaly score. In the case of the “bottle”, it is evident that using a normal reference image generally guarantees the accuracy of anomaly detection, as it provides a clear baseline for identifying outliers. Conversely, when the reference image contains anomalies, such as breaks or contamination, these imperfections can misleadingly provide a false reference for the query image, erroneously highlighting the reference image’s damaged region in the query. This phenomenon suggests that the abnormal condition of the reference image can “pollute” the anomaly score of the query image.

Interestingly, integrating textual cues with visual data can mitigate this negative effect. By leveraging textual features from prompts, the model can effectively counter the false prediction associated with the anomalous references. As depicted in Fig. 5(b), the joint predictions that combine both visual and language information exhibit a notable increase in accuracy, underscoring the potential of language-vision joint anomaly detection.

We also studied the impact of various training steps on model performance, Fig. 6(c) demonstrates the AUROC and PRO for training steps ranging from 1 to 6. We can see both the pixel and sample AUROC and PRO scores reach the optimal when the training step is set to 2, and start to decrease. Therefore, we opted for a $\text{training step}=2$. In Tab. 3, we showcase the performance enhancements by the TTA module. Here we take the text-only approach as the baseline. The table shows that implementing the TTA module on the baseline yields a notable increase in performance. When the TTA module operates alongside a paired reference image, the results are further amplified. In this scenario, the AUROC for AL climbs to $92.8\%$, and AC reaches $93.2\%$. Further, the improvement in the F1Max and PRO indicates a more balanced and effective model, particularly in terms of its localization capabilities. The influence of the TTA module is further presented in Fig. 6(a), where a marked distinction in anomaly scores between normal and anomalous patches is observed post-adaptation. Prior to adaptation, the “missing cable” region was not adequately identified, and the adaptation process leads to a refined alignment between visual perception and language context, resulting in superior AL and AC performance.

Our approach, while robust in many scenarios, is not without its limitations. One notable constraint is the requirement of inputting two images during inference, which may not be feasible with certain scenarios where single-image processing is crucial. Despite this, our method still demonstrates a significant performance enhancement in most cases.

Moreover, while our framework achieves SOTA performance in the zero-shot inductive learning setting, it reveals a gap when compared to SOTA models trained under full-shot regimes and zero-shot transductive learning approaches. As Tab. 2 shows, our method outperforms many existing models in unified AC and AL. However, it falls short of the benchmarks set by UniAD [42] and MuSc [29], particularly in scenarios where MuSc excels using visual features alone. This discrepancy suggests that there is substantial untapped potential for further exploration of visual reference features.

Additionally, our study offers novel insights into the application of the CLIP model for fine-grained anomaly detection: We demonstrate that joint visual and textual discrimination is a key contributor to enhancing fine-grained anomaly localization capabilities within the CLIP framework. Our findings also indicate that even when the visual reference images are anomalous, they can still serve as references for accurate anomaly scoring. These insights not only affirm the effectiveness of our proposed method but also open avenues for future research in refining visual-language models for more precise and versatile anomaly detection tasks.

In this study, we introduced an innovative framework, the Dual-Image Enhanced CLIP for anomaly classification and localization, in the realm of zero-shot learning. Our approach leverages pairs of unlabelled images utilizes the pseudo anomaly in the TTA module, and demonstrates remarkable enhancement in performance, outperforming several SOTA methods. This advancement was achieved without the need for additional training, showcasing the framework’s practicality and efficiency. Our findings also highlighted the untapped potential in combining textual features and visual references, suggesting room for further exploration in this domain.