Industrial Anomaly Detection with Domain Shift: A Real-world Dataset and Masked Multi-scale Reconstruction

The success of current deep learning methods heavily relies on the diversity of data distribution. This diversity encompasses a large number of data categories, such as ImageNet [34], as well as different domains of the same data category, such as NICO [50] and OOD-CV [51]. Evaluating the performance of target algorithms on datasets with different domains is a crucial step in applying them in real-world scenarios. Despite the availability of several datasets containing various industrial products for the task of IAD, like MVTec AD [4], DAGM [40], KSDD [38] and SensumSODF [31], to the best of our knowledge, all of them overlook the presence of domain shift between the training and test sets, which exists in crack segmentation [25], railway track maintenance using robotic vision [32], surface defect detection [37] and etc. Addressing this issue and considering domain shift is vital for developing more robust and effective IAD algorithms that can be reliably applied in practical scenarios.

Bergmann et al. [4] proposed the MVTec Anomaly Detection dataset (MVTec AD), which focuses on detecting subtle changes and larger structural defects. The main feature of MVTec AD is that all objects are roughly aligned, as shown in Figure 1. Zou et al. [52] proposed the Visual Anomaly (VisA) Dataset, which considers more complex structures and presents multiple objects in a single image. The recent proposed MVTec Logical Constraints Anomaly Detection (MVTec LOCO AD) dataset [3] aims to detect logical anomalies in addition to structural anomalies. Additionally, Bergmann et al. [6] introduced a 3D dataset for IAD.

However, all existing datasets ignore the presence of domain shift between the training and test sets. In this paper, inspired by the real-world application of aero-engine blade anomaly detection, we propose a novel dataset that includes domain shifts between the training and test sets. These domain shifts encompass different views and illuminations, and our proposed dataset can address some of the deficiencies of current datasets.

While there are different types of setups in IAD, such as normal data mixed with few noisy data (anomalies) [41], this paper focuses on practical applications and reviews only one-class learning in IAD, where only normal data is available in training. We categorize these methods into three categories based on the formulation of anomaly detection.

The first category formulates anomaly detection as a matching problem. These methods establish a pattern based on normal data and match the pattern of test data with the established pattern during testing. For example, PaDiM [10] establishes the distribution of every patch of normal data and calculates the likelihood of test data to detect anomalies. Similarly, FastFlow [45] and CFLOW-AD [14] model the normalizing flows of normal data. PatchCore [33] builds a “database” of multi-scale features and matches the test data through k-nearest neighbor search. MKDAD [35] and ReverseKD [12] match the general pattern generated by the teacher network with the pattern of normal data generated by the student network to detect anomalies.

The second category hopes the model can generalize the ability that distinguishes between normal and synthetic abnormalities to distinguish unseen abnormalities. The main difference among these methods is the construction of synthetic anomalies. For instance, CutPaste [21] crops the parts of a certain object into different objects to synthesize anomalies. NSA [36] uses Poisson image editing to seamlessly clone an object from one image into another image, which can significantly reduce discontinuities. DRAEM [47] generates anomalies based on the mask generated by perlin noise and an additional dataset.

The third category includes methods based on reconstruction. These methods [7, 11, 27] rely on the hypothesis that reconstruction models trained on normal samples only succeed in normal regions but fail in anomalous regions [44]. The main problem with this idea is that the model could learn tricks that anomalies are also restored well. To tackle this problem, different strategies are carried out. RIAD [48] and InTra [30] propose a masked image inpainting pretext task. Since some parts of the image are masked out, the model needs to perceive the causality in the image to recover the information of masked parts, which avoids the trivial solution. UniAD [44] proposes a neighbor masked encoder and a layer-wise query decoder to prevent the ”identical shortcut”.

The methods most related to our work are RIAD [48] and InTra [30]. The main differences are the following. First, we neglect to reconstruct intricate details in the full image and instead reconstruct features at different scales. This empowers the output of the model with more semantic information. Second, our masked strategy is different, where the masked parts are completely hidden, resulting in the complete removal of features in the masked parts, while in the previous methods, these features are shared with the features of the visible parts.

The aero-engine is the heart of an airplane, and its healthy blades are critical for ensuring stable operation. Defects in aero-engine blades can significantly impact the engine’s efficiency or even cause it to fail [22]. Due to their high cost and infrequent occurrence, obtaining a variety of blades with different defects (anomalies) can be difficult. As a result, blade defect detection is often approached as a one-class anomaly detection problem. In real-world blade detection scenarios, images of blades acquired by borehole instruments can vary in view and illumination, depending on the location of the borehole on the aero-engine. Since the aero-engine to be detected is unknown, the view is also unknown. This motivates us to explore one-class IAD under the domain shift of test set. For meeting the need of the real-world application, the blades in the experiment are either from the real engine or a 1:1 replica.

To better serve the needs of blade anomaly detection in real-world scenarios and bridge the gap between current datasets and practical applications, we introduce the Aero-engine Blade Anomaly Detection (AeBAD) dataset, consisting of two sub-datasets: AeBAD-S and AeBAD-V. AeBAD-S contains images of single blades, while AeBAD-V consists of videos of multiple blades on blisks of aero-engines. The training set of AeBAD only contains normal samples, while the test sets include both normal and abnormal samples. The anomalies of the blade consist of the following four types of defects: breakdown, ablation, groove, and fracture. Breakdown is caused by foreign objects (e.g., sand, metal, birds, hail, etc.), which usually cause the blade to break [43]. Ablation comes from the high temperature gas that the blades are exposed to. Ablation will reduce the performance of the blade. When the ablation reaches a certain level and the performance of the blade material cannot meet the requirements, fracture will occur. Groove is caused by a large number of elastic stress cycles, which will expand causing the fracture of the blade. Some abnormal samples are shown in Figure 2. The statistical overview of AeBAD is listed in Table 1. The total training samples are 1228 and the test samples are 4342. In addition, for all anomalies present in AeBAD-S dataset, we provide pixel-level ground-truth annotations. And for all frames in AeBAD-V dataset, we provide sample-level annotations.

Different from the current datasets, where all images are roughly aligned, the shooting positions of acquired images in the training set of AeBAD-S are sampled from the position of blue dotted lines, which are enclosed by the negative half axis of x, the positive half axis of y, and the positive axis of z. The shooting positions are shown in Figure 3 top. Besides, there are three background colors for the training image: black, gray and blue. Some training samples are shown in the first row of Figure 2. The test set of AeBAD-S is divided into four categories: same, background, illumination, and view. In the ”same” category, the distribution of the normal samples in the test set is the same as that in the training set. The ”background” category involves changes to the image background, while the ”illumination” category involves changes to the lighting conditions. Finally, the ”view” category involves changes to the shooting positions, with the new positions being symmetrical to those in the training set. The shooting positions of the “view” test set are shown in Figure 3 bottom. In the above test sets, there is a domain shift in all except the “same” test set. The samples of different test sets are shown in Figure 2.

AeBAD-V contains videos taken from different viewpoints, and the training set consists of videos taken from four viewpoints, while the test set has videos taken from three viewpoints different from those of the training set. Some frames are shown in Figure 4. The environment in AeBAD-V is more consistent with the real anomaly detection of aero-engine blades compared to AeBAD-S, which is in a simulated environment.

The proposed AeBAD dataset enables us to evaluate the performance of current IAD methods under domain shifts. We evaluate the performance of four SOTA methods on the AeBAD-S dataset: PatchCore [33], ReverseDistillation [12], DRAEM [47], and NSA [36]. It should be noted that in these methods, the anomaly score of an image is the maximum value of the heatmap generated by that image.

The qualitative results of the above methods are shown in Figure 5, and more results are available in Figure 9. In the qualitative results, the red areas are regarded as abnormal areas, and other colors as normal areas. We find the following problems: 1.) the methods based on synthetic abnormalities (e.g., NSA) tend to miss defects and produce false negative areas. For DRAEM, the cause of failure is more complicated, we will analyze it in Section 5. 2.) the methods based on matching, such as PatchCore and ReverseDistillation, tend to incorrectly identify normal regions as abnormal regions under domain shifts, resulting in many false positives.

Regarding problem 1, the reason is that the synthetic anomalies are mostly distributed in background areas as the samples are not aligned and at different scales. This significantly reduces the difficulty of the task of discriminating normal samples and synthetic anomalies, resulting in poor generalization for unseen anomalies. As for problem 2, existing methods heavily rely on modeling the distribution of normal samples in the training set, which creates a discrepancy between the features of normal samples in the training and test sets when the domian shifts. We think that the problem 2 is likely due to a lack of consideration for causality in normal samples.

MMR consists of two parts. In the first part, as shown in Figure 6 (a) left, the image is evenly divided into several patches (16 for Figure 6), and a certain percentage of these patches are masked while others are preserved, where the patches have equal height and width. The preserved patches are fed into a vision transformer (ViT) [13], which generates embeddings of these patches. The random initialized embeddings of the masked patches are then appended to the embeddings of the preserved patches. A simple feature pyramid network (simple FPN) [23] is used to take the embeddings of all patches as input and generate multi-scale features. In the second part, as shown in Figure 6 (a) right, a frozen pre-trained hierarchical encoder is used to encode the same input used in the first part into multi-scale features. The goal of MMR is to align the multi-scale features produced by the two parts. The underlying motivation for recovering the information on the masked patches is that we hope the model can perceive the dependence of spatial positions of different parts in images. In MMR, the reconstruction task is responsible for focusing on the details, while the masked recovery task is for the causality of various parts.

In the test stage, MMR takes the original non-masked input (overall image) to the above two parts. The first part and the second part generate the features $\textbf{z}_{masked_{i}}$ and $\textbf{z}_{frozen_{i}}$ respectively. The anomaly map $\rm{AM}_{i}\in h_{i}\times w_{i}$ of every scale $i$ is calculated as follows:

During testing, the original non-masked input is used for both parts of the method. Since the frozen pre-trained hierarchical encoder contains the rich semantics, it can perceive the unseen anomalies. And for ViT and simple FPN, since they only model causality in normal sample, it cannot reconstruct the abnormal parts well. This leads to biases in the reconstructed and encoded multi-scale features for anomalies, where the biases correspond to the abnormal parts in the feature map.

The input image, augmented with random cropping, is resized to $256\times 256$ and then center croped to $224\times 224$. The resolution of patch is (16, 16). The masking ratio $\eta$ is 0.4. The backbone of ViT is the vanilla ViT-B [13], where the backbone is pre-trained by masked autoencoder [15]. The simple FPN generates feature maps of 4, 8, and 16 receptive fields. We use a widely adopted WideResNet50 [46, 16] as the frozen pre-trained hierarchical encoder and then extract the multi-scale features from the layer 1, 2 and 3. We use AdamW optimizer [26] ($\beta_{1}=0.9,\beta_{2}=0.95$) with step-wise learning rate decay. The learning rate is 0.001. For all datasets, we set training epoch as 200 and batch size as 16. In addition, all experiments were conducted on Ubuntu 18.04.5 and a computer equipped with Xeon(R) Gold 6140R [email protected] and an NVIDIA GeForce RTX 2080 Ti with 11GB of memory.

We evaluate MMR on AeBAD dataset including AeBAD-S and AeBAD-V. In addition, we also evaluate MMR on MVTec dataset to show the ability to detect different types of anomalies. The comparison methods include the recent SOTA methods and the methods most related to our work, including PatchCore [33], ReverseDistillation [12], DRAEM [47], NSA [36], RIAD [48] and InTra [30]. For PatchCore, ReverseDistillation, DRAEM and NSA, we use the official codes, where the backbone of PatchCore and ReverseDistillation is a pre-trained WideResNet50. We tune the hyperparameters to make sure the optimal performance. For RIAD and InTra, since they do not provide the official code, we use the unofficial re-implementations of these methods ³³3The unofficial re-implementations of RIAD and InTra are here and here respectively..

The quantitative results are shown in Table 2 and 3. The qualitative results are shown in Figure 9. We can observe the following phenomena: 1.) Except for DRAEM, all methods show a significant drop in performance on the ”view” subset compared to other subsets. This is mainly because changes in view increase false positives for normal samples, as shown in Figure 5 and 9. In the third row of abnormal samples of Figure 9, MMR also produces a high score for the normal region. 2.) The methods based on the pre-trained model, such as PatchCore, ReverseDistillation and MMR, are robust to the change for background and illumination. In addition, compared to PatchCore and ReverseDistillation, our proposed method can reduce the anomaly score of the normal region as the domain shifts, as shown in the normal part of Figure 9. Note that the anomaly score of an image is the maximum value of the heatmap generated by that image. PatchCore and ReverseDistillation produce more false positive areas on the ”view” datasets, which is also the reason their AUROC performance on the ”view” dataset is poor, although their PRO performance is close to MMR’s. 3.) NSA show a poor performance for anomaly classification and localization and show a large variance compared to other methods. This is mainly because the proposed dataset is not aligned, and the augmented synthetic anomalies can significantly affect performance. When the synthetic anomalies are distributed less on the background region, the performance is good, and vice versa. 4.) For DRAEM, since it generates the synthetic anomalies distributed at the full image, it can detect some apparent anomalies (e.g., the second row of the anomaly part in Figure 5). However, when the light and background change, it regards the full blade as an anomaly (e.g., the first and fourth row of abnormal samples in Figure 5 and 9). We think that the main reason for this is that DREAM overfits the data due to its two large models, which results in a poor generalization for data with domain shift. 5.) RIAD and InTra use pixel-level reconstruction, which leads to larger reconstruction errors for complex details.

In addition, although the input of the first part of MMR is a whole image in the test stage, the output of this part is not identical with the output of the frozen pre-trained hierarchical encoder. This can be observed from Figure 5 and Figure 9, where MMR can detect some tiny defects (anomalies). This result demonstrates that the first part of MMR can avoid an “identical shortcut”.

The quantitative results are shown in Table 4. The qualitative results are shown in Figure 10. Similar to the previous results, we observe that when the viewpoint changes, particularly for Video 2 and Video 3, PatchCore and ReverseDistillation produce more false positives, whereas DRAEM and NSA produce more false negatives. Our proposed method, MMR, is able to reduce the anomaly score of the normal region and detect multiple abnormal regions. It is worth noting that compared to AeBAD-S, the objects in AeBAD-V are more distributed in the center area. This allows NSA to locate apparent anomalies in some cases, but DRAEM still fails to predict anomalies. We believe that the main reason for this is also from the overfitting in DREAM. For the result of DRAEM on Video 1, since there is a similar view between video 1 and the training set, DRAEM is able to obtain a good result. In addition to the above results, we also present the qualitative results of the consecutive frames (videos) at https://github.com/zhangzilongc/MMR.

The quantitative results are shown in Table 5 and 6. We can observe that MMR also obtain the competitive results with SOTA methods. It demonstrates the detection ability of MMR for the anomalies of different types. It is worth noting that compared with the superior performance in AeBAD dataset, the performance of MMR in MVTec dataset is suboptimal. The main reason is that when the distribution of the test sample changes, MMR can better judge normal samples and abnormal samples, while other methods will be affected by domain shift. However, for the MVTec dataset, the distribution of test samples is consistent with the training set, so the advantages of MMR cannot be highlighted. In addition, we observe that compared with PatchCore, the acquired abnormal masks of MMR are coarser, which is also the reason for the suboptimal performance in the pixel-level performance.

In MMR, we take non-masked patches as input. In this part, we select the entire image (all patches) as input while maintaining a constant masking ratio. We regard $q\times q$ pixels as a unit and then randomly mask $\eta=0.4$ times the area of the entire image based on this unit. We choose $q=[1,2,4,8,16,32,56,112]$. An example of masked images of different $q$ is shown in Figure 11. Note that when $q=16$, the information included in the non-masked input is the same as the original input of MMR. The only difference is that the information in the non-masked part is shared, which is shown in Figure 6 (c). The results of different inputs are shown in Figure 12. We can observe that when the masking ratio is constant, the input with different masking forms can have a significant impact on performance. When $q\leqslant 8$, the performance is poor. The reason is that although the information of the masked patches is unknown, the adjacent region will leak information due to the small unit. This makes the model lose the ability to infer the information in the masked patches from the global context. When $q=16$, the performance improves due to the larger unit. However, since this still cannot prevent the leakage of information, the performance at $q=16$ is lower than the proposed MMR at $q=16$. In addition, we find an interesting phenomenon: as the smallest unit continues to increase, performance will continue to improve. We believe that the main reason is that when the smallest unit becomes larger, the association between image blocks will become weaker and weaker, even if the information leaks, it will not affect the overall reconstruction. However, the details of some reconstructions will become worse, which leads to a large increase in its mean PRO and basically no growth in mean AUROC. This is similar to 0.9 mask ratio in Figure 13.

We conduct experiments with different masking ratios in order to explore how they affect the model’s ability to perceive causal relationships. Figure 13 displays the results of this analysis for masking ratios ranging from 0 to 0.9. Our findings suggest that the model’s performance exhibits a pattern of low scores at both ends of the masking ratio, with the highest scores achieved in the middle. When the masking ratio is small, the model is able to make accurate inferences about the masked patches based on the surrounding regions that contain a lot of relevant information. However, as the masking ratio is high, the model becomes less able to infer the whole image based on just a few visible patches, resulting in lower performance scores. Note that different from the recent masked autoencoder [15], where the optimal masking ratio is high and the target is to perceive the semantic, the model for IAD needs to discriminate some tiny defects, and thus requires a smaller optimal masking ratio to reconstruct the details accurately.

In our implementation, the ViT backbone is pre-trained on ImageNet by a masked autoencoder [15]. In this part, we show the result that ViT is trained from scratch in Table 7. Intuitively, training ViT from scratch using normal samples should result in a model that is better able to identify unseen anomalies. However, we find that this is not the case. The main reason for this is that training ViT on a small dataset from scratch can lead to overfitting, which in turn reduces the generalization ability of the model. In addition, another advantage of choosing ViT pre-trained by a masked autoencoder is that the pre-trained upstream tasks are roughly the same as our proposed downstream task, both of which aim to reconstruct the original input. This similarity allows the model to transfer its learned representations more effectively, resulting in better performance on our task.

In this part, we choose different frozen pre-trained hierarchical encoders to train MMR. The results are shown in Table 8. We can observe that MMR can also generalize well to ResNet 18.

In this part, we report the effect of features at different scales on performance. The results are shown in Table 9. We can observe that the features at the intermediate scale are more conducive to detecting defects, while the features at the other scales can be a complement to improve performance.