SCL-VI: Self-supervised Context Learning for Visual Inspection of Industrial Defects

The feasibility of learning semantics through predicting the relative positions of patches has been well-established [8]. In an effort to enhance the encoder’s capability to capture semantics and improve feature discrimination, we introduce a novel self-supervised learning method, as illustrated in Figure 2.

In our pretext task, two patches, denoted as $p_{1}$ and $p_{2}$, are randomly selected within the 3$\times$3 kernel space in each iteration. The correct relative distribution of these two patches is determined through the training of the encoder and classifier, as illustrated in Figure 2(a). There exist $C_{9}^{2}=36$ cases of absolute position relationships between $p_{1}$ and $p_{2}$. However, it’s noteworthy that one relative position relationship encompasses multiple absolute position relationships. For instance, the relative position relationship is the same for pairs like 1 and 3, 2 and 4, 4 and 6, and 5 and 7. These four absolute positions belong to the same relative position of the diagonal left connection and share similar semantics. Consequently, only twelve distinct relative position relationships are considered among the 36 absolute relationships, as depicted in Figure 2(b). The relative positions are encoded as $y\in\{0,1,\ldots,10,11\}$, and the encoder $E$ and classifier $C$ predict the relative positions of $p_{1}$ and $p_{2}$ when input into the model. The loss for self-supervised learning is defined as follows:

$$\mathcal{L}_{SSL}={CrossEntropy}\left(y,C\left(E\left(p_{1}\right),E\left(p_{2}\right)\right)\right)$$

(1)

Our algorithm deviates from those presented in [32] and [8]. In comparison with Doersch’s method [8], our proposed pretext task captures more semantics and is not confined to determining the relative positions between a specific block and its eight surrounding blocks. During our experiments, to mitigate the classifier’s reliance on shortcuts, such as patch boundary information and color details, we introduce random dithering of 0-16 pixels for each patch and adjust the RGB value by randomly increasing or decreasing the gray value. This ensures that the classification is rooted in semantics rather than shortcuts. As a result, the encoder demonstrates improved feature discrimination ability following training with self-supervised learning.

Combing deep SVDD [18] with a patch-level idea, [32] trained a deep patch SVDD network, which outputs multiple smaller hyperspheres with patchwise data as the input. Each hypersphere contains the features of a specific region. In our approach, we regard patch-SVDD as a basic tool and our baseline.

In the training phase, the function of the encoder in patch SVDD is to wrap the features of the patches in the same position into one hypersphere as much as possible, that is, to minimize the feature distance between the patches. [32] defines the distance loss as follows :

$$\mathcal{L}_{pSVDD}=\sum_{i}^{N}\left\|E\left(p_{i}\right)-E\left(p_{i}^{{}^{\prime}}\right)\right\|_{2},i=1,2,\ldots,N$$

(2)

where $p_{i}$ and $p_{i}^{{}^{\prime}}$ are two extremely similar patches and $p_{i}^{{}^{\prime}}$ is randomly offset by several pixels on the basis of $p_{i}$ during training.

The fusion loss of patch SVDD and self-supervised learning can be expressed as follows [32]:

$$\mathcal{L}=\mathcal{L}_{SSL}+\alpha\times\mathcal{L}_{pSVDD}$$

(3)

where $\alpha$ in the formula adjusts the contributions of the different semantics, and the main loss $\mathcal{L}_{SSL}$ is our new self-supervised learning loss.

In the baseline method (patch SVDD) [32], using only the nearest feature as the normal feature can create a perception that the discovered image feature is not the most suitable due to variations in normal data features, leading to the issue of contingency. Conversely, taking the average of multiple normal features might result in the averaged feature being less similar to the nearest normal feature due to substantial feature differences. The process of obtaining normal features significantly influences the final results.

To address both the problem of contingency and differences in feature comparison, normal features are collectively stored in a structure referred to as a memory item. Assuming the Euclidean distance between the $N$ normal compositional patterns $(p_{1},p_{2},\ldots,p_{N})$ retrieved from the memory module and the latent representation $(p)$ being tested is $d_{1},d_{2},\ldots,d_{N}$ $(d_{i}=\|p_{i}-p\|_{2})$, the affinity $\beta_{i}$ is defined as follows:

$$\beta_{i}=\frac{e^{\gamma_{i}}}{e^{\gamma_{1}}+e^{\gamma_{2}}+\cdots+e^{\gamma_{N}}},\quad i=1,2,\ldots,N$$

(4)

where $\gamma_{i}=\min\left\{\frac{d_{1}+d_{2}+\cdots+d_{N}}{d_{i}},\lambda\right\}$. The threshold $\lambda$ prevents $d_{i}$ from being infinitely small enough such that $e^{\gamma_{i}}$ leads to computation overflow. Therefore, $\lambda$ as a value in the interval [15, 30] is recommended. In this paper, $\lambda$ is set to 20.

$N$ multiple normal compositional patterns(four in Figure LABEL:fig3) are fused to obtain a fusion pattern $p_{fused}$ :

$$p_{fused}=\beta_{1}p_{1}+\beta_{2}p_{2}+\cdots+\beta_{N}p_{N}$$

(5)

After the training phase, the encoder has learned the ability to map normal samples into several hyperspheres. Each hypersphere can represent the basic microstructure of the normal sample. In the test phase, the abnormal score $score(p)$ is obtained by calculating the distance between the latest representation of the test pattern and the fusion pattern. which is an improvement over the baseline, as follows:

$${score}(p)=\left\|E(p)-E\left(p_{fused}\right)\right\|_{2}$$

(6)

A pixel belongs to multiple patches because the sliding stride of the patch is not equal to the width of the patch. Therefore, the anomaly score of each pixel $pixel_{\delta}$ depends on the average value of the abnormality score of the $K$ patches to which it belongs:

$${score}\left({pixel}_{\delta}\right)=\frac{1}{K}\sum_{j}^{K}{score}\left(p_{j}\right),j=1,2,\ldots,K$$

(7)

where $p_{j}$ represents the $K$ patches to which the pixel belongs. Pixels with relatively high scores are considered defects.

To assess the effectiveness of the proposed method, we conduct various sets of experiments in this section. The key hyperparameter $\alpha$ has been explored in [32]; hence, this paper does not delve into discussing $\alpha$. First, we illustrate the boosting self-supervised learning paradigm that parses context information. Second, we examine the impact of the number of memory items associated with affinity. Following this, we compare the overall performance of our method with several state-of-the-art models. Lastly, we deploy the model in a real industrial environment for practical use.

For the outstanding method [32], [8], correct position identification of the scrambled patches is employed. For homogeneous textures, different adjacent relative positions may have similar characteristics; in other words, context information is easy to omit due to the shortcut that connectivity between textures is easier to extract. For object products, the receptive field is limited by the local patch locations; thus, only partial semantic information can be learned. In contrast, the proposed pretext task is innovative not only in randomly selecting two patches in the 3$\times$3 kernel space, eliminating the ”shortcut” problem but also in dividing the relative positions into 12 representative distributions. It exhibits relatively good performance due to the robust representations.

To comprehensively evaluate the effectiveness of the proposed method, its detection and segmentation performance is compared with several state-of-the-art methods, including autoencoders (AE-SSIM [4], MEM-AE [9], and Trust-MAE [23]), generative adversarial networks (F-Aon-GAN [20], AnoGAN [19]), the variational method VAE-GRAD [7], and other superior unsupervised algorithms (Patch SVDD [32], US [3], RIAD [33]). The hyperparameter settings are determined based on ablation experimental results; specifically, $\alpha$ is set to 0.0001, and $\eta=10$ in subsequent experiments.

The detection and segmentation results of these methods on the MVTec AD dataset are presented in Table 2 and Table 3. As observed in Table 2, the proposed method attains the best AUROC performance in detection. Compared to the best existing method, the proposed method improves the detection performance by a margin of 3.74%. The segmentation results in Table 3 demonstrate margins of 1.06% and 2.61% when compared with the best and second-best methods. The proposed method exhibits excellent performance across both textured surfaces and object products, showcasing its superiority, stability, and potential to serve as a unified model for defect inspection in industrial applications.

Some inspection results of our proposed methods on MVTec AD are shown in Figure 4. These results show that the model can accurately inspect various types of defects, both textures and object defects.

To further assess the practical performance of our method, we apply it to a dataset composed of texture and object samples in real industrial environments. This dataset contains two real-world products, coating products (CP) and suede silicon wafers (SSW) and there are about 200 training images and 100 testing images. The texture features of the SSW are complex and irregular, and there are significant similarities between the defects and textures. The anomaly maps are described in Figure 5 in Sec. 6, demonstrating the potential of our method in industrial applications. Moreover, the results of the comparison of different methods (AE-SSIM [4], US [3], RIAD [33],PADIM [6], and SPADE [5]) on our own dataset is shown in Table 4.