Self-supervised learning for hotspot detection and isolation from thermal images

1. 1.

   We are the first to employ a self-supervised algorithm for hotspot detection in thermal images.

2. 2.

   To enhance the performance of our approach, we made modifications to the SimSiam backbone: (i) by replacing it with the XceptionNet model and (ii) by adding a cross-entropy loss term to the loss function. We introduced an intermediate step of image classification in the hotspot detection pipeline by ensembling two encoders from different SimSiam models in order to minimize false positives in day-to-day use of this technique, which helps by filtering out false positives and thereby improving the robustness of the hotspot detection process.

3. 3.

   We combined the contrastive learning method, the classification results of the downstream task, and class-activation maps to come up with a novel thermal image deep-learning hotspot isolation method.

4. 4.

   We created a novel high-quality thermal image dataset for hotspot detection to address the issue of lack of appropriate thermal image datasets for the same.

Both supervised and unsupervised techniques have been proposed for thermal image hotspot detection. K-Means (MacQueen, 1965) clustering has been the most popular technique for hotspot detection. In Mohd et al. (2017) K-Means clustering algorithm was used for segmenting a thermal image into the foreground and background, with the expectation that the hotspot would get separated from the foreground. This way, they were able to limit the amount of image pixels for subsequent processing. However, before the segmentation, the images were converted into the CIE L*a*b* colour space since L*a*b includes all perceivable colours. Mohd et al. conducted tests on a number of images but they have not mentioned the nature of the targets in the images. However, they were successfully able to remove the unassociated background through segmentation, and thereby focused on a smaller region containing the hotspot. However, using this method the area containing the hotspot obtained through segmentation overestimated the area of the hotspot. This was especially the case with the existing technique when multiple hotspots were present in the subject. In Salazar and Macabebe (2016) a K-Means thermal image segmentation technique was also used for finding and isolating hotspots in solar cells, for which they developed a ’Hotspot Detection Program’ . From the examples provided we assume that the algorithm works well on hotspots having a large size. Furthermore, the technique requires a lot of manual input by the user. A fixed $k$ value of 3 was used for the purpose of segmentation. The user needs to specify the temperature range of the uploaded image for the hotspot average temperature function to work. The user is also supposed to manually select a bounding box around the relevant parts of the image.

Afifah et al. used a multilevel Otsu segmentation algorithm for isolation of hotspots from thermal images of photovoltaic cells (Afifah et al., 2021). This work came as an improvement to their earlier work based on simple Otsu segmentation (Afifah et al., 2020). The methodology consisted of taking multiple thresholds for segmenting the image and then utilizing the most significant threshold (largest) to binarize the original image. Afifah et al. utilized morphological opening to get rid of noise in the final binary image. A comparison between the K-means segmentation algorithm and the multilevel Otsu segmentation algorithm for the detection of hotspots in thermal images was done in Lim and Teoh (2022). For both these algorithms, they divided the image into six segments and then calculated the average temperature of each segment to find the one with the hotspot based on which segment had the highest temperature. They found that the Otsu segmentation algorithm provided higher accuracy and had performed faster than the K-means segmentation algorithm. This technique can be considered to be more robust than the one proposed by Afifah et al. (2021) since it relies on calculating the average temperature of each segment after segmentation for detecting the hotspots, rather than converting the image into a binary image based on the largest hotspot.

Alajmi et al. (2019) proposed a hotspot detection technique for photovoltaic cells using thermal images. They converted the thermal image to grayscale and applied edge preservation and smoothening filters. The processed image was then converted to the HSV space, using which a binary mask for each channel got created. The use of specific lower and upper thresholds to obtain the masks for each of the three channels was proposed. It is possible that the specific thresholds proposed in this paper do not work well to isolate the hotspots in all thermal images since the colours displayed in a thermal image is relative to the surrounding temperatures. This means that a particular colour can represent different temperatures in different images and therefore the thresholds might be different for different images.

Ali et al. (2020) proposed a hotspot detection and classification approach for classifying thermal images of PV panels as ’healthy’, ’hotspots’, or ’faulty’ using a hybrid features approach for dataset construction, along with an SVM classifier. The proposed features consisted of HOG, LBP, RGB, contrast, correlation and energy descriptors. Following this, Ali et al. (2022) proposed another PV thermal image classification technique using machine learning methods on colour image descriptors as features. Even though they did not attempt hotspot isolation, they classified images as normal, having hotspots, or as completely defective with high accuracy. Ali et al. carried out experiments with many image colour descriptors like histograms, SIFT methods, and colour moments, and got the highest classification accuracy using the k-Nearest Neighbours method with the rgSIFT descriptor on an image size of 71x71 pixels. Ahmed et al. extracted thermal image features using the SURF method, and utilized the K-Means clustering algorithm for creating a visual vocabulary. Shallow classifiers like SVM, Naive Bayes, k-Nearest Neighbours are then utilized with this vocabulary for classifying images of photovoltaic panels as healthy, hotspot, or faulty (Ahmed et al., 2022). Following this work, Ahmed et al. merged the benefits of pre-trained deep learning models like ResNet18, SqueezeNet, and GoogleNet through transfer learning and of shallow classifiers like SVM, k-Nearest Neighbours, naive Bayes to classify thermal images of photovoltaic panels as healthy or as one of the five different types of defects (Ahmed et al., 2023). In Açikgöz et al. (2022) different deep learning models like ResNet50, AlexNet, SqueezeNet, GoogLeNet, ShuffleNet and MobileNet were compared for classifying thermal images of solar panels as ’Hotspot’ or ’No anomaly’ depending on whether the image had a hotspot or not. They found that AlexNet and MobileNet had the best performance on different metrics. In Song et al. (2023) the Bag-of-Features model(BOF) was integrated with the SURF technique to come up with a Bag-of-SURF-Features (BOSF) vector for each image. They used these features in a SVM for classification of two types of circuit board images, into faults that lead to hotspots in circuit boards. Song et al. showed that the BOSF feature extraction technique performed better than other feature extraction techniques, and that the SVM classifier performed better than a Naive Bayes, or an error back propagation neural network.

From these works it was observed that a lot of the existing hotspot detection techniques are not precise in pinpointing hotspots, require manual inputs at times, and lack robustness and generalizability in their application. Furthermore, most of the existing techniques do not deal with hotspot isolation at all. Therefore, in order to tackle these issues and make the hotspot detection process reliable, we propose a self-supervised learning hotspot detection approach.

A few recent works that utilize self-supervised and semi-supervised learning techniques for classification have been proposed for different applications. In Montanaro et al. (2022), the task of land cover classification through the use of multi-modal image data with the help of self-supervised contextual learning methods was undertaken. The approach involved a preprocessing step that dealt with feature extraction, followed by two steps of self-supervised learning. The first self-supervised step’s goal was to reduce the distance between features from the same classes of different modalities and the goal of the second self-supervised step was to enable learning of pertinent features for facilitating land-cover classification. In Xu et al. (2022), a modified Simple Siamese model for classifying ships using SAR (Synthetic Aperture Radar) images was proposed. They took into account polarized images in addition to regular augmentations as positive samples. Xu et al. also compared different similarity functions and achieved an accuracy of 66% by using this approach. However, self-supervised methods have not been employed for hotspots detection.

This article has been organised as follows: This introduction is followed by a section on the methods (Section 2) used for the development of the proposed approach. This is followed by a section on the experiments and results (Section 3) obtained on different datasets and metrics using the developed approach. Lastly, a brief section on discussion and conclusions (Section 4) that can be derived from the performed experiments and corresponding results is presented as well.

Self-supervision allows networks to learn from unlabelled data. Its beauty lies in its ability to leverage on the immense amount of unlabelled data that is always available but are expensive and time consuming to label. Self-supervised learning overcomes this limitation by solving a proxy or surrogate task that requires just the availability of unlabelled data. With these proxy tasks, the model learns to predict certain aspects of the data without any supervision or annotations. The proxy tasks are designed to provide the model with a different type of supervised learning paradigm in which just the unlabelled data are utilized. Usually, some kind of data-augmentation or transformation of the unlabelled data is undertaken so as to generate pseudo-labels. This can be thought of as withholding or distorting a part of the data. Given this augmented or transformed data, the network is then tasked with reconstructing the original image with the original image or some part of it, thus behaving like pseudo-labels. Some examples of proxy tasks include image in-painting, image colourization, rotation prediction, etc. Once the network is successfully able to reconstruct the original image to a close degree of similarity, it can be understood that the model has been able to learn a rich and meaningful representation of the data.

A common approach in self-supervised learning attempts to learn a meaningful representation from unlabelled data by contrasting between differently augmented views of the same image. This means that two different augmentations from the same set of data augmentations are applied on an image before feeding the pair to the model. This is known as contrastive learning. The aim in this case is to learn an embedding space in which alike samples (different augmentations of the same image) get pushed close to each other whereas different samples get pushed apart. This allows the model to learn discriminative representations. The knowledge of the dataset that the model gains in this process can then be utilized in an efficient manner for different downstream tasks like classification, detection, etc. Some of the common losses used for contrastive learning include contrastive loss (Chopra et al., 2005), triplet loss (Schroff et al., 2015), and the InfoNCE loss (Oord et al., 2018).

In summary, the beauty of self-supervised learning lies in its ability to unlock and leverage the potential of unlabelled data, allowing the models to learn meaningful representations of the data. The methodology requires designing a proxy task to guide the learning process, leveraging the structures within the data. Once the model is able to learn a rich representation of the unlabelled data, it can be transferred to a downstream task like classification, segmentation, detection etc. where it can be fine-tuned using a smaller labelled dataset for the specific application. Thereby reducing dependency on labelled data.

One of the popular contrastive learning framework is SimSiam (Simple Siamese) representation learning (Chen and He, 2021). It consists of a shared encoder that receives two augmentations or views ($v_{1}$ and $v_{2}$) of an image, and a predictor that transforms one of the encoded views to match the encoding of the other view. The network learns through a loss by matching a transformation of this encoded view of the image to the encoding of a different view of the same image. This means that an attempt is made by the model to minimize the distance between the different augmentations of the same image. In this process, the encoder of the first view is able to learn a meaningful data representation that can be fine-tuned and transferred to several downstream tasks like classification, detection, etc.

Given the paucity of labelled images, the importance of hotspot detection and isolation, and due to the competitive performance of self-supervised methods (Jaiswal et al., 2020; Chaves et al., 2023), we propose a self-supervised contrastive learning approach for the identification of thermal images containing hotspot anomalies through classification. For this purpose, we propose a process that utilizes a modified SimSiam framework, in which an XceptionNet (Chollet, 2017) is used instead of the ResNet50, the loss function is modified to include a cross-entropy function, and an ensemble of two encoders from two independently trained SimSiam frameworks is used for classification. The illustration of our model overview can be seen in Figure 1. SimSiam was picked as the framework of choice among other frameworks like SimCLR (Chen et al., 2020b), MoCo (He et al., 2020) and BYOL (Grill et al., 2020) because in the case of SimSiam, there is no need for a momentum encoder, large batch sizes, or negative samples because of its use of a stop gradient. These benefits further improve the training duration and require significantly fewer computing resources. Therefore, a SimSiam framework was used to train on a large number of unlabelled image dataset to let the model learn a good representation of the data. Following this, the encoder of the model was fine-tuned using a labelled dataset as appropriate for the application.

Once the training with the objective function in (4) is complete, the trained encoder is transferred to the downstream classification task. For this, a classification network was created using the weights of the trained encoders followed by a softmax layer, as shown in Figure 3. A labelled dataset consisting of labelled ’anomalous’ and ’normal’ images was used for tuning the classification network. The classifier consists of the trained encoder with an intermediate layer for the purpose of extraction of image features. This is followed by a dense layer with two units having softmax activation function since this is a 2-way classification task. The loss function used for fine-tuning is the sparse categorical cross-entropy loss and the Adam optimizer was used for learning. Following the completion of the fine-tuning process, this classifier is used for classifying images as ’normal’ or ’anomalous’. For our proposal, two SimSiam frameworks were trained, and an ensemble of two encoders from these models was used for classification. One of the SimSiam framework training consisted of training with the original loss function in (2), and the training dataset, whereas the other framework was trained with the compound loss in (4), and the training dataset. This was done in order to utilize the benefits of both the independently and uniquely trained SimSiam frameworks while carrying out the downstream classification task. The ensemble network was limited to two models to control the training time, and the number of parameters. Further details are presented in Section 3.

In order to precisely identify the hotspots and further verify the working of our classifier, and for the purpose of model explainability, GradCAM (Selvaraju et al., 2017) was utilized to visualize the importance maps generated by our network and then isolate the hotspots.

This was done by creating a new model, taking the same inputs as our Xception network but truncating it to the final convolution layer (final layer outputting a 4 dimensional output). Therefore, this model outputs a batch of features of the input image. This is followed by calculating the gradient of the prediction made by the classifier model with respect to the feature map outputted by the final convolution layer of the classifier. The gradients of the different feature maps were averaged (summarized). This outputs a vector with the values representing the mean intensity of each feature map channel. These values represent the importance of each channel with respect to the particular prediction made by the classifier. Therefore, in order to find the importance heatmap, these mean values of the gradient of each feature map is multiplied with the original feature map channels. This would tune the values of each feature map channel to its relative importance. Finally, in order to obtain the heatmap, the feature volume obtained from the previous step of multiplying the gradients and the original feature map is averaged along the channels dimension to get a 2D resulting heatmap. The heatmap is then normalized for ease of visualization. The heatmap allows us to see where in the input image, the classifier focuses while making its predictions. Therefore, if the classifier has been trained in a reasonable manner, the heatmaps generated in this manner would capture the hotspots specifically, while ignoring the other elements in the image. This way, the application of GradCAM in our model was furthered by utilizing it for pinpoint or isolate hotspots directly from the importance heatmap outputted by it. The overview of this method is given in Figure 4(a). A few sample images showing the output of the GradCAM approach are also given in Figures 4(b) and 4(c).

Self-supervised learning models were trained and fine-tuned on two datasets: AAU VAP Trimodal dataset and the Thermal Images for Hotspot Detection (TIHD) dataset.

In one of the SimSiam frameworks the encoder and the predictor were trained by minimizing the cost function in (4), whereas in the other one they were trained by minimizing the cost function in (2). For the compound loss equation given in (4), the best weight $\beta$ was determined empirically to be $\frac{2}{3}$. For this, estimate of the weight was conducted to investigate the effects of different loss function weights on the model performance. Three weight ratios were selected at intervals of approximately 0.3 to cover a wide range of configurations. Specifically, the weight ratios selected were $\frac{1}{3},\frac{2}{3},\frac{17}{18}$. Upon training three different models with these weights, and then testing the corresponding encoders on a classification task on the TIHD dataset, we got accuracies of 90.48%, 92.86%, and 90.48% respectively. Therefore, a weight ($\beta$) of $\frac{2}{3}$ was picked. A reasonable loss curve representing the training undertaken without the use of a large batch size, or any explicitly defined negative samples was obtained, which without the use of a stop-gradient would have degenerated rapidly. With this we were able to successfully train the encoder and also train the predictor. The encoders of these two trained SimSiam models would be combined into an ensemble network for the purpose of classification.

The batch size used for training was 81. The Stochastic Gradient Descent Optimizer was used, with a constant learning rate of 0.0008 for the AAU-VAP dataset, and 0.001 for the TIHD dataset. A momentum of 0.6 was also used. Training was conducted for 200 epochs. The duration of each iteration was approximately 65s.

For fine-tuning the classifier, the 1800 images were labelled as anomalous or normal from the AAU VAP dataset. The fine-tuning task was defined as a 2-way softmax based classification task. Figures 6(a) and 6(c) shows an example of a normal image and an anomalous image (image containing hotspot) in this dataset.

The encoders used for this downstream task were taken from the two trained modified SimSiam models and loaded with the trained encoder weights so that the learned representations from the training could be utilized by the classifier while fine-tuning on the new labelled dataset. The fine-tuning was conducted for 200 epochs as well along with early stopping constraints. It was observed that the training and validation losses decreased reasonably with the number of epochs and the training and validation accuracies increased with the number of epochs up until a certain point.

The average accuracy achieved through this method on the AAU-VAP dataset was around 94%. Similarly, training and fine-tuning was conducted for TIHD dataset. The average accuracy of fine-tuning on TIHD dataset was found to be around 97%. Sample images (normal and hotspot present) from the thermal dataset created by us are shown in Figures 6(e), 6(f), 6(g), and 6(h).

The training step is an important step for the encoder to be able to learn meaningful characteristics of the data. To test this, an XceptionNet based encoder was ’fine-tuned’ without any pre-training. It was observed that the training degenerated fast and stops, with not much learning taking place in the model. This comparison shows the effectiveness and benefits of using this contrastive learning method with unlabelled data for this kind of classification tasks.

We conducted an ablation study of the developed method in order to evaluate the contributions of the different elements of the process and the modified SimSiam framework. For this, starting from the complete finished process, elements were removed sequentially till a single SimSiam framework with a ResNet50 backbone remained.

All in all, there were three SimSiam frameworks trained under different conditions for the purpose of experimentation. Of these, the encoders of two (Models 1 and 2) were used in different ensemble models for the ablation study. The list of these three models is as follows:

1. 1.

   Model 1: SimSiam with Xception based encoder, trained with compound loss.

2. 2.

   Model 2: SimSiam with Xception based encoder, trained with regular loss.

3. 3.

   Model 3: SimSiam trained with regular loss but with ResNet50 as the encoder.

where the regular loss is the cosine similarity loss in (2), and compound loss represents a weighted sum of the cosine similarity loss and the cross-entropy loss in (4).

This ablation study consisted of the following:

1. 1.

   Our method (Ensemble of the encoders of Model 1 and Model 2)

2. 2.

   Removal of cross-entropy loss in the second SimSiam framework. (Ensemble of the encoders of two separately trained Model 2)

3. 3.

   Removal of the second SimSiam framework. (Model 2)

4. 4.

   Replacing XceptionNet with ResNet50 as the backbone in the remaining single SimSiam framework. (Model 3)

The accuracy, precision, sensitivity, specificity and F-scores were calculated for each of the models in the ablation process as shown in (5).

where TP, TN, FP, and FN refer to true positives, true negatives, false positives, and false negatives respectively. Table 1 summarizes the results of the ablation study on the TIHD dataset, whereas Table 2 summarized the results on the AAU-VAP dataset.

The weights of the encoder predictions in the ensemble models from Table 1 and Table 2 were found using grid search for maximizing the accuracy.

From the metrics in Table 1 and 2 it can be seen that our proposed method which involves an ensemble network and a compound loss consisting of the negative cosine similarity and the cross-entropy loss performs the best. Once the different modules were removed one by one, the accuracy and the F-score seem to decrease. It can also be seen that using XceptionNet as the encoder leads to better performance than using ResNet50. Specifically, the F-score increases from 0.88 to 0.90 for the TIHD dataset, and from 0.83 to 0.90 for the AAU-VAP dataset. Furthermore, once the cross-entropy loss is added to the loss function, it is observed that the accuracy shows significant improvement on both datasets. The accuracies increase from 91% to 97%, and from 90% to 94% on the TIHD and AAU-VAP datasets respectively. Therefore, major improvement was seen because of the inclusion of the compound weighted loss function.

For the purpose of comparison with existing supervised learning techniques, machine learning models like SVM (support vector machines), Logistic Regression, and Random Forest were chosen, along with CNN models like VGG16 (Simonyan and Zisserman, 2014), XceptionNet, EfficientNet (Tan and Le, 2019), ResNet (He et al., 2016) and a custom neural network. Given that it is a classification task, the AUC (Area under the curve) - ROC (Receiver operating characteristics) plot was plotted for each model, and the AUC Score was compared alongside other metrics of importance.

All the models were tested on a test dataset consisting of 100 images from the two datasets respectively. Based on the results on the testing dataset, model accuracy, precision, sensitivity, specificity and F-Score were calculated. This was done by first counting the number of true positives (TP), true negatives (TN), false positives (FP) and false negatives (FN). Following this, the metrics were calculated.

At first, the created model was compared to the existing supervised machine learning and neural network models trained on the AAU VAP Thermal Dataset. Upon comparison with the existing machine learning models, it was found that our method outperforms all of the chosen machine learning techniques on all the metrics when it comes to the AAU VAP Dataset. Table 3 summarizes the comparison of our model with the existing supervised machine learning techniques and the existing neural networks on the AAU VAP dataset.

Upon plotting the AUC-ROC curve from the results obtained by aforementioned machine learning models as well as by our model, the plot shown in Figure 7(a) was obtained. The corresponding AUC (area under the curve) scores of each model are also mentioned in Table 3. From the AUC-ROC plot in Figure 7(a), as well as from the AUC scores, it can be seen that the area under the curve that represents our model (green) is much larger than the area under the curve of each of the other models. The SVM curve (yellow) shows reasonable class separation capability whereas the Logistic Regression curve shows little class separation capability in this case. The curve for the Random Forest model is hidden behind the reference blue curve (representing equal false positive and true positive rate) which means that it has almost no class separation capability. This confirms that our model outperforms the existing supervised machine learning techniques for classification.

Secondly, in a similar manner a comparison with existing supervised convolution neural network models was also conducted. Upon comparison with these models, it was found that our model performs in a competitive manner compared to them. Our model is able to identify a greater number of true positives compared to these models because of the training, and is able to perform competitively in the other metrics as well. Table 3 summarizes the results of this comparison.

As seen from Table 3, the performance of our model is comparable to the performance of EfficientNet as is shown by the F-scores of the two models (0.94 and 0.93 respectively). The accuracy and sensitivity of our model is the highest among the different models, except for EfficientNet whose accuracy is the same as our model’s. The custom neural network which consists of 3 sets of 2D convolution and max pooling layers, followed by a dropout layer, and ending with 2 dense layers also performs reasonably, and is close to the other models. This experiment proves that self-supervised learning is a competitive approach with respect to the supervised learning approaches when it comes to classification tasks. As far as XceptionNet is concerned, it is unable to learn any meaningful data characteristics for this dataset.

Upon plotting the AUC-ROC plot for this comparison, the plot shown in Figure 7(b) is obtained. The corresponding AUC scores of each model are also mentioned in Table 3. From the AUC-ROC plot in Figure 7(b), as well as from the AUC scores, it can be seen that the area under the curve that represents our model (red), is highly competitive to the area under the curve of ResNet and EfficientNet (silver and cyan), which have slightly more AUC. The VGG16 AUC-ROC curve, and the custom NN curve show good class-separation capability as well. However, at certain points our model is able to provide a greater true positive rate coupled with a low false positive rate, because of the training, which increases its overall AUC score. The curve for the XceptionNet model is unable to rise to a reasonable level, and has minimal class-separation capability. This confirms that our model provides competitive performance when it comes to existing supervised neural network techniques.

To bolster the efficacy of the self-supervised model, it was also compared with the mentioned supervised learning models by training on TIHD dataset. In order to make the classification task more challenging for the models, hotspot-like features of random shape and size were introduced in random regions of certain images of the dataset, and these images were labelled as 0 (do not contain hotspots). The models were then trained to identify the images with the real hotspots. The augmentations include introduction of salt and pepper noise and brightness changes apart from the introduction of fake anomalies in the images that do not contain a hotspot originally.

Comparison results with existing supervised machine learning models and neural networks are summarized in Table 4. It can be observed from this table that our model achieved the highest F-score of 0.97. Even though the machine learning models achieve a perfect sensitivity, i.e., they are able to identify all positives correctly, there performance with the negative samples is not the best. This impacts their overall performance. Our model provides a reasonably balanced result when it comes to the positive and the negative samples and provide an overall dominant result. The logistic regression and the random forest models perform reasonably well with this dataset compared to the AAU-VAP dataset. In Table 4, the AUC scores have been summarized for the machine learning models trained on our dataset. Upon comparison with the traditional machine learning models, it can be observed that our model achieves the highest AUC score of 0.97, followed by logistic regression and random forest models. The same result has been visualized in Figure 8(a), where it can be observed that our model (green) has the largest area under the curve.

The model was also compared to the existing supervised neural network models trained on the TIHD dataset. It can be observed from Table 4 that our model gives a highly competitive performance to the supervised neural network models like VGG16, EfficientNet, and the custom NN. Our model is able to achieve an F-score of 0.97, whereas EfficientNet achieves an F-score of 0.94. It was noted in TIHD dataset specifically, that the particularly challenging images that were misclassified by the existing supervised learning networks, were classified correctly by our self-supervised network. This point often led to an overall improvement in its performance compared to the other models. The AUC scores obtained for this comparison, and the corresponding AUC-ROC plot can be seen in Table 4 and Figure 8(b) respectively. Again, from Figure 8(b) it can be observed that our model (red) has more points in regions with greater true positive rates and low false positive rates, leading to its better performance compared to the other models. Our model’s performance is almost at par with the performance of EfficientNet.

With all these results, it can be seen that the self-supervised learning model so developed is suitable for use in an industrial setting such as in the oil and gas exploration industry where hotspot detection in metallic equipment is of immense importance. What works in favour of this method is its higher sensitivity and specificity compared to other models since it is of utmost importance to not miss out on any possible defects (few false negatives), as well as to not classify negatives as positives.

Even though neural networks achieve reasonable performance, it is often unclear as to why a particular model makes particular decisions when provided with an input. In other words, neural networks can be thought of as ’black-box’ models whose internal decision-making is often unknown. This is the reason why we have made use of the GradCAM (Selvaraju et al., 2017) model to see what exactly our model focuses on when provided with an input, to produce its classification result. The GradCAM model is able to output heatmaps showing areas of high attention. In our case, if the model performs as expected, the same heatmaps can also be used for pinpoint location detection of the hotspot in the provided image. Figure 9 shows the input images, and the isoloated hotspots from GradCAM outputs.

From Figure 9, it can be observed in rows 1 and 2 that GradCAM is able to focus solely on the hotspots on the metal sheets for making its classification decision. Highest attention levels are depicted in red, and it can be seen that the hotspot centres are the regions in highest focus. These images also depict that different degrees of attention is placed on different degrees of hotness, with the highest attention being placed on the hottest regions. These results show that our model is able to make classification decisions by focussing on hotspots in a reasonable manner, and therefore works as expected. Its behaviour behind the scenes is not random or ambiguous. Furthermore, apart from model explainability, columns 2 and 3 of Figure 9 provide us with exact locations of the hotspots in the thermal images. Knowing the hotspot positions would facilitate any remedial work or further inspection that needs to be carried out in response, by providing engineers with a reference location of where the hotspot could be present.

Existing hotspot isolation techniques were reproduced for the purpose of comparison with our technique. On comparison with the approach proposed in Mohd et al. (2017), we found that our model is able to provide a more precise view of the hotspots present in a thermal image. The K-Means segmentation technique proposed by Mohd et al. is able to isolate the region containing hotspots as well, however the result still contains a lot of the background region (noise). The output of the technique can be observed in Figure 10(b). Therefore, after the use of the K-Means approach, further processing of the output will be required by the user to accurately identify the hotspots present in the image. Moreover, this method was not able to identify a few hotspots which got separated with the background, leaving holes in the foreground. On the other hand, the output of our method in 11(b) shows the precise isolation of all 3 hotspots.

Comparison was also made with the approach proposed in Salazar and Macabebe (2016) for hotspot detection in photovoltaic cells. This work also utilizes the K-Means segmentation algorithm. Even though this K-Means method is able to isolate the multiple small hotspots, it is not a precise isolation, given that a lot of background is also present. Moreover, it relies on a lot of manual user input to narrow down on to the region containing the hotspots. The outputs of this method can be observed in Figure 10(c). Furthermore, it was observed that this approach seems more suitable for thermal images containing large single hotspots. To test this, the comparison was also made with an input that has one large hotspot. Both, our method and the method proposed in Salazar and Macabebe (2016) outputted a precise identification of the single hotspot. This comparison shows that our method is a more general technique, applicable to a wide variety of applications.

The technique in Alajmi et al. (2019) was also reproduced for the purpose of comparison with our model. This method utilizes the HSV colour space and binary masks for the purpose of localization of the hotspots. However, the work depends on fixed specific threshold values that might not work for all cases, and therefore may not be generalizable. The outputs of this method can be observed in Figure 10(d). It can be observed that even though the method is able to precisely identify the locations of the hotspots, it is only able to identify their boundaries. On the other hand, our method is able to identify the complete hotspots precisely (Figure 11(b)). It was also found that when we tune the suggested thresholds, we are able to identify the complete hotspots rather than just the boundaries. This upholds our thoughts about this method that using a fixed set of thresholds is not tractable for different image sets, and that threshold tuning will sometimes be required to adjust for particular thermal images for the identification of hotspots.

On comparison with the technique proposed in Afifah et al. (2021), we found that even though this is a reasonable approach, it is not generalizable to all sets of thermal images of photovoltaic cells. This is because the heat signatures of different thermal images vary depending on the method used to capture them. Therefore, limiting the threshold number to given values is not the best approach, and the conclusion that higher the number of thresholds the more accurate the result does not apply every time. When testing the approach on our dataset, we empharically found the best thresholds; the results can be seen in Figure 11(a).

A quantitative comparison was also done of the methods mentioned above with our method by measuring the Dice coefficients from the results of each of the method. We randomly selected ten representative images from the TIHD dataset and the AAU VAP dataset for this purpose. The dice coefficient results have been summarized in Table 5. It can be seen from the results that our method was able to outperform the existing techniques for hotspot identification. Afifah et al. (2021) had suggested the use of 7,8 or 9 thresholds and concluded that higher number of thresholds leads to better results. However, this strategy did not work on the TIHD dataset, as the maximum number of thresholds that this method was able to use to segment the images from this dataset was 4. Any number higher than that failed to segment the image. Therefore, even though this was not suggested in Afifah et al. (2021), the number of thresholds being used for this method in Table 5 is 4, for the sake of comparisons. These results show that our method is highly precise when it comes to hotspot identification.