Supervised Anomaly Detection Method Combining Generative Adversarial Networks and Three-Dimensional Data in Vehicle Inspections

The railroad is one of Japan’s most widely known means of transportation, and it necessitates high quality maintenance to ensure daily stability. However, because inspecting inputs requires a considerable amount of work and there is a risk of oversight, methods for automatic railroad inspection are being developed[1]. JR East Japan Group, Japan’s largest railroad company, is working on smart maintenance that conforms to the state of the equipment and vehicles[2]; however, because maintenance is performed before anomalies occur owing to meticulous quality management, much anomaly data is not available. The visual inspection of the underfloor equipment of rolling stock, which is the subject of this research, is performed by human eyes, but anomaly detection has been studied to automate part of this visual inspection[3]. They were successful in detecting anomalies in the study by comparing the luminance of normal and anomalous images. However, it is difficult to apply anomaly detection per supervised learning, which necessitates a large number of anomaly images for the supervised data. In machine learning, securing sufficient data is essential. There are numerous data expansion methods via creating machine learning data via amplifying a small amount of data, which are summarized in a previous study[4]. As reported in Ref.[5], the amount of data was increased by generating images of liver lesions using deep convolutional GAN (DCGAN)[6], to classify the illness images. Moreover, CycleGAN was used to amplify a small number of facial image data[7, 8]. The method for amplifying anomaly data described above, however, uses GAN. As a result, because supervised learning requires different anomaly data for learning data and test data, it cannot be used when only one set of anomaly data is available. Alternatively, AnoGAN is an unsupervised learning method that does not use anomaly data[9]. AnoGAN can generate anomaly detection models without the use of anomaly data, but it has some limitations, such as a high computational cost. Therefore, f-AnoGAN was developed afterward[10], which greatly reduced the computational cost. In generative anomaly detection, a model trained only with normal images cannot reproduce anomaly regions, and errors occur during reconstruction. Recently, extensive research has been done to better understand the range of reconstruction of normal images. Using multiple hypotheses, the normal image’s range was clearly captured[11]. According to previous studies, the range of the normal image is defined as a Gaussian distribution[12]; a model was built using double autoencoders GAN[13]; and a rough reconstruction model was combined with a detailed reconstruction model[14]. The model accuracy has been improved by reducing the influence of the background[15, 16]. Another study increased the accuracy through training model with a small number of anomalous images[17]. Furthermore, the accuracy was improved by generating abnormal images based on normal images and using them for training[18, 19]. The anomaly detection using only normal images as described above is very important in the railroad field. As a substantial amount of anomaly imagery is not available in the railroad maintenance field, many railroad corporations and organizations are exploring it. To address cases where there is a lot of data but a small percentage of anomaly images, luminance was improved by detecting anomaly data via semisupervised learning and adding it to the training data[20]. Simultaneously, a model was developed for detecting anomalies in an overhead wire using a DCGAN discriminatory mechanism, as well as a model for detecting anomalies in railroad equipment using unsupervised learning that does not use anomaly data[21]. Moreover,, models were used that also include encoders that convert image data to vector spaces, to create anomaly detection models for electric equipment[22, 23]. These models do not use anomaly images, but there was still room for improvement in terms of luminance. A method for detecting anomalies was proposed by overlaying rail images of the same area and detecting the differences[24, 25]. This method can be used if there is only one image as a sample per shot or area, but it required machine learning using anomaly images to prevent false positives that could likely occur. Alternatively, there is a method for adjusting for a lack of anomaly data in industrial and robot detection by creating anomaly images with three-dimensional (3D) computer graphics (CG) software and using them as supervised images. It is possible to generate anomaly images that do not exist using this method, but there will be issues with color and texture differences between CG images and actual photos, as well as being unable to distinguish actual images even when training with CG images. CG simulation images were used to train the object detection model of robotic controls, and they enable recognition of “one of many variations” for actual images by changing the rendering settings, generating images in various brightness, and colors, and using them as supervised images[26]. Further, images were generated by adjusting the texture settings of the objects to detect, approximate the actual texture, and use images from the actual site for the background portions[27]. Based on this research, we can see that “object detection” using CG for the supervised data was successful, but it does not refer to “anomaly detection” that detects small changes in images. Anomaly detection is a more difficult task than object detection because the precision changes depending on small differences in the texture and luminance when compared to object detection. In this study, we used GANs on CG to generate anomaly images that were infinitely close to the real images shot with an external inspection device and then used those as supervised data to detect anomalies.

In this research, we verified that anomaly detection is possible using fictional anomaly images generated by combining CG and GCGAN as supervised data. A common issue in railroad maintenance is the scarcity of anomaly data. When a specific number of anomaly patterns can be obtained from a small amount of data (for example, when there are several dozen to several hundred anomaly data), there is existing research that allows for an increase in data through data expansion. Furthermore, even if there was a way to build a model using unsupervised learning without using any anomaly data, and the images were normal in the unsupervised learning, there would be more false positives owing to environmental changes, causing precision issues. In this research, we were able to identify anomaly patterns by generating fictional anomaly images and succeeded in detecting anomalies without increasing the number of false positives. In this study, efforts in creating models specific for this research using 3D CG software were required, but we believe that, in the future, we could collect learning data with a simulator, as has been done in related research on robot recognition. In the future, we will expand on this research, address the issue of scarce anomaly data that is shared across the railroad industry, and aspire to implement automatic anomaly detection in society.

We wish to thank East Japan Railway for supplying sample data. We would like also to thank colleagues in Technical Research Center in JR East Information Systems for useful discussions.