Wooden Sleeper Deterioration Detection for Rural Railway Prognostics Using Unsupervised Deeper FCDDs

While analyzing fatal train accidents on Europe’s mainline railways from 1980 to 2019, Evans discovered that train collisions, derailments, and railroad crossings were the primary causes of accidents [3][4]. In Japan, according to the traffic safety committee’s 1987-2018 derailment statistics, there were 173 cases, with three main causes: 65 natural disasters, 47 railroad crossings, and 33 railway infrastructures [11]. To prevent the potential accident of derailment, condition-based and risk-based maintenance in railway infrastructures have been one of key activities for daily operations. In detail, derailment accidents have been categorized into two types based on the railway tracks: derailment among inter-rails and riding on the outside rail.

Frequently, inter-rail derailments occur because decayed wood sleepers weaken the supporting load of the rail, causing the distance between the parallel rails to expand. Rural railway managers typically operate on a small-scale and have a weak financial status, where inter-rail derailment can negatively impact the profit per day kilometers because of the short length of running operations. Given that rural rail tracks predominantly use wood sleepers, derailment among the inter-rails is more likely to occur. Therefore, in rural railway maintenance, monitoring wooden sleeper deterioration is critical to reduce the risk of derailment. However, repairing or renewing a decayed sleeper typically requires significant human labor. To improve the performance of weekly inspections of rural railway tracks, deep learning-based visual inspection techniques can be employed.

Tang et al. reviewed articles on artificial intelligence applications in railway systems from 2010 to 2020, categorizing them into five subdomains: maintenance and inspection, traffic planning and management, safety and security, autonomous driving and control, and passenger mobility [16]. The most popular research field was found to be maintenance and inspection, with over 81 papers (57%). Ji et al. reviewed existing deep learning applications for rail-track condition monitoring from 2013 to 2021, specifically focusing on supervised deep learning and recent adoption by rail industries [7]. The number of papers on this topic surged in 2018, and 14 regions worldwide were represented in rail-related research. This indicates that the rail industry has shown growing interest in adopting deep learning methods. Of the studies, 70% used raw image-type data for deep learning models, while the remaining types included acoustic emission signals, defectograms, maintenance records, and synthetic data from generative models. The purpose of these studies was to detect, classify, and/or localize rail surface defects, including various components such as rails, insulators, valves, fasteners, switches, and track intrusions. Over 38 deep learning models have been adopted by researchers, with CNNs being popular for feature extraction and RNN/LSTM being used for sequential data types. Researchers followed a consistent process flow for deep learning applications for rail-track condition monitoring, with the first subsystem being image acquisition (using cameras/recording devices) installed on rail maintenance vehicles to capture input data. The second subsystem involved optional preprocessing, where images were resized, enhanced, noise-removed, or cropped for target areas using image-processing techniques. The input data were then prepared for training and testing deep learning models. Finally, the trained model was produced using the parameters for real-world applications. Given the criticality of rail-track condition monitoring, inspections by human operators could double the accuracy of the system. However, the distribution of rail-track image data is often uneven and extremely disproportional, resulting in class imbalance problems. This study proposes an unsupervised learning method using a one-class classification algorithm as a novel application in rail-track condition monitoring.

Figure 1 depicts several railway inspection applications that utilize deep learning models for the detection of defective classes of railway components. These applications are often based on supervised deep learning approaches such as classification [1, 2] and object detection [9, 6, 5]. In supervised learning studies, the authors assigned class labels by annotating whole images and bounding boxes that enclosed defective regions. The railway components targeted for defect detection include rail tracks, fasteners, and sleepers. Railway defects are inherently uncertain events, and the number of anomalous images is often imbalanced toward the normal class. Defect classes are not yet completely defined in railway inspections. For instance, Hsieh et al. [6] defined six normal classes and four defective classes by focusing on clips on wooden/concrete crossties, spikes, fishplate, slide-bed plates, and guard rail plates. The authors collected a limited number of real images, which resulted in seven classes having fewer than 100 defective images and three classes having two or more hundred images. It is not easy to reconstruct synthetic images to represent the health condition of railway components amidst a complex background consisting of trees, grass, and ballast stone. Furthermore, generating defective features as annotation data that can contribute to architectural performance is challenging. Collecting defective status data to build a railway inspection application always requires significant time investment, given that the temporal occurrence of defects is extremely imbalanced. To achieve stable and high performance, a supervised learning approach demands thousands of paired datasets consisting of defective real images and annotated labels or bounding boxes. In contrast, the unsupervised anomaly detection approach has the advantage of requiring fewer images to optimize the parameters for training normal and anomalous features. Moreover, the visual heat map explanation enables us to discriminate between localized defective features. The authors [17] found that the deeper fully-convolutional data descriptions (FCDDs) has been applicable to several damage data sets of concrete/steel components in structures: pavement, bridge, and dam, and fallen tree, and wooden building collapse in disasters: typhoon, earthquake. However, it is not yet known to feasible to railway components that includes deterioration of wooden sleepers. In this study, we propose a prognostic discriminator pipeline to automate the one-class classification of defective railway components.

The authors [17] have already formulated the deeper FCDDs and found the applicability to damage data sets of bridge, dam, and building. However, as an unsupervised anomaly detection approach, the deeper FCDDs has been not yet known to feasible to video frame images in front angle of railway track that contains ballast stones, rail, spike, fastener, and concrete/wooden sleepers. For risk-based maintenance of rural railways, visualizing hazard-mark heatmaps and computing risk-weighted anomaly scores is crucial for rural railway inspection and prognostic support for effective repairs under usable resources: time, labors, and budget.

Let $F_{i}$ be the $i$-th frame of an image with a size of $h\times w$ and let $c$ be the center of the hypersphere boundary between the inlier normal region and the outlier anomalous region. We consider the number of training images and the weight $W$ of the fully convolutional network (FCN). The deep support vector data description (SVDD) objective function [13] is formulated as a minimization problem for a deep support vector data description as follows:

$$\min_{W}\frac{1}{n}\sum_{i=1}^{n}\|\Phi^{B}_{W}(F_{i})-c\|^{2},$$

(1)

where the $\Phi^{B}_{W}(F_{i})$ denotes a mapping of the deeper CNN to backbone $B$ based on the input frame image. The one-class classification model was formulated using the cross-entropy loss function as follows:

$$\begin{split}\mathcal{L}_{DeepSVDD}=&-\frac{1}{n}\sum_{i=1}^{n}(1-z_{i})\log\ell(\Phi^{B}_{W}(F_{i}))\\ &+z_{i}\log[1-\ell(\Phi^{B}_{W}(F_{i}))],\end{split}$$

(2)

where $z_{i}=1$ denotes the anomalous label of the $i$th frame of the image and $z_{i}=0$ denotes the normal label of the $i$th frame of the image. A pseudo-Huber loss function is introduced to obtain a more robust loss formulation [14] in Equation (2). Let $\ell(u)$ be the loss function and define the pseudo-Huber loss as follows:

$$\ell(u)=\exp(-H(u)),~{}H(u)=\sqrt{\|u\|^{2}+1}-1.$$

(3)

Substituting Equation (2) into Equation (3), we obtain the following expression:

$$\begin{split}(2)\equiv&-\frac{1}{n}\sum_{i=1}^{n}(1-z_{i})H(\Phi^{B}_{W}(F_{i}))\\ &+z_{i}\log[1-\exp\{-H(\Phi^{B}_{W}(F_{i}))\}].\end{split}$$

(4)

Therefore, a deeper FCDD loss function can be formulated:

$$\begin{split}&\mathcal{L}_{deeperFCDD}=\frac{1}{n}\sum_{i=1}^{n}\frac{(1-z_{i})}{uv}\sum_{x,y}H_{x,y}(\Phi^{B}_{W}(F_{i}))\\ &-z_{i}\log\left[1-\exp\left\{\frac{-1}{uv}\sum_{x,y}H_{x,y}(\Phi^{B}_{W}(F_{i}))\right\}\right],\end{split}$$

(5)

where $H_{x,y}(u)$ are the elements $(x,y)$ of the receptive field of size $u\times v$ under a deeper FCDD. The risk-weighted anomaly score $S_{i}^{rw}$ of the $i$th image is expressed as the sum of all the elements of the receptive field as follows:

$$S^{rw}_{i}(B)=r_{i}\sum_{x,y}H_{x,y}(\Phi^{B}_{W}(F_{i})),~{}i=1,\cdots,n.$$

(6)

Here, $r_{i}$ is the weight of the derailment risk caused by the wooden sleeper deterioration. For example, for larger ratios of the curve, the weight can be set higher. Specifically, we provided $i$th ratio of the curve to match the GNSS-based position. We herein present the construction of a baseline FCDD [17] with an initial backbone $B=0$ and performed CNN27 mapping $\Phi^{0}_{W}(F_{i})$ from the input frame of image $F_{i}$ in the dataset. We also present deeper FCDDs focusing on elaborate backbones $B\in\{$VGG16, ResNet101, Inceptionv3$\}$ with a mapping operation $\Phi^{B}_{W}(F_{i})$ to achieve a more robust detection. In this paper, we present ablation studies on a rural railway dataset for detection towards decayed wooden sleeper.

Convolutional neural network (CNN) architectures, comprising millions of common parameters, have exhibited remarkable performance for visual inspection, but the underlying reasons for this superiority remain unclear. Heatmap visualization techniques for detecting and localizing anomalous features are typically categorized as masked sampling and activation map approaches. The former includes methods such as occlusion sensitivity [18] and local interpretable model-agnostic explanations [12]. The latter category includes activation maps such as class activation maps (CAMs) [19] and gradient-based extensions (Grad-CAM) [15]. Nonetheless, aforementioned methods of disadvantage is its requirement for parallel computation resources and iterative computation time for local partitioning, masked sampling, and for generating a gradient-based heatmap. In this study for railway inspection applications, we adopt the receptive field upsampling approach [8] to visualize anomalous features using an upsampling-based activation map with Gaussian upsampling from the receptive field of the FCN. The primary advantages of the upsampling approach are the reduced computational resource requirements and shorter computation times. The proposed upsampling algorithm generates a full-resolution anomaly heatmap from the input of a low-resolution receptive field $u\times v$.

Let $H\in{R}^{u\times v}$ be a low-resolution receptive field (input), and let $H^{\prime}\in{R}^{h\times w}$ be a full-resolution of hazard-mark heatmap (output). We define the 2D Gaussian distribution $G_{2}(m_{1},m_{2},\sigma)$ as follows:

$$\begin{split}&[G_{2}(m_{1},m_{2},\sigma)]_{x,y}\equiv\\ &\frac{1}{2\pi\sigma^{2}}\exp\left(-\frac{(x-m_{1})^{2}+(y-m_{2})^{2}}{2\sigma^{2}}\right).\end{split}$$

(7)

The Gaussian upsampling algorithm from the receptive field is implemented as follows:

1. 1.

   $H^{\prime}\leftarrow 0\in{R}^{h\times w}$

2. 2.

   for all output pixels $d$ in $H\leftarrow 0\in{R}^{u\times v}$

3. 3.

   $u(d)\leftarrow$ is upsampled from a receptive field of $d$

4. 4.

   $(c_{1}(u),c_{2}(u))\leftarrow$ is the center of the field $u(d)$

5. 5.

   $H^{\prime}\leftarrow H^{\prime}+d\cdot G_{2}(c_{1},c_{2},\sigma)$

6. 6.

   end for

7. 7.

   return $H^{\prime}$

After conducting experiments with various datasets, we determined that a receptive field size of $28\times 28$ is a practical value. When generating a hazardous heatmap, unlike a revealed damage mark, we need to unify the display range that corresponds to the anomaly scores, which range from the minimum to the maximum value. In order to strengthen the defective regions and highlight the hazard marks, we define a display range of [min, max/4], where the quartile parameter is 0.25. This results in the histogram of anomaly scores having a long-tailed shape. If we were to include the complete anomaly score range, the colors would weaken to blue or yellow on the maximum side.

We developed a railway-purpose application to automate one-class anomaly detection by replicating a baseline FCDD using a light backbone CNN network with 27 layers. To ensure feasibility of a railway application, we assessed an unsupervised approach for deeper FCDDs with pretrained backbones of VGG16, ResNet101, and Inceptionv3, and performed ablation studies by comparing with the baseline FCDD. In order to support the decision to repair the wooden sleeper and minimize the possibility of derailment for safe railway operations, we proposed an index of the risk-weighted anomaly score. Additionally, we visualized hazard-mark heatmaps using direct Gaussian upsampling of the receptive field of the FCN. We evaluated the deeper FCDDs model on experimental targets, such as decayed wooden sleepers. The heatmap indicates that the hazard marks of the decayed wooden sleeper can cause a spike out of the rail and, in the worst case, a running train could result in derailment. Our experiments produced a little better accuracies around 79% for AUC and recall. Deeper FCDDs improved the hazard marks for visual explanation, even without annotating the decayed wooden sleeper regions. We discovered that a hazard-localized approach of deeper FCDDs outperformed the baseline FCDD on rural railway datasets. Our work presents a new solution for deeper FCDDs that offers a wooden sleeper deterioration detection tool for rural railway inspections with better accuracy and hazard-explainability, providing a novel contribution to the field in rural railway track.

This study discovered the feasibility of unsupervised deterioration detection highlighting wooden sleeper using the deeper FCDDs in rural railway. However, this scope is too limited for railway inspection to make a decision of repair. We are challenging to detect any deterioration using our application, so another target remains such as fastener, and spike that are pushing out or slipping out. This region of target is smaller and closer, but the impact of defective status is larger to be potential accident of derailment. The key of data preparedness is to divide the cropped image with the size of 1280$\times$2560 into the left-side and right-side of rail with the size of 1280$\times$1280, respectively. Though the deterioration scope of wooden sleeper contains both side of rail, but the defective scope of fastener and spike must be recognized as either left-side or right-side of rail. In addition, the season of this study was at spring, April and May 2023. Another season, especially winter may influence the background noise: larger shadow, decayed grass, iced, and snowy. For more robustness, we can acquire video at various scene, and the prepared data enable to update the parameters of our anomaly detector.

Several promising directions exist for future research to improve the usability of visual inspection applications. To address the challenges of background noise and imbalanced data, augmentation preprocessing such as mixup, and random erasing can be effective for one-class classification models. However, the imbalance issue remains for infrequent defects such as spikes out of rail from wooden sleepers, cracks of concrete sleepers, and holes on ballast tracks. To overcome this challenge, the risk-weighted anomaly score generated by our deeper FCDDs can be used in edge devices for effective data acquisition of rare classes. By collecting only the frames that have hazard marks with significantly higher anomaly scores than a predefined threshold, the data acquisition process can be made more efficient. For risk-based maintenance to incorporate the potential hazard of derailment in each track, we are able to add the label of curve ratio [11]. Then, we can utilize the risk-weighted anomaly score for supporting to make a decision of priority to repair effectively for sustainable and safety operation in rural railway. For prognostic monitoring, we could continue to record the risk-weighted scores every inspection, and we could compare the recent score from the previous scores at each wooden sleeper. For deterioration forecast, we could mine the scores every inspection, and apply hazard models based on the threshold that a railway manager make a priority for repair.

The authors wish to thank MathWorks and Takuji Fukumoto for providing helpful MATLAB resources for Automated Visual Inspection. We also thank Nakasha Creative Co.,Ltd., for providing the opportunity of railway study.