Weakly Supervised Patch Label Inference Networks for Efficient Pavement Distress Detection and Recognition in the Wild

The traditional pavement distress analysis approaches mainly include filter-based methods and hand-crafted feature-based classical classifiers. For example, in [17], wavelet transform is used to decompose a pavement image into different-frequency subbands. Hu et al. [18] propose a novel Local Binary Pattern (LBP) based operator for pavement crack detection. In [19], a random structured forest named CrackForest, which is combined with the integral channel features, is proposed for automatic road crack detection. Kapela et al. [20] propose a crack recognition system based on the Histograms of Oriented Gradients (HOG). Pan et al. [21] use the four popular supervised learning algorithms (KNN, SVM, ANN, RF) to discern pavement damages. However, the traditional methods usually have weak performance owing to numerous artificial design factors, and separate optimization procedures, and they cannot be adapted to a large number of data currently.

Inspired by the recent remarkable successes of deep learning in extensive applications, simple and efficient convolutional neural networks (CNN) based pavement distress analysis methods have gradually become mainstream in recent years. In general, these methods can be divided into three parts according to the task objective: pavement distress segmentation [12, 22, 23, 24], pavement distress location [25, 26], and pavement distress classification [13, 27, 28]. Among them, pixel-based pavement distress segmentation is a hot research field. Zhang et al. [22] leverage CNN to classify the image patch for segmenting pavement distress. In [12], a CNN is used to learn the structure of the cracks from raw images, then the segmentation result is generated by the obtained structure information. Based on the fully conventional network (FCN), Yang et al. [23] fuse multiscale features from top-to-down for pavement crack segmentation. In DeepCrack [24], multiscale deep convolutional features learned at hierarchical convolutional stages are fused together to capture the line structures. For distress localization, Ibragimov et al. [25] propose a method for localizing signs of pavement distress based on a faster region-based conventional neural network. Zhu et al. [26] compare the performance of three state-of-the-art object-detection algorithms on an Unmanned aerial vehicles(UAV) pavement image dataset, which includes six types of distress. Because pavement distress annotation requires professional knowledge and a large amount of time, the datasets used in the above methods are low-resolution and small-scale. However, it remains to be determined whether models derived from small-scale datasets can be applied to real-world practice.

For pavement distress classification, Dong et al. [27] propose a metric-learning based method for multi-target few-shot pavement distress classification on the dataset, which includes ten different kinds of distress. In [28], discriminative super-features constructed by the multi-level context information from the CNN are used to determine whether there is distress in the pavement image and recognize the type of distress. All of these methods do a good job of classification on the dataset they use, which is small and only contains distressed images, and on which the test accuracy even achieves 100% [28]. There have been few works to systematically evaluate the model’s performance on a difficult large-scale multi-type dataset. Moreover, these approaches only regard the pavement distress detection or recognition problem as a common image classification problem and directly apply the classical deep learning approaches. In [13], patch-based weakly learning model IOPLIN and large-scale distress datasets CQU-BPDD are proposed to solve these problems. However, the main drawback of IOPLIN is that the patch label inference strategy based on the pseudo label makes IOPLIN incompatible with pavement recognition, and its optimization process is quite complex and time-consuming. Our approach takes inspiration from IOPLIN but operates with different patch inference strategy and uses more effective and hierarchical patch collection strategies. Besides, Li et al. [29] leverage Spatial Transformer Network [30] to automatically recognize different types of defects and measure the coverage percentage on the ship surface, which introduces a possible solution for pavement image classification.

In recent years, due to the popularity of the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) [31], many computer vision algorithms based on deep learning have emerged. Among them, a series of convolutional neural networks (CNNs) play a leading role in the field of image classification. For example, AlexNet [32] first applies the structure of convolutional neural networks to large-scale image classification datasets. Simonyan et al. first propose the deep and large-scale convolutional neural network VGGNet [33] (i.e., the VGG19 model has 19 layers and more than 130 million parameters, while the previous convolutional neural network has less than 10 layers and millions to tens of millions of parameters). In InceptionNet [34], $1\times 1$ convolution kernel application is proposed for the first time. He et al. [35] propose a residual structure and network extension strategy to construct a network family for the first time. Zoph et al. [36] bring CNN into the embedded mobile terminal and propose MobileNet, specially designed for low computing power and low memory computing platform. Then based on MobileNet and Neural architecture search (NAS) [37], Tan et al. propose an efficient CNN, dubbed EfficientNet [38]. In the past year, inspired by the field of Natural Language Processing (NLP), Dosovitskiya et al. [39] propose a visual classification model based on Transformer [40], named as ViT.

However, it is very challenging to employ the general classification models based on CNN and Transformer for addressing pavement distress analysis issues directly. This is mainly because pavement images have many specific characteristics, such as high image resolution, low distress area ratio, and uneven illumination, compared with object-centric natural images. Our approach intends to employ a patch collection strategy in an image pyramid to incorporate both local and multi-scale information for pavement distress analysis.

Both pavement distress detection and recognition can be deemed as image classification tasks from the perspective of computer vision. Let $X=\{x_{1},\cdots,x_{n}\}$ and $Y=\{y_{1},\cdots,y_{n}\}$ be the collection of pavement images and their pavement labels, respectively. $y_{i}$ is a $C$-dimensional one-hot vector where $C$ is the number of categories and $y_{ij}$ indicates the $j$-th element of $y_{i}$. In the detection case, such a classification task is a binary image classification issue (distressed or normal) where $C=2$. In the recognition case, this classification task is a multi-class image classification problem where $C>2$. In a pavement label $y$, if the $j$-th element is the only nonzero element, it indicates that the corresponding pavement image belongs to the $j$-th category. The pavement distress detection or recognition is to learn a classifier $F_{det}(\cdot)$ or $F_{reg}(\cdot)$ can label the pavement image correctly, $y_{i}\leftarrow F(x_{i})$.

There are two strategies for accomplishing the pavement distress recognition task. One is the two-stage recognition flow path, and the other is the one-stage recognition flow path. The two-stage recognition is to identify distressed images first via pavement distress detection and then apply the pavement distress recognition to further classify each distressed image into a specific type of pavement distress. The one-stage recognition directly considers the normal case as an additional category in the recognition procedure. Therefore, the pavement distress detection and recognition tasks are jointly tackled with one image classification model.

Similar to Iteratively Optimized Patch Label Inference Network (IOPLIN), WSPLIN is a patch-based pavement image classification method whose main obstacle is to train Patch Label Inference Network (PLIN) only with the image label. WSPLIN introduces an additional module named Comprehensive Decision Network (CDN) to guide the optimization of PLINs in an end-to-end weakly supervised learning manner. The flow path of WSPLIN is very concise. In WSPLIN, the pavement image is divided into several patches first, and then PLIN is used to infer the labels of these patches. Finally, the inferred labels are fed into CDN to yield the final pavement label. WSPLIN has two core modules, namely PLIN and CDN, whose corresponding mapping functions are $\Gamma_{\zeta}(\cdot)$ and $\Theta_{\xi}(\cdot)$, respectively.

We adopt three different patch collection strategies for producing patches. They are Slide Window (SW), Image Pyramid (IP), and Sparse Sampling (SS). The first strategy is also adopted by IOPLIN. WSPLIN uses the second strategy to exploit image information from different scales fully. The third strategy is newly designed by us based on the IP strategy for speeding up WSPLIN via reducing the patch amount for training.

Slide Window: The pavement image is simply divided into a series of uniform scale patches following a non-overlapping strategy. We adopt $300\times 300$ as the sliding window size with 300 sliding stride. The patch collection can be denoted as $P_{i}=\tau(x_{i})=\{p^{i}_{1},...,p^{i}_{m}\},m=12$ where $\tau(\cdot)$ is our patch extraction operation.

Image Pyramid: The slide window strategy does not consider the scale information. So we resize the pavement image into three resolutions, $300\times 300$, $600\times 600$ and $1200\times 900$ (the original size), to construct a three-layer image pyramid from top to down, and then employ sliding window method for dividing the image into patches. The patch collection can be denoted as $P_{i}=\{\tau(x_{i}^{l})\}_{l\in\{0,1,2\}}=\{p^{i}_{1},...,p^{i}_{m}\},m=\sum_{l}m_{l}=17$ where $x_{i}^{0}=x_{i}$ and $l$ indicates the layer ID. Similar to the slide window strategy, we also apply $300\times 300$ as the sliding window size with 300 sliding strides in the image pyramid. Therefore, $m_{0}=12$, $m_{1}=4$, and $m_{2}=1$.

Sparse Sampling: The patch number determines the scale of training data, and the patches in the same image pyramid also contain some redundant information in the scale space. Therefore, we can sample some patches for each image to reduce the training burden and speed up the model. More specifically, let $\alpha$ be the sparse sample ratio to control the number of sampled patches for each layer, $n_{l}=\left\lceil m_{l}\times\left(1-\alpha\right)\right\rceil$ where $\left\lceil\cdot\right\rceil$ returns the smallest integer that is greater than or equal to the input. We design a simple strategy for sampling patches in each layer. In this strategy, the sampled patches of all three layers should cover all scales while maximizing spatial coverage. The optimal patch sparse sampling strategy is mathematically denoted as follows,

where $\text{Vol}(\cdot)$ returns the volume of the given set and $\mathcal{C}_{l}$ denotes an index subset to patches in $l$-th layer. Since the solutions to the above problem are limited, we can use the enumeration method to address this issue efficiently when $n_{l}$ is fixed. In this paper, we empirically set $\alpha=0.25$. In such a manner, $n_{0}=1$, $n_{1}=1$ and $n_{2}=3$.

For distinguishing different versions of WSPLIN, WSPLIN-SW, WSPLIN-IP, and WSPLIN-SS indicate the versions that use sliding window, image pyramid, and sparse sampling patch collection strategies, respectively. The default version of WSPLIN is WSPLIN-IP, since WSPLIN-IP is the best-performing version. WSPLIN-SS is the lightweight version of WSPLIN, which takes a good trade-off between performance and efficiency due to its special design. WSPLIN-SW is an ideal choice in the case that the distresses have not suffered from the high diversity in scale.

Similar to IOPLIN [13], we adopt EfficientNet-B3 [38] as our Patch Label Inference Network (PLIN) due to its good trade-off between performance and efficiency. We denote $\Gamma_{\zeta}(\cdot)$ as the mapping function of PLIN. The patch label inference procedure is denoted as,

where $S_{i}$ is an $m\times C$-dimensional matrix in which every column encodes the label inference confidences of patches. Such confidences are expected to be zero if the patch does not contain the distress and reflect the possibility of the certain distress that the corresponding patch has. Please note, there is no supervised information on patches, so all these labels are randomly produced just via forward propagation. We need to leverage the follow-up comprehensive decision network to guide the PLIN to generate reasonable patch labels with image-level labels in a weakly supervised manner. We will introduce such a procedure later.

We establish a Comprehensive Decision Network (CDN) for accomplishing the final pavement image classification based on the aforementioned patch label results. CDN consists of four layers where the first two layers are all the $m\times C$ fully connection layers followed by a ReLU, and Dropout layer, the third layer is also an $m\times C$ fully connection layer, and the size of output fully connection layer is $C$. Here, $C$ is the number of categories. Let $\Theta_{\xi}(\cdot)$ be the mapping function of CDN, then the predicted pavement distress label $\hat{y}_{i}$ can be obtained by,

We use the cross-entropy to measure the discrepancy between the predicted label and ground-truth and denote it as the classification loss $\mathcal{L}_{c}$,

Finally, the optimal WSPLIN model is learned by minimizing the following loss,

where $\lambda$ is a positive parameter for reconciling the classification loss and the sparsity constraint.

WSPLIN is an end-to-end deep learning framework that uses back-propagation to compute the loss deviation and update the parameters layer by layer. In WSPLIN, CDN requires the patch label results produced by PLIN that should be useful for the final classification. The patch label sparsity constraint forces WSPLIN to highlight only a few of the most crucial patches for participating in the final decision. Clearly, these highlighted patches should be distressed ones, and their inferred patch label results should be nonzero since only the distressed patches can provide helpful information for the final detection and recognition. In such a manner, CDN essentially guides the training of PLIN in a weakly supervised manner.

Detection and One-Stage Recognition: The pavement detection and one-stage pavement distress detection can be deemed as a one-stage pavement image classification problem. To tackle these tasks, we can train our model as a pavement image classifier. Once the model is trained, the pavement image $x$ can be divided into patches with different patch collection strategies, which are fed into WSPLIN for yielding the final classification,

where $F\in\{F_{det},F_{reg}\}$ and the predicted category should be corresponding to the maximum element of $y$.

Two-Stage Recognition: The two-stage recognition has two stages to accomplish pavement distress recognition. The first stage is to train our model as a pavement distress detector $F_{det}$ for filtering out the normal samples and finding the distressed samples. The second stage is to train our model as a multi-class pavement image classifier $F_{reg}$ for completing the final distress recognition,

where $x_{i}\in\{x|F_{det}(x)\neq y_{\text{normal}}\}$ and the maximum element of $y_{i}$ reflects the specific pavement distress category of the distressed pavement image $x_{i}$.

Histogram of Oriented Gradient (HOG) [5], Local Binary Pattern (LBP) [46], Fisher Vector(FV) [47], Support Vector Machine (SVM) [48], ResNet-50 [35], Inception-v3 [49], VGG-19 [33], ViT-S/16 [39], ViT-B/16 [39], EfficientNet-B3 [38], Spatial Transformer Networks (STN) [30, 29], and Iterative Optimized Patch Label Inference Network (IOPLIN) [13] are selected as baselines. The first four approaches are the shallow learning-based approaches. ResNet-50, Inception-v3, VGG-19, and EfficientNet-B3 are the classical Convolutional Neural Network (CNN) models. ViT-S/16 and ViT-B/16 are the recently popular transformer models. IOPLIN is a well elaborated pavement distress detection approach. The architectural design of STN is derived from [29].

Table IV records the pavement distress recognition performances and parameter scales of different approaches under different application settings on the CQU-BPDD dataset. Similar to the pavement distress detection performances, WSPLIN-IP achieves better recognition performances than baselines under all settings, while enjoys a smaller parameter scale. In I-REC, the performance gains of WSPLIN-IP over Inception-v3, which is the second-best method, are 1.8% and 3.4% in top-1 accuracy and $F1$, respectively. In II-REC(n) and II-REC(i), the EfficientNet-B3 achieves the second-best performance. The performance gains of WSPLIN-IP over it under II-REC(n) are 1.1% and 3.3% in top-1 accuracy and $F1$, respectively. Such gains under II-REC(i) are 6.4% and 6.9%. The distribution of different pavement image categories is imbalanced. Top-1 accuracy is sensitive to this data imbalance, while $F1$ is more stable to this imbalance. Therefore, $F1$ can better reflect the comprehensive performances of recognizers. According to the observations, WSPLIN-IP shows more advantages compared with baselines in $F1$.

The test settings of I-REC and II-REC(n) are identical as seen in Table I. However, the models trained under I-REC outperform the ones of II-REC(n). For example, Inception-v3, ViT-B/16, EfficientNet-B3, and WSPLIN-IP trained under I-REC achieve 3.1%, 1.6%, 2.0%, and 1.8% improvements over the ones under II-REC(n) in $F1$. This implies that the end-to-end pavement distress recognition solution, which addresses detection and recognition tasks jointly (I-REC), enjoys more advantages than the conventional two-stage implementation solution, which addresses detection and recognition tasks individually (II-REC(n)), in the real-world application. We attribute this to that the end-to-end solution exploits the complementarity of these two tasks and introduces global optimization.

An interesting phenomenon is observed from Table IV that the top-1 accuracies of II-REC(i) are lower than the ones of the rest two settings while its F1-scores are higher than the F1-scores of the rest settings. This is because the test setting of II-REC(i) is different from the settings of I-REC and II-REC(n), which does not involve any normal pavement sample. Top-1 accuracy is measured in sample-wise but $F1$ is measured in category-wise. The normal samples comprise a large proportion of the whole data in I-REC and II-REC(n). Therefore, the superabundant normal samples will make the recognizers trained under I-REC bias to the classification of the normal sample, which leads to the high top-1 accuracy but the low $F1$. With regard to II-REC(n), the massive normal samples push up the top-1 accuracy. However, the measure of $F1$ is independent of the sample amount and the classification error of II-REC(n) is accumulated from both the detection and recognition stages. Therefore, it achieves a lower $F1$ in comparison with II-REC(i).

In this section, we evaluated the impact of various components and hyperparameters on the performance and efficiency of our model, as well as compared to IOPLIN. The results of these experiments are recorded in Table V.

Our method is a fully supervised learning approach from the perspective of image classification, since the output and input of our model are pavement images and category labels, respectively, and all this information is available for training. Since our method is a fully supervised learning method, why do we call it a Weakly Supervised Patch Label Inference Network (WSPLIN)? This is because we transform this image-level fully supervised learning problem as a patch-level weakly supervised learning issue for a solution. The main idea of our work is to divide images into patches and then use neural networks to infer the patch labels. Finally, the patch labels are aggregated to infer the label of images. The core step of our method is patch label inference. Since this step has no patch-level supervised information, the patch label inference is conducted in a weakly supervised learning manner.

In order to further analyze pavement image classification in different learning manners, we employ some recent advanced weakly and self-supervised learning approaches, namely Grad-CAM [51], C²AM [52], DINO [53], and SimSiam [54], for addressing pavement image classification issue. Grad-CAM and C²AM are the weakly supervised learning approaches, while DINO and SimSiam are the self-supervised learning approaches. The experimental results are reported in Table VI. The results show that the fully supervised learning manner performs much better than the weakly and self-supervised learning manners. For example, the performance gains of WSPLIN-IP over Grad-CAM, C²AM, DINO and SimSiam in F1 are 3.1%, 15.3%, 55.3%, and 54.9%, respectively. Grad-CAM is the best-performed weakly supervised learning approach. It is a post-processing method. Thus, it actually has not introduced any impact on the training process of the original classification model, and the classification performance only depends on its backbone–EfficientNet-B3. C²AM is a SOTA one-stage weakly supervised learning method. However, it performs even worse than its backbone (-9.9% on I-DET, -12.2% on I-REC). We believe this is due to the interference from the pseudo label generation process, since all the processes are optimized together and the model is originally designed for object detection and segmentation instead of classification. The self-supervised learning approaches, such as DINO and SimSiam, almost fails in pavement distress detection and recognition tasks. This reveals the importance of supervised information in the feature learning step. Note, we here only use self-supervised learning methods to train the feature learning part and train the classifiers with supervised information. In summary, directly applying the weakly or self-supervised learning approaches to address pavement distress detection and recognition issues will lead to unsatisfactory performances, and the fully supervised fashion is still the best way for pavement image classification.

In highway maintenance, the engineer will use a professional pavement vehicle to capture pavement images of the highway at regular intervals. Once they obtain the pavement images, they will find out the distressed pavements, assess the severity of the distress, and then accomplish the pavement distress statistics for a highway. The pavement distress detection and recognition all have actual application values in this pipeline. For a 120-kilometer length highway with four lanes, each inspection will produce more than 240,000 pavement images. More than 95% are actually normal pavement images that will not be further analyzed in the next step. Conventionally, the engineers must manually filter out these normal pavement images, leading to heavy labor costs. Like IOPLIN, WSPLIN enables engineers to filter out most normal images automatically, thereby significantly reducing manual labor costs. More specifically, WSPLIN often produces a confidence score for each pavement image to measure the probability that the image belongs to the diseased one. Thus, we can filter out most of the normal pavement images by setting a confidence score threshold.

Moreover, WSPLIN has wider application scenarios in comparison to IOPLIN. IOPLIN can only address the pavement distress detection problem, which is a typical binary image classification issue and attempts to find the distressed samples only. WSPLIN can tackle both the pavement distress detection and the recognition tasks under various aforementioned application settings shown in Table I. Roughly distress localization and quality pavement distress recognition can assist engineers in automatically accomplishing detailed pavement distress statistics for roads over a while. This statistics information is crucial to assess the health situations of roads and speculate on the risk elements that cause pavement distress. This information is also vital for designing a suitable pavement maintenance strategy and estimating pavement maintenance expenditure. Moreover, not all pavement distresses are cracks, and some pavement distresses may need some special analysis. The pavement distress recognition system can automatically group the distress pavement images based on their distress types, thereby facilitating the engineers to analyze the different types of pavement distress further. Figure 6 gives some examples of this scenario via visualizing the patch labels produced by the trained WSPLIN.