Kernel Inversed Pyramidal Resizing Network for Efficient Pavement Distress Recognition

Pavement distress recognition is an image classification task to classify the images of damaged pavements into different distress categories. Let $X=\{x_{i}\}_{i=1}^{n}$ and $Y=\{y_{i}\}_{i=1}^{n}$ be the pavement images and their corresponding labels respectively. $y_{i}$ is a $C$-dimensional one-hot vector, where $C$ is the number of pavement distress categories. $y_{ij}$ represents the $j$-th element of $y_{i}$. If the $j$-th element is the only non-zero element, it indicates that the corresponding pavement image has the $j$-th type of pavement distress. The goal of pavement distress recognition is to train a PDR model to recognize pavement distress in a given pavement image.

In deep learning-based PDR, the high-resolution images are often resized into a fixed size to meet the input or efficiency requirement of these models. Moreover, some studies also show that exploiting the scale information of pavement images can benefit pavement distress recognition. Thus, image resizing is an inevitable process in pavement distress recognition based on deep learning. However, the conventional linear interpolation-based image resizing process is independent of the optimization of the pavement distress model, and, thereby, often causes the loss of discriminative information.

To address this issue, we propose an end-to-end network named KIPRN for training to resize the pavement images and, thereby, aiding the PDR model. KIPRN can be plugged into any deep learning-based image classification models as a pre-network and optimized with these models. The pavement distress recognition process based on the KIPRN can be represented as follows,

where $\Gamma(\cdot)$ and $P(\cdot)$ are the mapping functions of the KIPRN and the PDR model respectively, while $\phi$ and $\theta$ are their corresponding parameters. As shown in Figure 1, the KIPRN consists of three modules, namely Image Resizing (IR), Pyramidal Convolution (PC) and Kernel Inversed Convolution (KIC). On the one hand, the original pavement images will be resized into different sizes through IR. On the other hand, PC extracts the features of original pavement images from different granularities, and then KIC mines the scale information of the extracted features with different branches via inversed kernels. The mined information will be compensated into the resized images to yield a three-layer image pyramid for each pavement image. Finally, the produced image pyramid will be input into the subsequent deep learning-based PDR model to accomplish the label inference.

The Image Resizing (IR) module is used to resize the original pavement image into $m$ different sizes. Here, we set $m=3$. This process can be represented as follows,

where $H(\cdot)$ is any chosen traditional image resizing algorithm, and $\mathcal{I}=\{I_{j}\}_{j=1}^{m}$ is the collection of resized images. $I_{j}$ is the $j$-th resized pavement image. We followed [24] and adopted bilinear interpolation as the image resizing algorithm. The KIPRN will compensate the subsequently learned discriminative and scale information into $\mathcal{I}$ to generate a pavement image pyramid that is more conducive for a deep learning-based PDR model to correctly recognize pavement distress.

The next step of the KIPRN is to leverage a Pyramidal Convolution (PC) module to mine and preserve the relevant information of the original images from different feature granularities. The golden dash-line rectangle in Figure 1 shows the details of the PC module, which is a two-layers of pyramidal convolution [9]. The first layer adopts three convolution kernels, whose sizes are $3\times 3$, $5\times 5$, and $7\times 7$ respectively, while two convolution kernels of the second layer are $1\times 1$ and $3\times 3$ respectively. Let $Q(\cdot)$ be the mapping function of the PC module where $\eta$ is its corresponding parameter. The pyramidal feature map can be generated as follows,

which sufficiently encodes the detailed features of pavement images under different granularities.

Once we obtain the feature map $f$, a Kernel Inversed Convolutional (KIC) module is designed to better exploit the scale information of pavement images by enlarging the differences in the receptive fields in different convolution branches, as shown in the red dash-line rectangle in Figure 1. In KIC, the feature map $f$ is resized into $m$ different sizes, which are identical to the ones of those resized images in the image resizing module,

where $\hat{F}=\{\hat{f}_{j}\}^{m}_{j=1}$ is the collection of the resized feature maps, and $\hat{f}_{j}$ is the resized feature map corresponding to the resized image $I_{j}$. Thereafter, these resized feature maps $\hat{F}$ are fed into different convolution branches to produce the information compensations for different resized images, and yield the final image pyramid. The whole image pyramid generation process can be represented as follows,

where $S=\{s_{j}\}_{j=1}^{m}$ is the generated image pyramid, $R(\cdot)$ is the mapping function of the Kernel Inversed Convolution (KIC) module, and $\zeta$ is its learned parameters.

To better retain the discriminative information of pavement images across different scales, we adopted an idea from  [4], and elaborate a series of kernel inversed ResBlocks for feature learning. In these ResBlocks, as shown in the orange dash-line rectangles in Figure 1, the smaller convolution kernel is applied to the larger feature map, which enables the mining of the local detailed information of images, while the larger convolution kernel is applied to the smaller feature map, which enables the capturing of the global structural features of images. In other words, the kernel inversed convolution enlarges the perception range in the scale space, and thereby preserves more conducive information for the solution of the subsequent task. In this manner, the whole KIPRN process is a composition of these modules, $\Gamma=H\circ Q\circ R$, where $\theta=\{\eta,\zeta\}$.

In the final step, the generated image pyramid was input into an image classification network $P_{\phi}(\cdot)$ to predict the distress label of a given pavement image as follows,

The KIPRN was optimized with the image classification network together in an end-to-end manner. Let $\mathcal{L}(\cdot,\cdot)$ be the loss function of the image classification network. The optimal parameters of the KIPRN and the image classification network can be obtained by solving the following programming problem,

In this study, we chose three well known Convolutional Neural Networks (CNNs), namely EfficientNet-B3, ResNet-50 and Inception-v1, as the image classification networks to validate the effectiveness of the KIPRN.

A large-scale pavement dataset named CQU-BPDD [26] was employed for evaluation. It consists of 43,861 typical pavement images and 16,795 diseased pavement images and includes seven different types of distresses, namely alligator crack, crack pouring, longitudinal crack, massive crack, transverse crack, raveling, and repair. Since pavement distress recognition is a follow-up task of pavement distress detection, we only used the diseased pavement images for training and testing. For a fair comparison, we followed the data split strategy in [13], so that 5,140 images were selected for training while the remaining ones were used for testing.

As shown in Table 1, we chose three traditional shallow learning methods, five well-known CNNs, two classical vision transformers and WSPLIN-IP as baselines. For all deep learning models, we adopted AdamW [16] as the optimizer and the learning rate was set to 0.0001. The input images were resized by the KIPRN into three resolutions, $300\times 300$, $400\times 400$, and, $500\times 500$, to yield the image pyramid. Recognition accuracy is used as the performance metric following [13].

Table 1 tabulates the pavement distress recognition accuracies of different methods on the CQU-BPDD dataset. The results show that the KIRPN significantly boosts all three chosen CNN-based image classification models. The KIPRN improves the the accuracy of EfficientNet-B3, ResNet-50 and Inception-v1 by $7.5\%$, $11.5\%$ and $7.7\%$ respectively. Moreover, EfficientNet-B3 enhanced by the KIPRN achieved the best performance among all the methods. As shown in Figure 2, the Class Activation Mapping (CAM) of the EfficientNet-B3 enhanced by the KIPRN is better than that of the original EfficientNet-B3. Another interesting observation regarding the multi-scale approaches is they often perform better than their single-scale versions, even though they simply use bilinear interpolation to resize images. The second best performed method, WSPLIN-IP, another multi-scale approach, adopted EfficientNet-B3 as its backbone networks. However, EfficientNet-B3 enhanced by KIPRN achieved 2.4% accuracy gains over WSIPLIN-IP under the same experimental settings. Overall, the multi-scale approaches based on the KIPRN consistently outperformed the versions based on the bilinear interpolation. These results clearly reveal two facts, namely 1$)$ exploiting scale space of images can improve pavement distress recognition, and 2$)$ the KIPRN enables learning conductive information for distress recognition during image resizing.

Table 2 tabulates the impacts of different convolution settings on the performance of the KIPRN deployed on EfficientNet-B3. These experiments were conducted on KIPRN deployed on EfficientNet-B3. The results show that the kernel inversed strategy outperforms all the other ResBlock settings. This verifies that our strategy can capture more information in the scale space by enlarging the differences in relative receptive fields. Another interesting observation is that putting pyramidal convolution on the first two layers led to the best performances. We attribute this to the fact that pyramidal convolutions are more capable of capturing low-level visual details than learning the abstract semantic features that the last layers are designed for.

Figure 3 reports the training times of different models per epoch. The observations show that EfficientNet-B3 enhanced by the KIPRN (ours) has similar training efficiency compared to the single-scale EfficientNet-B3, while enjoying the 10X training speeds over WISPLIN-IP, which is also a multi-scale approach.