Lightweight Regression Model with Prediction Interval Estimation for Computer Vision-based Winter Road Surface Condition Monitoring

Road surface condition monitoring is essential for different vehicle applications, and the topic has been studied in the past with several approaches. Current interest in automated driving solutions has increased the importance of the field, as vehicle control algorithms require information of road surface condition to properly assess the situation. Road surface condition monitoring is especially important in winter conditions, where the road surface friction properties greatly vary. The existing methods can be divided to contact and non-contact based methods.

Contact methods are not analysed here in-detail, as this work is focused on computer vision methodology. For a more thorough overview of contact-based methods, the review by Acosta et al. [2] is recommended. Fundamentals of contact-based estimation methods are still briefly presented here. Contact-based methods generally measure the actual friction between the tyre and the road. Road surface condition can be consequently estimated from this information.

Of the contact-based methods, slip-based methods are the most common. Measuring wheel rotation information and inertial measurement unit readings, friction between the tyre and the road can be estimated [6]. Utilising this information, the road surface condition can be defined based on the friction characteristics. In a commercial vehicle, this information is readily available in the anti-lock braking unit and electronic stability control. However, slip-based methods typically cannot accurately estimate the road surface conditions until severe slippage has already occured. The review by Acosta et al. [2] notes that slip-based approaches are typically considered inadequate for reliably improving the functionality of advanced driver assistance systems. Other contact-based methods for road surface monitoring are based on vibration. Low frequency methods can utilise signals such as the vehicle rotation speed [7]. However, the approaches rely largely on slip-slope assumptions, and consequently lack robustness. High frequency methods have yielded impressive results, yet these methods rely on additional sensor equipment. Works on the topic have utilised microphones installed on the vehicle to monitor the tyres [8]. This might not be a commercially feasible solution due to the burden of additional sensor installations.

To enable alternative means for estimating road surface condition, a number of non-contact methods have been developed. Non-contact methods typically exhibit different operation characteristics and attributes compared to the contact-based methods. They offer alternative ways for the measurement, or they could be fused with contact-based methods. Review of non-contact friction estimation has been presented by Ma et al. [3]. The non-contact approaches can be roughly divided into methods that utilise special optical sensors, and methods which utilise computer vision algorithms to process images captured with traditional visible light cameras. Additionally, there have also been some studies exploring road surface condition monitoring with automotive radars [9].

With dedicated optical sensors, non-contact approaches commonly utilise infrared spectroscopy [10]. The approach is based on the different reflectance characteristics of water, ice, and snow. This measurement approach is commonly utilised, and multiple commercial products applying the method are available on the market [11, 12, 13]. Another optical technique for road surface condition monitoring is based on analysing polarisation of light [14].

Computer vision applied on regular visible light cameras has been a popular topic for non-contact road surface condition estimation. The approach is lucrative from a practical point-of-view, since modern vehicles are equipped with forward-facing cameras. Review of computer vision-based estimation has been prepared by Wu et al. [15]. Generally, computer vision approaches utilise machine learning models such as convolutional neural networks (CNNs) to analyse images of the road.

Several works utilising computer vision have applied classification techniques to assess the road surface condition. Nolte et al. [16] proposed applying CNNs to perform this classification task. They trained ResNet50 [17] and InceptionV3 [18] models to recognise six distinct categories of road surface: asphalt, dirt, grass, wet asphalt, cobblestone, and snow. Similar studies have been conducted by Šabanovič et al. [19], who developed a CNN capable of classifying the road pavement type (asphalt, cobblestone, gravel) as well as the surface condition (dry, wet). The effectiveness of the developed road surface monitoring system was demonstrated with vehicle braking tests. The tests highlighted that the stopping distance was shorter when utilising an adaptive anti-lock braking system control strategy, which was tuned based on the classification results. To extend the capabilities of road surface classification, Wang et al. [20] proposed applying a segmentation CNN to perform the classification task. Their classification task featured a total of nine different pavement and road surface condition combinations, including categories from winter conditions.

To further enhance the development of computer vision-based road surface monitoring in summer conditions, Cordes et al. [21] published an open classification dataset called RoadSaw. They collected the dataset in an automated manner, mounting an optical road surface monitoring sensor on a vehicle, as well as a forward-facing camera. As a result, a large dataset of images in realistic road conditions was captured, along with the corresponding road surface condition readings from the sensor. For further processing, the images were transformed to bird’s-eye-view and differently sized patches (2.56 $m^{2}$, 7.84 $m^{2}$, 12.96 $m^{2}$) of the road were cropped. The optical road surface monitoring sensor readings were synchronised to the image patches by utilising timestamps and vehicle velocity information. The dataset featured three different pavement types (asphalt, cobblestone, concrete), which had four different surface conditions (dry, damp, wet, very wet). The authors evaluated the performance of a MobileNetV2 [22] classifier on the data. They also proposed adding deterministic uncertainty quantification [23] functionality to the network, allowing assessment of the uncertainty of the classification predictions.

Classification-based road surface monitoring has also been extended to model the road surface in finer spatial detail. In the camera view, several road surface conditions may be visible simultaneously, such as patches of snow on an asphalt road. Providing a single classification for the entire visible road may result in inadequate assessment of the prevailing situation. Roychowdhury et al. [24] proposed a two-stage approach for detailed analysis of the road surface in winter conditions. After classifying the overall road surface condition with a CNN, they split the image of the road into a grid, in which each cell was classified separately.

The resolution of road surface condition estimation has been also extended by applying regression models to the task, instead of utilising classification models. In terms of friction-related information, regression models allow for more accurate representation of the road surface properties. This is due the model predicting a continuous value, instead of utilising a discrete number of categories labelled with certain values. Vosahlik et al. [4] proposed automated training of a CNN regression model based on corresponding friction information derived from a slip-based contact method. Based on data acquired with a 1:5 scale car model, they created a dataset of matching friction values and images, which included samples from winter conditions. Their contact-based method estimated the friction between the road and the tyres, which was used to label the images. From the images, they cropped out a region of 1.5x1.5 metres in front of the vehicle, which was transformed to bird’s-eye-view for processing. Known vehicle speed and timestamps were utilised to synchronise the friction readings and the image patches. They trained a ResNet50 network on the data to perform the regression task. Regression-based road surface friction estimation was also developed by Du et al. [25], who studied the problem in summer conditions. They collected road surface condition information with a commercial contact-based monitoring system, which utilised two measurement wheels for estimating the tyre-road friction coefficient. The measurement system was equipped on a truck, which was followed by a vehicle equipped with a camera gathering image data of the road. The test route was divided into 50 metre long sections, in which the captured images were labelled with the mean of the friction readings within the specific section. For image processing, smaller patches of the road were cropped from the road area between the regions through which the measurement wheels had traversed. In their work, they trained different CNN regression models to predict the road surface friction. The vision-based estimators were fused with another machine learning model, which processed the vehicle dynamics information. The fusion model seemed to offer clear benefits, as several key performance metrics were improved.

Uncertainty assessment has been a key topic in deep learning due to the obscurity of the data-driven the models. In order to evaluate the reliability of the estimates produced by deep learning models, several approaches have been proposed to quantify related uncertainties. An extensive review on the topic has been written by Abdar et al. [26]. Most extensive uncertainty quantification methodologies include Bayesian neural networks and ensemble techniques. Bayesian neural networks model the network weights as distributions, allowing the predictions to be accurately modelled as posterior distributions. Ensemble models rely on utilising multiple neural network models to process the input, and determining the uncertainty based on the outputs of the networks. Bayesian and ensemble methods have been proven to generally provide reliable uncertainty information. However, these methods typically require immense computational resources to operate, and applying them is often not practically feasible.

A common approach for assessing uncertainty in regression problems is the estimation of prediction intervals [27]. Several approaches have been proposed for generating prediction intervals with deep learning regression models, including the previously mentioned Bayesian and ensemble methods. However, more lightweight approaches for estimating prediction intervals have also been developed. Typically this has been achieved by modifying the neural network architecture to feature an additional output node. In order to quantify uncertainty, Nix and Weigend [28] proposed adding a separate output node with its own hidden layers to a fully connected network architecture. The additional output node was responsible for estimating the variance of each prediction on each forward pass, whereas the other output node remained responsible for producing the point estimate. Modelling each network prediction as a probability distribution, the network could be trained with a maximum likelihood loss function. In such architecture, uncertainty is quantified by the value of the variance output, which can also be used to generate prediction intervals. Somewhat similar prediction interval estimation has been proposed by Khosravi et al. [29]. Their approach was based on two output nodes, which were responsible for predicting the lower and upper bound of the prediction interval. A special loss function was used for training, which determined the target coverage of the prediction interval.

This paper aims to enhance the existing state of the art of winter road surface condition evaluation. A novel CNN architecture, SIWNet, is proposed for the task of computer vision-based estimation of road friction properties. Similarly to the work of Vosahlik et al. [4], SIWNet is implemented as a regression model. SIWNet was trained and tested based on an optical road friction sensor ground truth. Such approach has not been previously applied in the literature in winter conditions, and therefore this study presents unique results highlighting the functionality of the chosen approach.

SIWNet expands state of the art by including uncertainty quantification in the regression architecture. This is achieved by including an additional prediction head, which enables representation of the model output as a prediction interval. The approach is based on the work of Nix and Weigend [28]. Uncertainty quantification is a vital feature for computer vision-based road surface monitoring systems, as visual estimation of road surface friction properties is bound to feature varying levels of uncertainty. For example, some surface conditions may be difficult to detect with computer vision, or other road users may be partially blocking the view of the road. Cordes et al. [21] have previously proposed an uncertainty estimation approach in their classification-based work. However, studies developing regression models for the prediction task have not proposed methods for uncertainty quantification.

In addition to including an uncertainty quantification mechanism, SIWNet is also designed to feature a computationally lightweight architecture. The findings of this paper highlight that computer vision-based winter road surface friction regression is not dependent on an extensively large neural network architecture. Previous research [4] on the topic has applied a relatively large model for the task. This finding is therefore a notable improvement, considering the generally limited vehicle on-board computational resources.

The goal of the research was to develop a model capable of accurately assessing the winter-related road surface friction properties based on images captured from a typical vehicle windshield camera. Additionally, the work aimed to enhance the robustness of computer vision-based road friction regression by introducing uncertainty quantification methodology to mitigate the natural inaccuracies of visual road friction estimation. Simultaneously, the developed model must be lightweight to allow for on-board deployment with limited computational hardware. These goals were pursued by formatting the computer vision problem as a task of predicting a ground truth road friction value from a corresponding image of the road.

To acquire relevant data for the research, the publicly available SeeingThroughFog dataset [5] was utilised. The dataset has been gathered by driving an instrumented vehicle in central European and Nordic countries during winter. Part of the dataset has also been recorded in the springtime, to include summer-like conditions. The instrumented vehicle contained a forward-facing camera, as well as an optical road friction sensor. The sensor setup is illustrated in Fig. 1. In this paper, the camera images and corresponding road friction sensor readings were utilised for developing a neural network model, SIWNet, for assessing the road surface condition. SIWNet is trained to process an image of the road, and predict a value corresponding to the road friction sensor reading.

The optical friction measurement unit used in the dataset is a prototype from a widely recognised manufacturer, seemingly similar to those of the manufacturer’s other models [11, 12, 13]. Similarly to the commercial equivalents of the sensor, the unit measures the amount of snow, ice, and water on the road surface. By manufacturer software, these road surface friction properties are summarised into a single factor, grip factor, within range 0.09…0.82. This factor effectively indicates the slipperiness of the road. Here, this grip factor was normalised to a range 0.00…1.00, and called friction factor, $f$. It should be noted that the friction factor is not the friction coefficient between the tyre and the road, as determining this value depends on the tyre properties. The friction factor represents a value that can be used to estimate slipperiness of the road, and consequently the actual friction coefficient in case relevant tyre parameters are known.

The images in the dataset had a resolution of 1920x1024 pixels. To focus the analysis on the most relevant portion of the images, the road directly ahead of the vehicle, a predefined static section of all images was cropped and transformed to bird’s-eye-view. Similar pre-processing steps were utilised in [21] and [4]. The cropped section was here defined to approximately represent a short section of the lane ahead of the vehicle. A single image with clearly visible lane markings was used to find the boundaries for the cropped area. This process is presented in Fig. 2, with the cropped area represented by a red quadrilateral in the original camera image. The quadrilateral corresponds to the outline of the bird’s-eye-view representation, as highlighted in the visualisation. Based on knowledge of the dimensions of lane markings in Finland, in which the selected image was captured, the cropped section was inferred to represent an area with approximate dimensions of 11x4 metres. The consecutive bird’s-eye-view transformation of the cropped section was achieved by stretching the image to a rectangle and reshaping it into a square. The chosen boundaries for the crop operation were utilised to similarly crop each of the images in the dataset. All further processing steps were carried out on the cropped image sections.

The used dataset contained a total of 4330 samples, as this is the number of measurements in the SeeingThroughFog-dataset which contain readings from the road friction sensor. These samples were collected in different locations on 12 different days. Fig. 3 displays the number of samples collected on each date. The distribution of the friction factor values in the dataset is visualised in Fig. 4. Image samples from the dataset with corresponding friction factor readings are provided in Fig. 5. In addition, the overall road surface state reported by the friction sensor is presented for each sample.

The SIWNet model was developed with the main goal of presenting a lightweight model featuring uncertainty quantification for the friction factor prediction task. The architecture of SIWNet consists of a feature backbone, as well as a point estimate head and a prediction interval head. SIWNet architecture is presented in detail in Fig. 6. The feature backbone is responsible for processing relevant features from the images. Based on this information, the point estimate head outputs the predicted friction factor $\hat{f}$. The prediction interval head is responsible for assessing the uncertainty related to the point estimate. Based on the features and the point estimate, the prediction interval head outputs a predicted standard deviation $\hat{\sigma}$, which is utilised to establish a prediction interval around the point estimate.

The feature backbone is based on the ResNet [17] architecture, applying the same basic residual building blocks. Contrary to typical ResNet implementations, each block is used only once. This design resulted in an extremely lightweight architecture, facilitating practical applicability in embedded on-board systems. The residual blocks were utilised as the ResNet models are widely known for their excellent accuracy and generalisablity.

In the point estimate head, a single fully connected layer is applied, which is typical for regression tasks. A sigmoid activation function was added to the final output, as the friction factor values were bound between 0 and 1. The prediction interval head is also constructed of fully connected layers. These layers analyse the features provided by the feature backbone, as well as the prediction provided by the point estimate head. The prediction interval head analyses the friction factor prediction process, and quantifies the related uncertainty. At the end of the prediction interval head, a sigmoid activation was added to enhance the stability of the output. This conveniently limited the predicted standard deviation $\hat{\sigma}$ to positive numbers. The upper limit of 1 for $\hat{\sigma}$ was deemed reasonable, since with the probabilistic model applied to train the network, this corresponded to a nearly uniform distribution within the range 0…1.

The prediction interval head of SIWNet is trained to quantify the uncertainty related to the prediction of the point estimate head. Based on the quantified uncertainty, a prediction interval can be generated. In order to perform this task, the output of SIWNet is modelled as a truncated normal distribution. The goal of the training process is to maximise the likelihood of the training labels with regard to the predicted distributions. Previous work has applied similar methodology with a regular normal distribution [28]. A truncated normal distribution of a random variable $x$ has a probability density function (PDF) of the form [30]

where $\mu$ and $\sigma$ denote the mean and standard deviation of the underlying normal distribution, respectively. The lower and upper truncation bounds are denoted by $a$ and $b$, respectively. The truncated normal distribution PDF representation is based on the underlying normal distribution PDF, defined as

as well as the underlying normal distribution cumulative distribution function, defined as

where $erf(\cdot)$ is the error function. Consequently, the negative log-likelihood of the truncated normal distribution has the form

The training process of the prediction interval head of SIWNet is based on maximising the likelihood of the corresponding ground truths and predicted truncated normal distributions. Consequently, this means minimising the negative log-likelihood with the predictions. The utilised loss function for a training batch has the form

where $n$ denotes the number of samples in the batch and $f$ denotes the ground truth friction factor. The underlying normal distribution mean is the predicted friction factor $\hat{f}$ from the point estimate head, whereas the predicted standard deviation $\hat{\sigma}$ is the output of the prediction interval head. To improve stability, $\hat{\sigma}$ is thresholded to a minimum value of $1\cdot 10^{-4}$ when computing the loss.

For training SIWNet and conducting the presented experiments, the following steps were taken. The utilised dataset was randomly divided into train-validation-test sets with a respective 70%-15%-15% split. When splitting the data, timestamps were utilised to ensure that data samples gathered from the same location were not included in different sets. Hyperparameters were optimised by training the network on the training set and finding the best result on the validation set. For the evaluation on the test set, the model was re-trained on a combination set containing both the training and validation sets.

During the training, the feature backbone and point estimate head were first trained for 60 epochs utilising regular mean squared error as the loss function. After these parts of the network were trained, their weights were frozen. Afterwards, the prediction interval head was trained for 60 epochs with the loss function presented in Equation 5. Dropout with a probability of 0.5 was applied when feeding the feature backbone output to the prediction interval head during training.

Training was carried out with a batch size of 32, and stochastic gradient descent with a momentum of 0.9 was used as the optimisation method. During training, the learning rate was decayed with a step-based scheduler. Every twenty epochs the learning rate was reduced to one tenth of the previous value. As for data augmentation during training, the images were randomly flipped horizontally as well as rotated with a value from [-4,4] degrees. Furthermore, the pixel values were slightly scaled by random color jitter in the range [0.9, 1.1]. For both training and testing, images were reshaped to 324x324 pixels before being fed to the network. Additionally, the pixel values were normalised with the mean and standard deviation of the pixel values in the training set.

The presented experiments demonstrate comparison of SIWNet to regression models applied in the previous literature. ResNet50 [17] has been previously applied for winter condition road friction estimation [4], whereas ResNet50v2 [31], VGG19 [32] and EfficientNet-B0 [33] have been applied in summer conditions [25]. The same training and testing procedures used with SIWNet were applied with these models, except for steps related to the prediction interval head, which the other models do not include. A sigmoid activation was also added to the output of the these models, since this was noted to boost performance. All models were implemented on the PyTorch deep learning framework [34]. Number of trainable parameters in the models as well as floating point operations (FLOPs) executed during inference were analysed with the ptflops-tool [35].

In order to benchmark the prediction intervals and the probabilistic properties of the predictions in the experiments, interval score and continuous ranked probability score (CRPS) were utilised [36]. The 90% prediction interval was used for defining the interval score. For constructing the prediction interval based on the truncated normal distribution output of SIWNet, the 90% interval of the distribution with the highest likelihood was used. Since the compared models produce only point estimates, there was no straightforward approach to define interval scores for the models. Thus, the compared models were given static prediction intervals surrounding their point estimates. The boundaries were set at a distance from the point estimate, which corresponded to the 90% error threshold $e_{90\%}$ of the model on the validation set. The 90% error was defined as the value below which 90% of absolute errors on the validation set were located. The prediction interval boundaries were clamped by the plausible friction factor values, 0 to 1.

The neural network models were evaluated in terms of computational load by analysing the number of trainable parameters in the architectures, as well as the FLOPs required to perform inference on a single input image. The SIWNet architecture was designed to require minimal computational resources, as evident from the results presented in Table I.

Overall, SIWNet featured the lowest number of parameters of all compared models, yet utilised more FLOPs during inference than EfficientNet-B0. Compared to ResNet50, ResNet50v2, and VGG19, SIWNet featured several times fewer parameters. Additionally, compared to these models SIWNet also required notably fewer FLOPs to process an image.

Training the feature backbone and point estimate head was the initial step for training SIWNet. Hyperparameters were tuned with grid search, resulting in the initial learning rate and weight decay values of 0.1 and $1\cdot 10^{-3}$, respectively. Identical hyperparameter tuning was performed on the compared models. Table II reports the point estimate accuracies achieved by the networks when predicting friction factors on the test data. Accuracies are reported in terms of mean absolute error (MAE) and root-mean-square error (RMSE).

The presented results highlight that SIWNet achieved a nearly identical point estimate accuracy as ResNet50 and ResNet50v2. VGG19 performed slightly worse, whereas EfficientNet-B0 was clearly the least accurate model. Considering the minimal size of the SIWNet architecture, the model clearly demonstrated a favourable combination of accuracy and computational load.

The key feature of SIWNet is its capability of assessing the uncertainty of the friction factor prediction, enabled by the prediction interval head. The prediction interval head was trained while keeping the feature backbone and point estimate head frozen. Hyperparameters were optimised with grid search, tuning the initial learning rate, weight decay, as well as number of neurons in the fully connected layers of the prediction head. Based on the optimisation, the initial learning rate and weight decay were set at values of $5\cdot 10^{-4}$ and $1\cdot 10^{-3}$, respectively. Acquired average interval score results are presented in Table III for SIWNet and the compared models. As described previously, the prediction intervals of the compared models were formulated based on the 90% error on the validation set.

Based on the comparison, SIWNet clearly achieved a more favourable average interval score. This indicates that the prediction interval head was capable of assessing the uncertainty related to the friction factor predictions of the point estimate head.

As another test to evaluate the performance of the prediction interval head, the CRPS metric was used to assess the quality of the distributions predicted by SIWNet. The point estimates of the other detectors were also scored on the metric for comparison. For point estimates, the average CRPS is equal to the MAE score.

The acquired results highlight that the probabilistic forecasts of SIWNet were more representative predictions than the point estimates of the other detectors. This indicates that the prediction interval head of SIWNet did learn a functional strategy for assessing the uncertainty of the friction factor estimates.

In order to ensure that the design decisions behind the SIWNet architecture were sensible and beneficial for the performance, an ablation study was carried out. Similarly to the previously presented results, the ablation study analysed the point estimate head and prediction interval head separately.

The point estimate head was modified by removing the sigmoid activation, and the related training and testing procedures were repeated. Identical actions were taken with the compared models, removing the sigmoid activations from their outputs. During testing, the outputs of the networks were clamped between 0 and 1. Resulting accuracies on the test set are presented in Table V. Based on the results, it is evident that the sigmoid activation was a beneficial addition for all networks.

Ablation study of the SIWNet prediction interval head also investigated the effect of the sigmoid activation. The sigmoid activation was removed from the prediction interval output, and training and testing were repeated. In another test, the prediction interval head was trained and tested without using dropout regularisation on the feature backbone output. Additionally, the SIWNet model was studied by training the model without the proposed technique of pretraining the feature and backbone and point estimate head, and freezing their weights. Instead the whole network was optimised simultaneously utilising the loss function presented in Equation 5. Finally, the overall efficacy of the prediction interval head was evaluated by formulating a static prediction interval around the point estimates based on the 90% error on the validation set, similarly to the prediction intervals used for the predictions of the compared models. The ablations were evaluated with the interval score, and results from the tests are presented in Table VI. All tests concluded that the originally proposed SIWNet architecture was capable of scoring higher interval scores.