Superior Scoring Rules for Probabilistic Evaluation of Single-Label Multi-Class Classification Tasks

The classification decision is commonly partitioned into categories of true positive, true negative, false positive, and false negative. Let us examine an illustrative scenario involving a true vector of $[0,1,0]$ and two predicted vectors: $A=[0.33,0.34,0.33]$ and $B=[0.51,0.49,0]$. Vector $A$ corresponds to a true negative prediction, while vector $B$ represents a false negative prediction. It is crucial to critically evaluate which vector demonstrates superior performance, taking into account that the $\arg\max$ of vector $A$ accurately predicts the outcome, while vector $B$ assigns a higher probability to the correct class. Upon analyzing the performance of vectors $A$ and $B$ in a single-label scenario, it becomes evident that vector $A$ possesses an advantage over vector $B$ in terms of classification accuracy. This attribute holds significant importance as accurate decision-making is highly desirable and essential in numerous applications. Consequently, considering the paramount significance of precise decisions, it can be concluded that vector $A$ surpasses vector $B$ in quality. Therefore, this paper advocates that the scoring rule ought to consistently assign better scores to decisions about true positives and true negatives. However, a scientific discrepancy arises when considering the Brier Score and Logarithmic Loss metrics, as they occasionally contradict this advocated principle. Surprisingly, vector $A$ exhibits higher Brier Score and Logarithmic Loss values compared to vector $B$ contrary to expectations. This inconsistency between the advocated principle and the observed outcomes highlights a scientific gap or inconsistency that this paper aims to address and resolve. Therefore, this paper proposes a novel attribute for strictly proper scoring rules to tackle this issue. Our contributions can be stated as the following:

• •

  It hypothesizes that evaluation metrics should favor accurate predictions (true positives and true negatives) over misclassified ones (false positives and false negatives) when evaluating multi-class single-label classifications.

• •

  We show traditional strictly proper scores like Brier Score and Logarithmic Loss can rate incorrect predictions higher than correct ones.

• •

  A new property termed superior is defined for scoring rules to formally establish that evaluations should preference true predictions.

• •

  To this end, this research proposes two novel scoring rules called Penalized Brier Score (PBS) and Penalized Logarithmic Loss (PLL).

• •

  This research modifies The scoring rules Brier Score and Logarithmic Loss by incorporating a penalty term to make them superior. This penalizes incorrect predictions more strongly than the original scoring rules, aligning with the initial hypothesis that favors correct decisions over wrongs.

• •

  The utility of the proposed evaluation metrics for early stopping and checkpointing is experimentally assessed.

• •

  Experimental results demonstrate that the proposed evaluation metrics enhance the efficiency of the classifier. Furthermore, these metrics exhibit a high correlation with the F-1 score while encompassing the measurement of prediction uncertainty and calibration.

The paper is structured in the following way: Section 2 provides background information and sets the stage for the research. Section 3 reviews previous work related to the topic. Section 4 details the motivation for developing new scoring rules and introduces the proposed Penalized Brier Score and Penalized Logarithmic Loss methods. Section 5 presents experimental results obtained from spatio-temporal data. Finally, section 6 concludes the paper.

A scoring rule $S(\cdot,\cdot)$ is a bivariate function that takes two arguments: a probability measure, representing a forecast, and an observed outcome [21]. The function returns a real number that measures the deviation between a forecast probability and reality. They provide a summary measure for the evaluation of predictions that assign probabilities to events, such as probabilistic classification of a set of mutually exclusive outcomes or classes [22]. Scoring rules can be thought of as cost function or loss function and are evaluated as the empirical mean of a given sample, simply called score [17]. It is worth noting that in neural networks, loss functions are typically utilized during the backpropagation process to optimize the network. However, in this research, we are focusing on evaluation metrics for assessing model performance.

In single-label multi-class classification, the objective is to predict the specific class label, denoted as $\omega$, for a given sample characterized by a feature vector, denoted as $X$. In probabilistic classification, the goal is to learn the entire conditional distribution $p(\omega|X)$, which represents the probability distribution over classes conditioned on the given features [23]. This is accomplished through the use of a probabilistic classifier denoted as $f:\mathcal{X}\rightarrow\mathcal{F}$. The set $\mathcal{F}$ of probability measures is typically identified with the probability $c$-simplex $\mathcal{C}=\{p\in[0,1]^{c}|\sum_{i=1}^{c}p_{i}=1\}$ and probability distributions are represented by vectors $p\in\mathcal{C}$. Therefore, the overall form of the scoring rules for single-label multi-class classification problems is $S(p,i)$, where $p$ is a probability vector obtained from the classifier and $i$ is the true class.

In practice, competing forecast procedures are ranked by the average score [17]. Suppose that we wish to fit a parametric model $P_{\theta}$ based on random samples $X^{(1)},\cdots,X^{(n)}$. To estimate $\theta$, we might measure the goodness-of-fit by the mean score [24]

where $S$ is a strictly proper scoring rule. If $\theta_{0}$ denotes the true parameter, asymptotic arguments indicate that $\arg\min_{\theta}$ $S_{n}(\theta)\rightarrow\theta_{0}$ as $n\rightarrow\infty$ [17]. Let’s suppose $S$ is a negatively oriented scoring rule like the Brier Score. It suggests a general approach to estimation: choose a strictly proper scoring rule that is tailored to the problem at hand that minimizes $S_{n}(\theta)$ over the parameters space and take $\hat{\theta}_{n}=\arg\min_{\theta}S_{n}(\theta)$ as the optimum score estimator based on the scoring rule [17]. For positively oriented scoring rules like accuracy, $\hat{\theta}_{n}=\arg\max_{\theta}S_{n}(\theta)$.

The primary objective of any classifier is to accurately assign observations to their respective classes. Observations that are correctly classified are considered to be of greater value than those that are incorrectly classified. A classifier that can achieve a high rate of accurate classifications and a low error rate is deemed superior because it indicates a greater ability to identify patterns and differentiate between classes. Conversely, observations that are incorrectly classified suggest that the classifier has not accurately modeled the relationship between features and classes. These errors compromise the classifier’s capacity to generalize and make accurate predictions on new samples. Correct classifications represent successful outcomes for the classifier, while errors indicate failures. Therefore, it is crucial to maximize correct classifications and minimize errors in order to develop an effective and useful classifier. The reliability of a classifier’s performance is determined by its ability to consistently reproduce the correct labels instead of settling for inferior results that are prone to mistakes.

Fig. 1 shows the big picture of the theoretical analysis presented in this section. The subsequent part of this section demonstrates that both Brier Scoring and Logarithmic Loss are indifferent to this preference. To prove this point, a novel attribute for scoring rules is introduced. This attribute guarantees that the scoring rule assigns a superior score to the observations that are correctly classified.

According to the Monte Carlo simulations provided in Appendix 8, it is shown that Brier Score outperforms Logarithmic Loss. However, to ensure that the scoring rule consistently assigns a higher score to accurately classified observations, a new criterion is necessary, which can be attained by modifying the current scoring rules. Specifically, if the probability vector belongs to set $\xi$, an extra penalty was integrated into the scoring rule. Since this research focuses on the Brier Score and Logarithmic Loss, which should be minimized to be optimal, our aim is to assign a penalty to incorrect predictions that the penalty value is higher than the highest score possible for a correct prediction. To accomplish this, the penalty should be established as equivalent to the maximum value of the scoring rule for set $\psi$.

Based on the highest possible scores for the Brier Score and Logarithmic Loss when predictions are incorrect, an adjusted form of these scoring rules can be defined. The modified Brier Score with the penalty term, Penalized Brier Score (PBS), can be expressed as:

The modified Logarithmic Loss with the penalty term, Penalized Logarithmic Loss (PLL), can be expressed as:

where $y$ is the ground-truth vector, $q$ is the predicted probability vector by the probabilistic classifier, and $c$ is the number of classes. As shown in the following theorems, the two proposed evaluation metrics are not only strictly proper scoring rules but also superior.

Penalties in Eq. (4) and Eq. (5) are designed to be applied only to incorrect predictions. Therefore, these penalties depend on the values of $p$ in Eq. (1) and Eq. (2). Furthermore, the more number of incorrect predictions by $\theta$ in Eq. (3), the worse score it receives. As a result, the modified evaluation metrics can more reliably identify optimal model checkpoints and early stopping points that achieve better generalization performance compared to the original metrics like Brier Score and Logarithmic Loss.

In this section, we introduce the vectorization methods of the proposed evaluation metrics. The pseudocode of the Penalized Brier Score and the Penalized Logarithmic Loss is demonstrated in Algorithm 2 and Algorithm 3, respectively. The PBS and PLL algorithms leverage Penalizing Algorithm 1 to incorporate penalties into the Brier Score and the Logarithmic Loss.

The Penalizing algorithm detects wrong predictions and penalizes them. The output is in the form of a vector, where $0$ means that the prediction was correct, otherwise the prediction was wrong and was penalized. This payoff vector is then used by other scoring functions to calculate performance metrics. It takes as input predicted probabilities $q\in[0,1]^{n\times c}$, ground-truth labels $y\in\{0,1\}^{n\times c}$, and a penalty value for wrong predictions. It first computes the hot values for each sample, which are the predicted probabilities for the correct class. It then subtracts the hot values from the predicted probabilities to obtain a container of values that are positive for incorrect predictions. All negative values in the container are set to zero, and the sum of the remaining values is computed for each sample. The penalty value is then multiplied by a vector of ones to become a vector. Finally, all positive values in the container are replaced by penalty and the container is returned as the payoff for each sample.

The PBS algorithm takes as input predicted probabilities $q$ and ground-truth labels $y$. It first computes the penalty factor as $(c-1)/(c^{2})$, where $c$ is the number of classes. It then calls the Penalizing algorithm to compute the payoff for each sample. Next, the Brier scores are computed as the mean squared difference between the predicted probabilities and the ground-truth labels. Finally, the PBS is computed as the mean of the sum of the Brier scores and the payoffs.

The PLL algorithm is similar to the PBS algorithm, but it uses the Logarithmic Loss as the base score instead of the Brier Score. The Logarithmic Loss is computed as the mean of the sum of the element-wise product of the ground-truth labels and the negative logarithm of the predicted probabilities. Finally, the payoff is subtracted from the Logarithmic Loss before taking the mean.

Scoring rules are commonly used in the field of spatio-temporal statistics to compare different models [33]. To assess the effectiveness of our proposed method, we have selected several spatio-temporal datasets. Classifying spatio-temporal data is generally challenging, as evidenced by the low accuracy scores achieved on these datasets. For this reason, thorough model evaluation takes on greater importance. Additionally, it is crucial to consider incorrect classes when the classifier’s ability to generalize is limited. While the probabilities related to incorrect classes may not be particularly informative if the probability of the true class for all observations is above 0.5, it becomes increasingly important when the probability of the true class is below 0.5 for many observations. As a result, any evaluation method must take into account incorrect classes when making decisions.

The proposed model was evaluated using various classification problems, as described below. The datasets used for evaluation included three-axis accelerometer signals obtained from participants’ activities such as running or lying, as described in [34, 35, 36, 37]. While these datasets were intended for research purposes related to activity recognition, they also presented challenges for identifying individuals based on their motion patterns. Also, the dataset presented in [38] was generated for predicting motor failure time using three-axis accelerometer data. The dataset in [39] provided information about power consumption such as temperature, humidity, wind speed, consumption, general diffuse flows, and diffuse flows in three zones of Tetouan City, which is suitable for predicting a zone based on consumption information. The dataset presented in [40] includes hourly air pollutant data such as PM2.5, PM10, SO2, NO2, CO, O3, temperature, pressure, dew point temperature, precipitation, wind direction, wind speed from 12 air-quality monitoring sites, and the proposed model predicted the sites based on their air-quality information. The location of participants was collected using two received signal strength indicator (RSSI) measurements, as described in [41]. While the primary task was indoor localization, the proposed model aimed to identify participants based on their location. Finally, the dataset presented in [42] included three-axis accelerometer and three-axis gyroscope data from ten drivers, and the goal was to identify drivers based on their driving behaviors.

Since the purpose of this evaluation is to analyze two proposed metrics, the classification model used is a Convolutional Neural Network (CNN). CNNs are widely used for spatial and temporal data modeling tasks [2]. The Fig. 2 illustrates the architecture of the proposed CNN model. As neural networks are optimized through iterative training procedures, the proposed metrics can be systematically tested during this process. Model checkpointing saves model weights periodically during training. This allows rolling back to previous checkpoints if overfitting or divergence occurs. Early stopping monitors validation performance and stops training if no improvement is seen for a set number of epochs, preventing overfitting. Implementing these techniques with the proposed metrics provides insights into their behavior at different stages of neural network optimization.

When dealing with spatio-temporal data, it is essential to employ a cross-validation technique that accounts for the temporal dependencies within the data. This is crucial because the data exhibits interdependence, and employing a random sample selection for training and testing could introduce biases in the results [43]. $h$-block cross-validation is a type of temporal cross-validation that is often used for temporal data [43]. It helps address the issue of data leakage that can occur in standard $k$-fold cross-validation when applied to time series. The Algorithm 4 shows the pseudocode of $h$-block CV. The time series data was first divided into $h$ non-overlapping blocks, each representing a continuous time interval. For each block $i$, a validation set $B_{v}$ is created by selecting a contiguous subset of $nv$ blocks starting from block $i$. A test set $B_{ts}$ is then created by selecting a contiguous subset of $nt$ blocks starting from the end of the validation block. The remaining blocks are assigned to the training set $B_{tr}$. Within each block, windowing techniques were applied to segment data. The model is trained on the segments of the training set $B_{tr}$ and validated on the segments of the validation set $B_{v}$. The trained model is then evaluated on the segments of the test set $B_{ts}$. For each iteration, 50% of the subsets are allocated for training, 30% for testing, and 20% for validation. By adopting this approach, the model is trained on a sufficiently large dataset that enables effective generalization to new data, while the testing phase utilizes an independent dataset.

The validation set helps guide hyperparameter tuning, such as selecting the optimal window length and overlap values of the sliding window evaluated via grid search. Table 1 summarizes the key attributes of each dataset used in the experiments. It also allows assessing model checkpointing and early stopping criteria by tracking validation performance over training epochs. Once hyperparameters are fixed, the final model performance is reported on the independent test sets from each fold. This rigorous evaluation process provides robust estimates of the proposed metrics. The CNN model is implemented using Python and the TensorFlow library. Nadam is employed to optimize network parameters. The results of the experiments can be accessed via GitHub.

To ensure a thorough evaluation of the proposed evaluation metrics, the model training incorporated model checkpointing (CP) and early stopping (ES) techniques, considering both traditional evaluation metrics and the proposed metrics. CP were saved during training at epochs where either the traditional metrics (such as Brier Score and Logarithmic Loss) or the proposed metrics demonstrated the best performance on the validation set. ES was implemented as well, terminating training if there was no improvement in either the traditional metrics or the proposed metrics for a specified number of epochs, thus preventing overfitting. To enhance the robustness of the results, this entire process was repeated $100$ times to identify the optimal model hyperparameters. Such an approach enabled a fair comparison between the proposed metrics and the sole use of traditional metrics in determining the best model checkpoint. Subsequently, the checkpoint yielding the highest performance, as indicated by each metric, was evaluated on the test set, providing an accurate assessment of the proposed metric’s ability to identify the model with the highest true performance.

The F1 score and accuracy are widely recognized as a prominent evaluation metric for classification tasks. To effectively evaluate model performance on such unbalanced datasets, accuracy alone is not a suitable metric [44]. As most of the real-world datasets are either heavily or moderately imbalanced, accuracy can be misleading when the number of samples is very different between classes [45]. For this reason, the macro-averaged F1 score is also reported. F1 score considers both precision and recall, providing a better sense of classification effectiveness on unbalanced problems [1]. However, it is important to note that this metric solely serves as a proper scoring rule and does not encompass the measurement of forecast uncertainty and overconfidence [11]. In order to address this limitation, it is proposed that strictly proper scoring rules be employed, as they possess the capability to gauge uncertainty. By exhibiting behavior similar to the F1 score while also measuring uncertainty, these proposed scoring rules can effectively fulfill both reliability and accuracy requirements. Therefore, incorporating such scoring rules in the evaluation process would provide a comprehensive assessment of classification models, encompassing not only their predictive accuracy but also their ability to quantify uncertainty.

Fig. 3 presents validation statistics over training epochs for each dataset, including the macro-averaged F1 score, Brier Score, and proposed Penalized Brier Score. This provides an illustrative example of how the different scoring rules change during model optimization. The Pearson correlation ($Cor$) between each scoring rule and the F1 score is also shown. When plotting and comparing the different validation metric trends together, it’s important to note that they are oriented in different directions. Specifically, the F1 score increases positively as the model improves, while the PLL and PBS decrease negatively as performance increases. Therefore, to enable the results to be interpreted more easily, the PBS values graphed have been multiplied by -1 to match the positive F1 score trend.

Importantly, the figure also marks the optimal point on each trend with a data point symbol. This optimal point corresponds to the epoch at which the metric reached its highest validation value. By pinpointing these peaks, we can see exactly where each scoring rule reached its best performance relative to the others. Notably, the PBS trends exhibit optimal points that are closer to the F1 score points compared to the standard Brier Score. This observation suggests that, in these specific examples, the PBS was more suitable for model checkpointing. Additionally, the slopes of the PBS trends display greater similarity to the F1 score trends versus the standard Brier Score slopes. Additionally, the negative correlation metric confirms the PBS consistently maintains a stronger relationship with the F1 score than the standard Brier Score. Therefore, given these observations, implementing early stopping based on the PBS rather than the Brier Score has the potential to yield models with improved test performance.

The previous discussion examining superior scoring rule trends during training provided initial insights into how the proposed metrics may help optimize model performance. First, the optimal points of the PBS trends were closer to those of the F1 score trends compared to the standard Brier Score. Additionally, the negative correlation between the PBS and F1 score was consistently higher. Given this observation, it was expected that employing the PBS for model selection could lead to improved performance. To decisively validate their effectiveness, rigorous quantitative evaluation was carried out using multiple experiments.

The first stage involved assessing the correlation between the metrics and the F1 score on validation data using k-fold cross-validation. To this end, the performance of models selected with different scoring rules using both model checkpointing (CP) and early stopping (ES) are evaluated. As shown in Table 2, Pearson correlation coefficients were calculated between each of the scoring rules and the macro-averaged F1 scores across datasets and folds. In this table, the symbol $Cr_{x}$ represents the correlation between the F1 score and the scoring rule $x$. Strong positive correlation within the $-1$ to $1$ range indicates close agreement between trends. The results demonstrate that PBS and PLL trends exhibited the highest degree of similarity to F1 score variations. With a correlation exceeding other scores, PBS and PLL can be reliably used to track changes in model performance.

Next, the ability of the proposed superior scoring rules to select high-scoring models was evaluated through k-fold cross-validation. Table 3 compares the macro-averaged F1 scores of the optimized classifier chosen by each metric on test data. These F1 scores were determined through ES or CP, which relied on the utilization of a superior scoring rule. The symbol $F1_{x}$ denotes the F1 score achieved when employing the scoring rule $x$. As depicted in the table, it is evident that the F1 score consistently performs better when the model is selected using each of the proposed superior scoring rules. This observation emphasizes the effectiveness of the superior scoring rules in identifying models that yield higher F1 scores. Furthermore, consistently better scores emerged when PBS guided selection rather than PLL.

Consequently, these findings underscore the importance of employing appropriate superior scoring rules to circumvent shortcomings of F1 score alone in model selection, ultimately enabling improved classification capability and more trustworthy predictions. Collectively, the correlational and benchmark results provide compelling quantitative evidence that proposed penalties within strictly proper scoring rules augment their capacity to reflect true performance changes, in addition to facilitating the identification of models with stronger predictive power for new samples. Therefore, by more faithfully reflecting F1 score behavior, the proposed criteria enhance optimal model evaluation, selection, and classification for challenging spatio-temporal applications.

It is possible to demonstrate that the variance of the non-hot part has the greatest impact on the score value of the Brier score function. The Brier Score for $q$ is given by the following expression:

where the index of $i$ represents the true class or $i=\arg\max\>y$. The non-hot part can be expanded in the following manner:

If $\tilde{q}$ is the average of the non-hat part, then:

It is evident that:

Consequently, it can be obtained that:

where $i=\arg\max\>y$. Therefore, based on Eq. (16), the Brier Score in Eq. (18) can be minimized by reducing the variance of the non-hot part and maximizing $q_{i}$.

Furthermore, $q$ may have $t$ non-hot values that exceed $\alpha$, and the sum of these $t$ non-hot values is equal to $t\alpha+\delta$. The following expression is utilized to formulate the sum of $c-t$ non-hot values that are less than $\alpha$:

This implies that the sum of the non-hot values is equivalent to $(t\alpha+\delta)+(1-(t+1)\alpha-\delta)$. As $\delta$ increases and approaches $1$, the difference between $t\alpha+\delta$ and $1-(t+1)\alpha-\delta$ also increases, leading to an increase in the variance of the non-hot part in Eq. (18). Therefore, it can be inferred that the variance of the non-hot part of $q$ is greater than $x$. Let $\tilde{x}$ represent the average of the non-hat part of $x$:

Now, as $q_{i}=x_{i}=\alpha$ and $\tilde{x}=\tilde{q}=\frac{1-\alpha}{c-1}$, we can proceed as follows:

Thus, the condition $S_{BS}(q,i)>S_{BS}(x,i)$ still holds even if a non-hot value is only slightly greater than $\alpha$. And $S_{BS}$ is not superior in this case.

To verify the validity of the condition of being superior, a Monte Carlo simulation can be conducted, which involves comparing $S_{BS}(q,i)$ to $S_{BS}(x,i)$ for numerous randomly generated values of $x\in\psi$ and $q\in\xi$. The variables in this simulation include $x$, $q$, $\alpha$, $\beta$, and $c$, where $\alpha$ denotes the hot value of $x$, $\beta$ represents the difference between the hot value of $q$ and $\alpha$, and $c$ represents the number of classes. All of these variables are randomly generated from a normal distribution.

For each comparison, a random $x$ is selected from $\psi$ such that $x_{i}=\alpha$ and $x\in\mathbb{R}^{c}$, where $\alpha$ represents a hot value and $c$ represents the number of classes. Next, a random $q$ is chosen from $\xi$ such that $q_{i}=\alpha-\beta$, where $\beta$ is a random positive value. The pair $(x,q)$ is then evaluated to determine whether $S_{BS}(q,i)>S_{BS}(x,i)$ holds.

Figure 4 presents the results of the Monte Carlo simulation for varying numbers of comparisons. The figure demonstrates the convergence of the Monte Carlo method and confirms that the condition $S_{BS}(q,i)>S_{BS}(x,i)$ is satisfied in almost 99.996% of comparisons. The figure displays the cumulative percentage of comparisons in which the condition $S_{BS}(q,i)>S_{BS}(x,i)$ holds. When $q_{i}<\alpha$, the Brier Score typically assigns a higher score to the observation $q$; however, this is not always the case. Therefore, the Brier Score cannot be regarded as superior.

To validate the condition of being superior, it is possible to conduct another Monte Carlo simulation, which involves comparing $S_{BS}(q,i)>S_{BS}(x,i)$ for a large number of randomly generated values of $x\in\psi$ and $q\in\xi$. The simulation includes variables such as $x$, $q$, $\alpha$, $\beta$, and $c$, where $\alpha$ represents the hot value of $x$, $\beta$ represents the difference between the hot value of $q$ and $\alpha$, and $c$ represents the number of classes. All of these variables are randomly generated from a normal distribution.

In each comparison, a random value of $x$ is chosen from $\psi$ such that $x_{i}=\alpha$ and $x$ belongs to $\mathbb{R}^{c}$, where $\alpha$ denotes a hot value and $c$ represents the number of classes. Then, a random value of $q$ is selected from $\xi$ such that $q_{i}=\alpha+\beta$, where $\beta$ is a random positive value. The pair $(x,q)$ is evaluated to determine if $S_{BS}(q,i)>S_{BS}(x,i)$ is true.

Figure 5 depicts the outcomes of the Monte Carlo simulation for different numbers of comparisons, indicating the convergence of the Monte Carlo method. The figure illustrates the cumulative percentage of comparisons where the condition $S_{BS}(q,i)>S_{BS}(x,i)$ is satisfied. The results confirm that the condition $S_{BS}(q,i)>S_{BS}(x,i)$ holds in roughly 63% of comparisons. When $q_{i}>\alpha$, it is common for the Brier Score to assign a higher score to the observation $x$; however, this is not always the case. Hence, the Brier Score cannot be considered superior.