Meta Pattern Concern Score: A Novel Evaluation Measure with Human Values for Multi-classifiers

A number of metrics have been proposed to train and pick out the desired classifier. For one thing, the evaluation measures based on simple observations and statistics of the output results such as classification accuracy rate are widely used. Most of these measures are derived from the confusion matrix, a structure to illustrate the results obtained from a classifier [3]. Given a $c$-class classification task, let $m_{ij}$ be the number of samples actually belonging to the $i$-th class while that are classified into the $j$-th class, the confusion matrix can be defined as $\mathcal{M}=\begin{bmatrix}m_{ij}\end{bmatrix}$. Given a target positive label $r$, there are four different situations namely “True Positive” (TP $=m_{rr}$), “False Negative” (FN $=\sum_{j=1}^{c}\hskip 1.99997ptm_{rj}-m_{rr}$), “False Positive” (FP $=\sum_{i=1}^{c}\hskip 1.99997ptm_{ir}-m_{rr}$) and “True Negative” (TN $=\sum_{i=1}^{c}\hskip 1.99997pt\sum_{j=1}^{c}\hskip 1.99997ptm_{ij}-\sum_{i=1}^{c}\hskip 1.99997ptm_{ir}-\sum_{j=1}^{c}\hskip 1.99997ptm_{rj}+m_{rr}$) [11], with which a series of evaluation measures are defined as follows:

For another, the objective of model training is to find the optimal parameters $\omega^{*}$ to minimize the value of global loss function $\mathcal{L}(\omega)$ over the whole training dataset $\mathcal{X}$ [6]:

where $l(x,\omega)$ denotes the specific loss value computed from sample $x$ with parameters $\omega$. So naturally, how well the objective is met can serve as a metric of model quality. Given a model $\mathcal{N}$ having fully connected final layer with the softmax activation, the supervised labels $y$ in the format of one-hot, and the probability outputs $\hat{y}$ predicted by $\mathcal{N}$, the famous loss mean squared error (MS) and the cross entropy error (CE) commonly used in classification [12] can be denoted as:

The two kinds of metrics mentioned above both have their own advantages in contrast, while they also struggle with different dilemmas. Compared with loss value, confusion matrix-based evaluation measures have advantages in the ability of flexible customization. Through the various combinations of TP, FN, FP and TN, evaluation measures allow one to define the objective in an intuitive and direct format according to the specific application. For instance, as the over-abundance of negative examples is commonly seen in information retrieval, recommendation systems and social network analysis [13], one of the reasonable metrics in such situations is the Precision who can measure the ability of the classifier not to label as positive a sample that is negative, so that the always-negative classifier will not be over-valued.

Nevertheless, evaluation measures directly refer to the classification results, which brings some potential problems. For one thing, each element in the confusion matrix is essentially a number of classification results having specific actual and predicted labels, which means the values of all the evaluation measures based on the confusion matrix are discrete. As a consequence, they are insensitive to slight improvements in model parameters and only vary discontinuously and abruptly when the prediction results of some samples change [4]. This can be especially disappointing in the latter stages of model training. For another, evaluation measures heavily focus on the existing data, while the ideal model is expected to reliably classify previously unseen objects in the real world. So what really matters is, according to the Bayesian classification theory, how well the model approximates the underlying posterior probability distribution [14]. Researchers have shown that improving observed accuracy rates and improving posterior probability estimation are not entirely synonymous [5]. In fact, efforts to improve posterior probability estimation may yield lower accuracy rates in the known dataset, but lead to better performance on future reality tasks.

The problems mentioned above are actually the important reasons why loss is proposed [4]. Carefully designed loss functions ensure the gradient of most parameters is not $\vec{\textbf{0}}$ in the training process, so as to guide the continuous optimization of the model. And for the latter problem mentioned, the theoretical relationship between Bayesian posterior probability estimation and MS cost functions had also been explored and widely adopted [7]. However, squared error assumes Gaussian target data, which is violated given the discrete targets used to train classifiers. Besides, it can be found from (2) that MS is likely to be dominated by a few outlier data points that have particularly large errors in practice.

As a log-linear error function based on the maximum likelihood estimate approach [12] that is impacted much less by these further problems, CE gradually became more popular in classification tasks to date. However, entropy-based measures are specifically designed for binary targets at the beginning, so when CE is applied in multi-classification tasks, it has to treat all the classes except the class with the correct label as a collective concept “incorrect class”, without considering the distinction between them. What’s worse, CE merely takes the probability of the correct label in the prediction results into error calculation, which means that of the incorrect labels, which are certainly more in quantity in every single output of multi-classification and may also impact the prediction effectiveness, are totally ignored.

All in all, there is indeed an uncharted territory: is the bias between the two kinds of metrics irreconcilable? Empirically, if we consider the multi-classification task from a similar perspective as the confusion matrix, but construct a specific formula with the mathematical principle closer to the loss, it is likely that a metric can be proposed with not only the customizability just as the former but also the theoretical completeness inherited from the latter at the same time. We take this idea into consideration when specifically designing the formula of our own evaluation measure.

The two scenarios to be talked about come from a typical safety-critical task, Traffic Light Recognition, which is important regarding the traffic participants’ safety in autonomous driving [15]. Note that this task usually involves a variety of technologies in practice, but for convenience here we just simplify it as a 3-class classification problem.

The first scenario happens when a red light shows, where there are two different kinds of misclassifications according to the specific prediction result. If it is classified as a green light, the autonomous vehicle will be incorrectly allowed to proceed, which may endanger both its passengers and others, and as a result, totally unacceptable. While if the light is predicted to be yellow, which means “stop, unless it is unsafe to do so” [10], at least in most cases the car will stop and not break the traffic rules. So it is obviously inappropriate to treat both of the errors with the same severity, which is, however, what the traditional metrics do.

Since this kind of scenario with negative classes having different damages is commonly seen in practice, it would make sense to design for it. But before providing formal definitions and descriptions in section III-B, showing our idea in an intuitive way firstly might be found helpful. As mentioned previously, CE is only affected by the predicted value of the correct label. Just as shown in Fig. 1, this is marked as follows: the significance of the correct class “Red” is $+1$, while that of others is assigned to be $0$ to represent no influence. On the other hand, the MS and all the evaluation measures from confusion matrix treat the prediction as either True or False, respectively corresponding to $+1$ and $-1$ in the “Others” line.

In our idea, we mark $-\alpha$ instead of $-1$ for the incorrect prediction “Yellow” to represent it is relatively less destructive. What’s more, we assign that of the “Red” to be $-((-\alpha)+(-1))=1+\alpha$, which is basically a compromise between CE and other metrics. Specifically, we neither ignore negative classes completely nor make each of them as important as the positive class, while we set that the positive class and the whole of negative classes are of equal significance. These marked values correspond to a concept to be formally defined later in our measure called concern degree.

The second scenario is presented aiming to illustrate the significance to introduce a threshold for the concession in prediction confidence. As shown in Table I, there are two classifiers trained for a task with a training size of 100 and the loss CE. The classifier A makes a mistake in one of the input samples, predicting the probability of the correct label to be only $0.49$, while the classifier B makes correct predictions in the whole dataset. However, the confidence of all the correct predictions of classifier A is $0.99$, while that of B is $0.98$. In this case, it is indeed that the CE value of classifier A is less than that of B, but does that really mean A is a better choice? At least in our simplified Traffic Light Recognition case, the answer is no because it is the failure of prediction that is more intolerable compared with the minor weakness of confidence in such a safety-critical task. Note that although here we take CE as an instance, MS and other loss functions also have the same problem to different extents.

To deal with this problem, we can set a threshold value for the confidence and keep the model from being punished in specific samples once the confidence value of the correct label exceeds it. In our idea shown in Fig. 2, we further divide the entire confidence field into several intervals according to the granularity determined by the threshold, and calculate the punishment of a specific confidence value based on the distance between the interval it falls into and the interval representing the highest confidence.

In this section, we propose the specific definition of MPCS, and provide a detailed process to calculate its value. As shown in Fig. 3, the calculation process can be divided into three parts. The first part is about running the multi-classifier and normalizing its results to the probabilistic distribution using softmax. The second part is where the two constructors extract and build meta patterns from the processed results. Finally, in the third part, concern degrees and interval punishments are calculated given specific human values, and the MPCS on the entire dataset is accordingly acquired.

To begin with, to realize our idea, it is not sufficient enough to merely take the label with the highest confidence into account. However, considering all the labels may also lead to a lot of unnecessary computational time costs. So we introduce a hyper-parameter $k$ to indicate the top-$k$ classes in the order of confidence decreasing to be considered according to the practical task. With $l_{i}$ denoting a specific label in a classification task, the prediction pattern is defined as a vector $\textit{P}_{\textit{pred}}$ with length $k$:

where the value of $k$ is between one and the number of classes of the task.

To realize the idea described in Fig. 2, we divide the confidence field into $t$ intervals and determine the confidence level of a specific label under an input sample by the interval in which its confidence value falls and whether the label is the correct one or not. The confidence pattern is defined as a vector $\textit{P}_{\textit{conf}}$ with length $k$ that must be the same as $\textit{P}_{\textit{pred}}$ in one task:

in which $C(l_{i})$ denotes the confidence level of the label $l_{i}$. Given a hyper-parameter $t\geq 1$ and the correct label $l$, and let $c_{i}$ be the confidence of label $l_{i}$, $C(l_{i})$ can be calculated as follows:

Notice that there is a correspondence between each pair of elements in the same position of $\textit{P}_{\textit{pred}}$ and $\textit{P}_{\textit{conf}}$ in any specific sample. As a pair of the two patterns can be used to record the key information to uniquely represent any specific output of the classifier, we always refer to them together. The meta pattern is an abstract representation of the probabilistic prediction result of multi-classifiers, which is defined as the combination of the corresponding prediction pattern $\textit{P}_{\textit{pred}}$ and confidence pattern $\textit{P}_{\textit{conf}}$.

Then it is time to recall the idea shown in Fig. 1. We introduce a release list as a hyper-parameter with every single element indicating a less destructive situation in practice, as well as a factor to indicate that in what extent the concern of them could be released. In every single data sample, for every label selected into its $\textit{P}_{\textit{pred}}$, there are three kinds of values of the corresponding element in its concern degree depending on whether the label is the correct one and whether it exists in any element of release list together with the correct label if not. Given a prediction pattern $\textit{P}_{\textit{pred}}$, its concern degree can be defined as a vector $\mathcal{I}$ with the same length $k$:

Given the correct label $l$, a release list $\mathcal{R}$ with $r_{\alpha}$ representing any element of it, and a release factor $f_{\mathcal{R}}$, $D(l_{i})$ can be calculated as:

Now we can propose our evaluation measure MPCS based on the above definitions. To calculate the interval punishment, given confidence pattern $\textit{P}_{\textit{conf}}$, we select the $log$ function just as CE to convert the interval distance into a concrete punishment value. Then with the concern degree $\mathcal{I}$, the meta pattern concern score $\mathcal{S}$ of the classification result of every single sample can be calculated as:

where $P_{i}$ and $I_{i}$ respectively represents the $i$-th element in the $\textit{P}_{\textit{conf}}$ and $\mathcal{I}$, and $k$, $t$ are the hyper-parameters defined previously. An important characteristic to be noticed is that MPCS approximates CE in the limit of $k=1$ and $t\to\infty$. In other words, CE can be viewed as a special case of MPCS under the limit condition. We illustrate the whole process of MPCS calculation in Algorithm 1, including the construction of meta pattern (line 3-5, 8), the calculation of concern degree (line 6-7, 9-17) and interval punishment (line 18-20), and the calculation of the MPCS value in the whole dataset (line 21-24).

There are six datasets from four sources used in the experiments, namely the IRIS and DIGITS from scikit-learn [16], the MNIST, the pedestrian counting dataset CT and the smooth subspace clustering dataset SS from UCR time-series archive [17], and the synthetic Autonomous Driving Dataset (ADD) generated by the Scenic scenario programming language and the Carla simulator [18]. Different models, including Multi-Layer Perceptron (MLP), Convolutional Neural Network (CNN), Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM), are adopted and Adam is chosen as the optimizer uniformly. The Accuracy, $F_{1}\text{-}$score, MCC, MS and CE mentioned in section II-A are the benchmarks adopted. The experiments are implemented with Python 3.8.8, and PyTorch 1.10.1.

In this section, we evaluate MPCS in two general aspects. For one thing, through the comparison of the value and trend between MPCS and the benchmark metrics, we verify its effectiveness as an evaluation measure in the training process of multi-classifier. For another, by illustrating the time cost of MPCS calculation, we clarify that it has considerable efficiency as the benchmark metrics.

So far we have confirmed that MPCS can work as a general evaluation measure, but the particular benefits it can bring given specific human values have not been discussed. In this section, we use a case study to illustrate how to use MPCS to introduce human values into the evaluation and even the learning of the multi-classifier. We build the case with MLP and MNIST like Exp. 3, and determine the less destructive misclassifications according to the t-SNE projection [20], a technique that can plot 2D embeddings for high-dimensional datasets while keeping the distance between the samples the same. To be specific, if the incorrect prediction result of a sample is adjacent to its correct label in the projected clusters, this mistake is less destructive compared to confusing non-adjacent labels. So we pick all that kinds of mistakes, which are framed in red in Fig. 5, into the release list $\mathcal{R}$. Notice that here we just take this as an example since its meaning is relatively simple to understand, while actually the release condition can be customized according to any specific complex requirements in practice.

With all the results shown above, now we are ready to answer what can MPCS contribute to the community. For the first time, we provide a general idea to introduce two kinds of specific human values into the evaluation and even Learning of multi-classifiers. Different from common metrics having a fixed form all the time, the MPCS allows people to flexibly declare what they care more about the model in different practices, and try to cater to their specific will to pick out the optimal model, under the premise of not violating the common metrics too much and having a similar time cost as them.

On the other hand, technically speaking, MPCS is designed regarding the advantages of the existing two common kinds of metrics. Specifically, it provides an abstract view of model output to enable customization as the confusion matrix-based evaluation measures, and calculates its value in a way that approximates CE in the limit to make the picked model approximate the posterior probability distribution as much as possible like the loss. And for the negative labels, MPCS neither ignores them completely like CE, nor gives them the same importance as MS, while it adopts a compromise to avoid their shortcomings and makes the assessment more reasonable. Although MPCS can still not avoid discrete values in non-limit conditions, not only the granularity is adjustable, but also the extent of discreteness is much lower than the confusion matrix-based measures in general.