Deep Neural Networks for Rank-Consistent Ordinal Regression Based On Conditional Probabilities

Ordinal regression is a classic problem in statistics, going back to early proportional hazards and proportional odds models [13]. To take advantage of well-studied and well-tuned binary classifiers, the machine learning field developed ordinal regression methods based on extending the rank prediction to multiple binary label classification subtasks [9]. This approach relies on three steps: (1) extending rank labels to binary vectors, (2) training binary classifiers on the extended labels, and (3) computing the predicted rank label from the binary classifiers. Modified versions of this approach have been proposed in connection with perceptrons [3] and support vector machines [22, 17, 2]. In 2007, Li and Lin presented a reduction framework unifying these extended binary classification approaches [9].

In 2016, Niu et al. adapted Li and Lin’s extended binary classification framework to train deep neural networks for ordinal regression [14]; we refer to this method as OR-NN. Across different image datasets, OR-NN was able to outperform other reference methods. However, Niu et al. pointed out that ORD-NN suffers from rank inconsistencies among the binary tasks and that addressing this limitation might raise the training complexity substantially. Cao et al. [1] recently addressed this rank inconsistency limitation via the CORAL method. To avoid increasing the training complexity, CORAL achieves rank consistency by imposing a weight-sharing constraint in the last layer, such that the binary classifiers only differ in their bias units. However, while CORAL outperformed the OR-NN method across several face image datasets for age prediction, the weight-sharing constraint may impose a severe limitation in terms of functions that the neural network can approximate. In this paper, we investigate an alternative approach to guarantee rank consistency without increasing the training complexity and restricting the neural network’s expressiveness and capacity.

Several deep neural networks for ordinal regression do not build on the extended binary classification framework. These methods include Zhu et al.’s  [25] convolutional ordinal regression forest for image data, which combines a convolutional neural network with differentiable decision trees. Diaz and Marathe [5] proposed a soft ordinal label representation obtained from a softmax layer, which can be used for scenarios where interclass distances are known. Another method that does not rely on the extended binary classification framework is Suarez et al.’s distance metric learning algorithm [24]. Petersen et al. [16] developed a method based on differentiable sorting networks based on pairwise swapping operations with relaxed sorting operations, which can be used for ranking where the relative ordering is known but the absolute target values are unknown. Liu et al. adapted pairwise ranking constraints from RankingSVM [7] to reformlate the multi-category loss as a constrained optimization problem for ordinal regression [10].

This paper focuses on addressing the rank inconsistency on OR-NN without imposing the weight-sharing of CORAL [1], which is why an exhaustive study of the methods mentioned above is outside the scope of this paper. However, additional experiments and comparisons with SORD  [5] and CNNPOR [10] are included in the Supplementary Material in section Comparison with Other Deep Learning Methods for Ordinal Regression.

Let ${D=\left\{\mathbf{x}^{[i]},y^{[i]}\right\}_{i=1}^{N}}$ denote a dataset for supervised learning consisting of $N$ training examples, where $\mathbf{x}^{[i]}\in\mathcal{X}$ denotes the inputs of the $i$-th training example and $y^{[i]}$ its corresponding class label. In an ordinal regression context, we refer to $y^{[i]}$ as the rank, where ${y^{[i]}\in\mathcal{Y}=\{r_{1},r_{2},...r_{K}\}}$ with rank order ${r_{K}\succ r_{K-1}\succ\ldots\succ r_{1}}$. The objective of an ordinal regression model is then to find a mapping $h:\mathcal{X}\rightarrow\mathcal{Y}$ that minimizes a loss function $L(h)$.

With CORAL, Cao et al. [1] proposed a deep neural network for ordinal regression that addressed the rank inconsistency of Niu et al.’s OR-NN [14], and experiments showed that addressing rank consistency had a positive effect on predictive performance.

Both CORAL and OR-NN built on an extended binary classification framework [9], where the rank labels are recast into a set of binary tasks, such that ${y^{[i]}_{k}\in\{0,1\}}$ indicates whether $y^{[i]}$ exceeds rank $r_{k}$. The label predictions are then obtained via $h\left(\mathbf{x}^{[i]}\right)=r_{q}$, where $q\in\{1,2,...,K\}$ is the rank index, which is computed as

The CORAL method ensures that the $\{f_{k}\}_{k=1}^{K-1}$ predictions are rank-monotonic, that is, ${f_{1}\left(\mathbf{x}^{[i]}\right)\geq f_{2}\left(\mathbf{x}^{[i]}\right)\geq\dots\geq f_{K-1}\left(\mathbf{x}^{[i]}\right)}$, which provides rank consistency to the ordinal regression model. While the rank label calculation via Eq. 1 does not strictly require consistency among the $K-1$ task predictions, $f_{k}\left(\mathbf{x}^{[i]}\right)$, it is intuitive to see why rank consistency can be theoretically beneficial and can lead to more interpretable results via the binary subtasks. While CORAL provides this rank consistency, CORAL’s limitation is a weight-sharing constraint in the output layer. Consequently, all binary classification tasks use the same weight parameters and only differ in their bias units, which may limit the flexibility and expressiveness of an ordinal regression neural network based on CORAL.

The proposed CORN model is a neural network for ordinal regression that guarantees rank consistency without any weight-sharing constraint in the output layer (Fig. 2). Instead, CORN uses a new training procedure with conditional training subsets that ensures rank consistency through applying the chain rule of probability.

Given a training set ${D=\left\{\mathbf{x}^{[i]},y^{[i]}\right\}_{i=1}^{N}}$, CORN applies a label extension to the rank labels $y^{[i]}$ similar to CORAL, such that the resulting binary label ${y^{[i]}_{k}\in\{0,1\}}$ indicates whether $y^{[i]}$ exceeds rank $r_{k}$. Similar to CORAL, CORN also uses $K-1$ learning tasks associated with ranks $r_{1},r_{2},...,r_{K}$ in the output layer as illustrated in Fig. 2.

However, in contrast to CORAL, CORN estimates a series of conditional probabilities using conditional training subsets (described in Section 3.4) such that the output of the $k-$th binary task $f_{k}\left(\mathbf{x}^{[i]}\right)$ represents the conditional probability¹¹1When $k=1$, $f_{k}\left(\mathbf{x}^{[i]}\right)$ represents the initial unconditional probability $f_{1}\left(\mathbf{x}^{[i]}\right)=\hat{P}\left(y^{[i]}>r_{1}\right)$.

The transformed, unconditional probabilities can then be computed by applying the chain rule for probabilities to the model outputs:

Our model aims to estimate $f_{1}\left(\mathbf{x}^{[i]}\right)$ and the conditional probabilities $f_{2}\left(\mathbf{x}^{[i]}\right),...,f_{K-1}\left(\mathbf{x}^{[i]}\right)$. Estimating $f_{1}\left(\mathbf{x}^{[i]}\right)$ is a classic binary classification task under the extended binary classification framework with the binary labels $y_{1}^{[i]}$. To estimate the conditional probabilities such as $\hat{P}\left(y^{[i]}>r_{2}\,|\,y^{[i]}>r_{1}\right)$, we focus only on the subset of the training data where $y^{[i]}>r_{1}$. As a result, when we minimize the binary cross-entropy loss on these conditional subsets, for each binary task, the estimated output probability has a proper conditional probability interpretation²²2When training a neural network using backpropagation, instead of minimizing the $K-1$ loss functions corresponding to the $K-1$ conditional probabilities on each conditional subset separately, we can minimize their sum, as shown in the loss function we propose in Section 3.5, to optimize the binary tasks simultaneously..

In order to model the conditional probabilities in Eq. 3, we construct conditional training subsets for training, which are used in the loss function (Section 3.5) that is minimized via backpropagation. The conditional training subsets are obtained from the original training set as follows:

Additional theoretical justification for constructing the conditional training subsets is provided in the Supplementary Material in section Theoretical Analysis of Conditional Probability Estimation. Section 5.1 compares the predictive performance of the CORN method with and without training subsets.

Let $f_{j}\left(\mathbf{x}^{[i]}\right)$ denote the predicted value of the $j$-th node in the output layer of the network (Fig. 2), and let $|S_{j}|$ denote the size of the $j$-th conditional training set. To train a CORN neural network using backpropagation, we minimize the following loss function:

We note that in $f_{j}(\mathbf{x}^{[i]})$, $\mathbf{x}^{[i]}$ represents the $i$-th training example in $S_{j}$. To simplify the notation, we omit an additional index $j$ to distinguish between $\mathbf{x}^{[i]}$ in different conditional training sets.

To improve the numerical stability of the loss gradients during training, we implement the following alternative formulation of the loss, where $\mathbf{Z}$ are the net inputs of the last layer (aka logits), as shown in Fig. 2, and $\log\left(\sigma\left(\mathbf{z}^{[i]}\right)\right)=\log\left(f_{j}\left(\mathbf{x}^{[i]}\right)\right)$:

To obtain the rank index $q$ of the $i$-th training example, and any new data record during inference, we threshold the predicted probabilities corresponding to the $K-1$ binary tasks and sum the binary labels as follows:

The MORPH-2 dataset³³3https://www.faceaginggroup.com/morph/ [19] contains 55,608 face images, which were processed as described in [1]: facial landmark detection [20] was used to compute the average eye location, which was then used by the EyepadAlign function in MLxtend v0.14 [18] to align the face images. The original MORPH-2 dataset contains age labels in the range of 16-70 years. In this study, we use a balanced version of the MORPH-2 dataset containing 20,625 face images with 33 evenly distributed age labels within the range of 16-48 years.

The Asian Face dataset (AFAD)⁴⁴4https://github.com/afad-dataset/tarball [14] contains 165,501 faces in the age range of 15-40 years. No additional preprocessing was applied to this dataset since the faces were already centered. In this study, we use a balanced version of the AFAD dataest with 13 age labels in the age range of 18-30 years.

The Image Aesthetic dataset (AES)⁵⁵5http://www.di.unito.it/~schifane/dataset/beauty-icwsm15/ [21] used in this study contains 13,868 images, each with a list of beauty scores ranging from 1 to 5. To create ordinal regression labels, we replaced the beauty score list of each image with its average score rounded to the nearest integer in the range 1-5. Compared to the other image datasets MORPH-2 and AFAD, the size of the AES dataset was relatively small, and we did not attempt to create a class-balanced version of this dataset for this study.

The Fireman dataset (Fireman)⁶⁶6https://github.com/gagolews/ordinal_regression_data is a tabular dataset that contains 40,768 instances, 10 numeric features, and an ordinal response variable with 16 categories. We created a balanced version of this dataset consisting of 2,543 instances per class and 40,688 from the 16 ordinal classes in total.

Each dataset was randomly divided into 75% training data, 5% validation data, and 20% test data. We share the partitions for all datasets, along with all preprocessing code used in this paper, in the code repository (see Section 4.4).

The model evaluations and comparisons are based on the mean absolute error (MAE) and root mean squared error (RMSE), which are defined as follows:

where $y^{[i]}$ is the ground truth rank of the $i$-th test example and $h(\mathbf{x}^{[i]})$ is the predicted rank, respectively.

Then, using the best hyperparameter setting for each method, we repeated the model training five times using different random seeds (0, 1, 2, 3, and 4) for the random weight initialization and dataset shuffling. We considered the exact same hyperparameter ranges for each method. (A detailed list of the hyperparameter configurations we considered is shown in Table 1.) Then, we selected the best hyperparameter configuration, using grid search, based on its validation set performance for each method before computing the test set performance. Note that both the hyperparameter configuration and the best training epoch were determined based on the validation set before computing the final model performance on the independent test set. The best hyperparameter values for each method are listed in Table 2.

The models were trained for 200 epochs using stochastic gradient descent via adaptive moment estimation [8] with the default decay rates and carefully checked for convergence such that training and validation MAE started to diverge and the validation MAE started to stagnate or decline. The complete training logs for all methods are provided in the code repository (Section 4.4).

All neural networks were implemented in PyTorch 1.8 [15]. The models were trained on NVIDIA GeForce RTX 2080Ti graphics cards on a private workstation as well as T4 graphics cards using the Grid.ai platform. We make all source code used for the experiments available⁷⁷7https://github.com/Raschka-research-group/corn-ordinal-neuralnet and provide a user-friendly implementation of CORN in the coral-pytorch Python package⁸⁸8https://github.com/Raschka-research-group/coral-pytorch.

Given the superior performance of CORN across several datasets, we studied the importance of the training subsets. In this ablation study, created an alternative CORN method without training subsets subsets. Here, the conditional probability of the $k-$th binary task is computed as

We shall note that the modified CORN method without training subsets sees at least as many training examples as the regular CORN method. This is because each task now has access to the full training batch rather than a subset.

As the results in Table 4 show, the subsets do not only play a crucial role for yielding meaningful and theoretically justified rank probability values in CORN but they also improve the predictive performance. Across all datasets, with the exception of MORPH-2, the neural network trained with the regular CORN method outperforms the alternative version without subsets.

Suppose we are interested in estimating a series of conditional probabilities

with the observed dataset ${D=\left\{\mathbf{x}^{[i]},y^{[i]}\right\}_{i=1}^{N}}$, where $f_{k}(\mathbf{x})$ is the functional form of the neural network model outputs that depend on the neural network model weights. The likelihood of the model weights can be written as

Hence, minimizing the loss function (Eq. 5) is equivalent to solving the maximum likelihood estimate of the functional form representations of the conditional probabilities. This is also the theoretical justification that we construct the conditional training sets in the data preparation for the CORN loss function. Without using the conditional set in the loss function, the estimated probabilities do not have a conditional probability maximum likelihood interpretation. After solving the maximum likelihood estimates of the conditional probabilities, it is natural to use the probability chain rule to find the unconditional probabilities of exceeding rank $r_{k}$ in Eq. 3 given each input $\mathbf{x}$.

Analogous to CORAL [1] and based on established generalization bounds for binary classification, Theorem 1 shows that the final rank prediction by CORN generalizes well when the binary classification tasks generalize well.

We compare CORN with two additional, recent ordinal regression methods that do not rely on the binary extension framework:

1. 1.

   the convolutional neural network with pairwise regularization for ordinal regression (CNNPOR) method by Liu, Long, and Goh [10];

2. 2.

   the soft ordinal vectors (SORD) method by Diaz and Marathe [5].

To facilitate a fair comparison, we adopted the exact same architecture and preprocessing steps from [5] and [10]. Similar to CNNPOR and SORD, we used a VGG16 [23] backbone pre-trained on ImageNet [4] where only the last layer (output layer) was re-initialized with random weights following. Also, following the preprocessing steps in CNNPOR and SORD, the training images in the AES dataset were resized to $256\times 256$ pixels and randomly cropped to $224\times 224$ as well as randomly flipped across the horizontal axis.

As these additional results on the AES dataset show, CORN also outperforms other recent ordinal regression methods for deep learning (CNNPOR [10] and SORD [5]) overall when trained with a VGG16 backbone that was pre-trained on ImageNet (Table S2).

This section describes additional results we obtained from comparing CORN to other methods on text datasets using recurrent neural networks (RNNs) with long short-term memory (LSTM) cells.

Both the 100K Coursera’s courses reviews dataset⁹⁹9https://www.kaggle.com/septa97/100k-courseras-course-reviews-dataset and TripAdvisor hotels reviews dataset¹⁰¹⁰10https://www.kaggle.com/andrewmvd/trip-advisor-hotel-reviews contain reviews with 5 rating labels ranging from 1 to 5 stars. We used balanced versions of these datasets to distribute the ratings evenly. The balanced Coursera dataset contains 11,852 reviews, and the TripAdvisor dataset contains 7,000 reviews. Each dataset was randomly divided into 75% training data, 5% validation data, and 20% test data. The dataset splits and preprocessing code can be found in the code repository (see Section 4.4 of the main manuscript).

For method comparisons on the text datasets, we use a standard RNN with one LSTM cell. Similar to the image datasets, we compare the performance of a neural network trained via the rank-consistent CORN approach to both Niu et al.’s OR-RNN method (no rank consistency) and CORAL (rank consistency by using identical weight parameters for all nodes in the output layer). We also implemented RNN classifiers trained with standard multicategory cross-entropy loss as a baseline, which we refer to as CE-RNN. All methods share similar backbone architectures and only require minor changes in the output layer.

The training and evaluation steps are similar to those of the image datasets in the main manuscript. The RNN models were trained for 200 epochs using ADAM with default settings. The model with the best validation set performance was then chosen as the final model for evaluation on the test set. The training logs for all runs are available in the CORN GitHub repository (see Section 4.4). The learning rates considered for hyperparameter tuning were 1e-5, 5e-5, 1e-4, 5e-4, 1e-3, 5e-3, and we considered batch sizes 16, 32, 64, 128, 256, 512.

As the results in Table S4 show, CORN outperforms all other methods on the two text datasets, TripAdvisor and Coursera, in terms of the test set MAE. All experiments were repeated for different random seeds to ensure that the results were not coincidental.

It is worth noticing that while the CORN method showed superior performance in terms of test MAE, the CORAL method performed better in test RMSE compared with all other methods. One possible explanation is that since RMSE penalizes large gaps more harshly than MAE, CORAL may behave slightly better on outliers while CORN may make fewer mistakes in total. However, both methods show reliable performance over the text datasets.

TableS5 is a more detailed version of the results table shown in the main paper, listing the performance for each individual random seed.

We can convert the CORN loss function,

This allows us to use the logsigmoid(z) function that is implemented in deep learning libraries such as PyTorch as opposed to using log(1-sigmoid(z)); the former yields numerically more stable gradients during backpropagation. A PyTorch implementation of the CORN loss function is shown in Fig. S1.