Uncertainty-oriented Order Learning for Facial Beauty Prediction

The earliest methods of FBP generally used handcrafted features, such as geometric features [10, 11, 12, 13, 14], holistic features (e.g., color features [19, 22], eigenface [2, 16, 17, 21], LBP features [15, 18, 20], etc.) or mixed features (i.e., geometric and holistic features), then built classification [10, 14, 21, 23, 25] or regression [11, 16, 19, 22, 26, 4] models.

Recently, convolutional neural networks (CNN) have become a mainstream method for FBP. Some studies considered FBP as a classification task. Gan et al. [42] proposed the 2MBeautyNet to improve the accuracy. Zhai et al. developed three models consecutively, i.e. the BeautyNet model [43], the AFBS model [44] and the BLS method [45].

More studies treated FBP as a regression problem. Xu et al. [29] proposed a CRNet that jointly optimized the classification branch and regression branch through classification and regression losses. Later, they introduced an improved expectation loss into ComboLoss [30] to boost the performance. Lin et al. [28] proposed a $R^{3}$CNN that used the relative ranking based loss. Shi et al. [4] used pixels to mark different parts of a face as meta information and applied the co-attention learning mechanism to characterize the importance of different regions and different facial components simultaneously. Lin et al. [3] proposed a facial attribute aware convolution neural network, AaNet, which used a parameter generator to adaptively adjust the filter of the main network. F. Bougourzi et al. [46] aimed to train a better regression model, so they proposed CNN-ER and applied dynamic loss parameters to minimize the effect of outliers.

As most FBP datasets are scored by multiple people, an alternative solution for FBP is label distribution learning (LDL). Fan et al. [31] used LDL to train CNN on the basis of residual neural network. Wang et al. [32] proposed the LDL method LDL-LDM to exploit global and local label correlations on label distribution manifolds.

All above methods model the FB features of a facial image as a point on the latent space, and learn a mapping between the point and the given FB score, ranking or label distribution. Few of them consider the human cognition bias and the FB standard bias across datasets. Such direct regression mappings may suffer from weak generalization.

In everyday life, people often cognize the world and make decisions through comparisons. In psychology, kinds of abstract attitude are commonly measured by pairwise comparison, in which the subject compares a series of objects in pairs and makes a choice between two objects based on a certain criterion. Some studies have introduced pairwise comparison into their tasks. Ko et al. [47] proposed a Pairwise Aesthetic Comparison Network (PACNet) to extract features for image aesthetic assessment (IAA). Lee et al. [48] established the prototype of order learning, constructed a pairwise comparison matrix to predict image aesthetic scores. Hu et al. [49] proposed a learning framework based on pairwise comparison by focusing on the relative quality ranking of restored images. Lim et al. [39] formally proposed order learning and applied it to facial age estimation. It aims to discover the order of sequential patterns for a robust classification task. Both pairwise comparison methods and order learning can only work on data with certain label other than uncertain data, as they need to construct pairs for comparison.

Thurstone argued that a term is needed for the process by which the organism identifies, distinguishes, discriminates, or reacts to stimuli. It is known as the discriminal process and also called as attitude. Psychologically some of these attributes can be measured. FBP is a classical attitude measurement in the field of psychophysics, and the inconsistency of human cognition has been studied by psychologists for a long time. Thurstone’s study showed that the discriminal processes generated by a stimulus were subject to noticeable fluctuation due to the different subjects who perceive the stimulus or the different environments. This fluctuation among the discriminal processes for a uniform repeated stimulus was designated the discriminal dispersion.

In his study, experiments showed that the discriminal dispersion which any given repeated stimulus produces on the psychological continuum is usually Gaussian. So Thurstone proposed the Law of Comparative Judgment [40] based on this theory and used the means and variances of discriminal dispersions to measure the attitudes on psychological scale. Later, Cohen [50] extended Thurstone’s theory to multi-dimensional spaces.

The uncertainty reflects the dispersion of a random variable. Since the FB ratings encode human cognition bias, FB is a kind of uncertain data. There have been a few preliminary works to model the uncertainty for other tasks. Gast et al. [51] proposed Probout to replace the intermediate activations with low-dimensional Gaussian distributions by adjusting the activation function, and obtained uncertainty in a lightweight manner instead of traditional Bayesian approaches. Liu et al. [52] considered the image quality as a distribution rather than a feature point, then modeled the uncertainty by a low-dimensional Gaussian distribution. However, low-dimensional Gaussian distribution naturally limited the feature expressiveness. In the field of face recognition, to reduce the uncertainty caused by image distortion, Shi et al. [53] applied probabilistic face embeddings to model the uncertainty of face features. Chang et al. [54] proposed data uncertainty learning (DUL) to achieve a joint learning of data embeddings and uncertainty. In the field of age estimation, Li et al. [55] proposed a probabilistic ordinal embedding (POEs) to treat each facial age data as a multivariate Gaussian distribution and applied a set of ordinal distribution to enforce ordinal property in the embedding space. Although DUL and POEs modeled facial images as multi-dimensional Gaussian distributions, this uncertainty aimed to alleviate the effect of the inherent noise in the image, so only the “mean” of distribution is used for inference. Also, to avoid the distribution degenerating into deterministic embedding, the information bottleneck loss in POEs only ensures that the covariance matrix of distribution is close to the Identity matrix $I$. Instead, the uncertainty in our UOL denotes the inconsistency of human cognition, so we expect that the FB distribution to be close to the discriminal dispersion of human rating, and then take the variance of multiple ratings as the uncertainty to constrain the distribution. Thus, both the motivation and modeling approach of UOL have differences from above methods.

Order learning aims to learn the order relations between instances. As the order between FB is independent on the reference base, it can largely avoid the bias of FB standards introduced by different datasets. Following is the principle of order learning. Given two faces images, $x_{i}$ and $x_{j}$, and their FB scores $y_{i}$ and $y_{j}$ respectively, the order between $x_{i}$ and $x_{j}$ can be defined and encoded by a one-hot label,

where $\theta=0.2$ is the threshold that represents the discrimination of FB.

The conventional order learning treats an instance as a fixed feature point on the latent space, which is learned by a network $g(\cdot)$, shown in the left of Fig.3. It then carries out pairwise feature comparisons between two instances. The comparator $f(\cdot,\cdot)$ in order learning consists of three fully connected layers and is formulated as

which learns order relation from precise labels of two samples in the pair.

Previous methods of FBP usually use the mean of human ratings as the FB score for regression, which underestimate the human cognition biases. According to Thurstone’s discriminal dispersion theory, the discriminal processes to the same stimulus are not always equal, but rather present a Gaussian distribution on the psychological scale. This theory is also applicable to FBP. We then design a high-dimensional psychological scale space to address the inconsistency of human cognition on FB. Specifically, we model the human ratings of an instance $x$ as a multi-dimensional Gaussian distribution $z\sim\mathcal{N}(\mu(x),\Sigma(x))$ in the space, which is used as a feature for pairwise comparisons, as shown in the right of Fig. 3. A VGG16 is applied to encode mean vector $\mu(x)$ and covariance matrix $\Sigma(x)$ of the distribution. $\Sigma(x)$ represents the dispersion of the rating distribution. As $\Sigma(x)$ is a diagonal matrix, the degree of discriminal dispersion is the Frobenius norm of it,

During training, the variance of the $i$-th instance’s ratings is set as the ground truth, $\eta_{i}$, representing discriminal dispersion degree of the instance, and the network is optimized by minimizing the KL divergence between the predicted distribution of dispersion degrees $(\|\Sigma(x_{1})\|_{F},\|\Sigma(x_{2})\|_{F},\cdots,\|\Sigma(x_{M})\|_{F})$ and the ground truth distribution $(\eta_{1},\eta_{2},\cdots,\eta_{M})$ of the $M$ training instances. Unlike POEs [55], such operation can optimize the prediction to be as close as possible to the human ratings rather than a standard normal distribution,

where $M$ is the total number of training instances.

It has been generally accepted in machine learning that data augmentation can improve the robustness of models. We think our uncertainty modeling can be considered as a specific form of data argumentation, because its process is similar to the feature-level data augmentation [56, 57]. Firstly, we build up a Gaussian distribution in the high-dimensional psychological scale space according to the human ratings. Then, we randomly sample from these Gaussian distributions for pairwise comparisons. This process can be considered as disturbing a single feature point on the latent space, which is the feature level augmentation. As the disturbed features usually belong to the same class of the original feature, such augmentation is often applied to classification tasks [56, 57]. Our order learning just is a triple classification problem.

After modeling the FB uncertainty, an order should be established for these multi-dimensional Gaussian distributions on the phycological scale space. However, the conventional order learning cannot learn the order between distributions. Thurstone compares the observations of multiple subjects as the order of two stimuli on the psychological scale. To mimic this process, we design a uncertainty-oriented comparison module based on the Monte Carlo sampling.

To have a better order relation, we first constrain the Wasserstein distance between distributions on the psychological scale space. It allows that instances with similar scores have smaller distances between their distributions, while instances with significant different scores have larger distances. To this end, we apply a hinge loss to constrain the ordinal property of the psychological scale space and form a triplet for any three instances $(x_{l},x_{m},x_{n})$ from the dataset, who have ground truth $(y_{l},y_{m},y_{n})$ and corresponding feature distributions $(z_{l},z_{m},z_{n})$,

where $S=\{(l,m,n)\,|\,\left|y_{l}-y_{m}\right|<\left|y_{l}-y_{n}\right|\}$ and $\tau$ is the margin. $d(\cdot,\cdot)$ denotes the Wasserstein distance between two Gaussian distributions,

where $\mu_{1}^{j}$, $\mu_{2}^{j}$, $\sigma_{1}^{j}$ and $\sigma_{2}^{j}$ are the $j$-th dimension of $\mu_{1}$, $\mu_{2}$, $diag(\Sigma_{1})$ and $diag(\Sigma_{2})$ respectively. $D$ is the dimensionality of the vector. The construction procedure of triplets can be found in Appendix A.

Afterwards, we apply $T$ times Monte Carlo sampling on the distribution of instance $x_{i}$, which is analogous to the observations of multiple subjects on a stimulus. To make network be backpropagated, the random sampling and forward propagation must be separated. Thus, we apply the reparameterization sampling method [58] to get the $t$-th sampling $z_{i}^{(t)}$ from distribution $z_{i}$,

where $\mathcal{N}(0,I)$ denotes the multi-dimensional Gaussian distribution with zero mean and identity covariance matrix $I$. The sampling process is shown in Fig. 4.

The comparator $f(\cdot,\cdot)$ in conventional order learning is applied to learn the order between two sampling feature points. The relative relation $Y^{\prime}$ between two distributions of $x_{1}$ and $x_{2}$ is obtained by calculating the mean of $T$ comparisons,

A cross-entropy loss $\mathcal{L}_{Cls}$ for triple classification is used to optimize the comparator $f(\cdot,\cdot)$,

where $Y_{c}^{\prime}$ and $Y_{r}^{\prime}$ denote the $c$-th and $r$-th dimensions of the output vector $Y^{\prime}$, $c$ is the dimension where the ground truth is.

Thus, the entire loss of our UOL is

where $\alpha$ and $\beta$ are weights to control the contribution of each loss function.

After establishing the order of samples, network cannot predict the FB score yet because the order is independent on the reference base. Conventional order learning [39] compares the input face image with a set of reference images whose labels cover the entire range of ages. These comparisons will find the most similar references to the input. Their precise labels will determine the label of input. This method is known as the maximum consistency (MC) rule [39], which requires the reference set must be balanced (the number of reference images must be the same for each interval) and continuous (no discontinue interval throughout the entire range).

However, most FBP datasets are unbalanced. Figure 5 (b) shows an example, the data distribution of SCUT-FBP5500. Thus, MC rule can only cover the range of 1.6$\sim$4.5, the blue box in Fig. 5 (c). To address this problem, we propose a score estimation method based on Bradley-Terry model. Specifically, an input with unknown score $s_{t}$ is compared with a reference image with known score $s_{r_{i}}$. Bradley-Terry model tries to estimate the best $s_{t}$, and then models the possible order result $Y$ and score difference $(s_{t}-s_{r_{i}})$ as the following probability distribution,

where 0, 1 and 2 represent the “$<$”, “$\approx$” and “$>$” relations. $\delta_{0}=-\delta_{1}=-\delta$, $\delta_{2}=+\infty$, $\delta$ is a positive parameter to control the probability of “$\approx$”. $k$ is a scaling parameter to determine the change rate of probability. $S_{ref}$ denotes the set of all scores in the reference set.

Suppose $n$ images exist in the reference set and their ground truth scores are $\{s_{1},s_{2},\dots,s_{n}\},s_{i}\in S_{ref}$. We apply the optimized comparator $f(\cdot,\cdot)$ to predict the order between the input and each reference image, which results in $\{R_{1},R_{2},\dots,R_{n}\}$, $R_{i}\in\{0,1,2\}$, then maximize the likelihood function,

Finally, the FB score $s_{t}$ of input image can be obtained.

The advantage of using maximum likelihood estimation is that FB scores can be estimated in the entire range by partial comparison results. Therefore, our UOL can work on unbalanced and discontinuous reference set. Obviously, the more complete the reference set is, the better performance our UOL achieves. Figure 5(c) shows the reference set for our UOL based on SCUT-FBP5500 dataset.

We use a pretrained VGG16 on ImageNet as the backbone of our method, and optimize it by Adam optimizer with a batch size of 32. The learning rate is $1e-4$ at the beginning and a Cosine Annealing scheduler with the minimal learning rate $1e-6$ is applied. We set the training epoch to 100 for all 5-fold cross validation. For data preprocessing, all the facial images ($350\times 350$) are resized to $256\times 256$ firstly. Then a $224\times 224$ center croping and a random horizontal flipping are performed, followed by per-pixel rescale to $0\sim 1$ and mean value subtraction. The hyperparameters $\alpha$ and $\beta$ in the loss function are $1e-4$ and $1e-3$, respectively.

We discretize the FB scores of training set at intervals of 0.1, and select $min(n_{i},10)$ images in each score interval from training set as the reference images to estimate the final score, where $n_{i}$ denotes the total number of training data in the $i$-th score interval.

To verify the performance and generalization ability of UOL, we compare it with the state-of-the-art methods $R^{3}$CNN [28], CRNet [29], Co-attention [4], AaNet [3], ComboLoss [30] and CNN-ER [46].