COMBOLOSS FOR FACIAL ATTRACTIVENESS ANALYSIS WITH SQUEEZE-AND-EXCITATION NETWORKS

In this section, we give a detailed discussion of our proposed ComboLoss. The target of deep CNN-based facial attractiveness analysis is to find a nonlinear mapping $y=\Phi(\mathcal{I})$, which means a deep CNN $\Phi$ maps an input facial image $\mathcal{I}\in\mathcal{R}^{W\times H\times C}$ to an output beauty score $y$. Given a set of labelled groundtruth training samples $\Omega=\{\mathcal{I}_{i},\hat{y}_{i}\}_{i=1}^{N}$, the target of the model training is to find an optimal $\Phi$ which minimizes:

$$\sum_{i=1}^{N}\mathcal{L}(\Phi(\mathcal{I}_{i}),\hat{y}_{i})$$

(1)

where $\mathcal{L}$ is a pre-defined loss function which measures the difference between a predicted beauty score and its groundtruth beauty score. In this phenomenon, the deep CNN is regarded as a regression model.

The design of a proper loss function is crucial for deep CNN-based facial beauty prediction. However, the majority deep CNN-based FBP systems [4, 10, 11] usually utilize MSE as supervision. The performance of solely leveraging beauty score as supervision to train a regression network is limited [6, 7, 13], researchers are paying attention to adopt multi-task learning [7], ranking [13], multi-stream network [6], label distribution learning [14], and utilize auxiliary network [8] to enhance performance. However, these methods bring too many parameters, result in quite heavy models. In this paper, we propose a new loss function, termed ComboLoss, which further improves the performance of deep CNN-based FBP systems. The components of ComboLoss are regression loss, expectation loss, and a classification loss.

• •

  Regression Loss. In contrast to mainstream FBP models with MSE loss, we adopt $L_{1}$ loss as regression loss instead. The regression loss is defined as:

  $$L_{reg}=\frac{1}{N}\sum_{i=1}^{N}|\hat{s}_{i}-s_{i}|$$

  (2)

  where $\hat{s}_{i},s_{i}$ and $N$ represent the output score of regression module, groundtruth score and sample capacity, respectively.

• •

  Classification Loss. In addition to regression module, we also insert a classification module before the last layer of SEResNeXt50 network. To enhance loss and gradient flowing, we adopt weighted cross entropy as classification loss, which is defined as Equation 3. The chosen of output neuron $\mathcal{C}$ and $w_{c}$ are discussed in Section 2.2. We introduce classification loss into a regression network to learn additional information, to better separates attractive faces and unattractive samples.

  $$L_{cls}=-\frac{1}{N}\sum_{i=1}^{N}w_{c}c_{i}log\hat{c}_{i}$$

  (3)

  where $\hat{c}_{i}$, $c_{i}$ and $w_{c}$ denote predicted probability output by softmax layer, correct classification indicator and weight of each category, respectively.

• •

  Expectation Loss. Despite the classification loss mentioned above can better separates different facial attractive levels, but it holds the assumption that categories are independent of each other, which is not suitable in this phenomenon (e.g. a face with a beauty level of 4 is undoubtedly more attractive than a face with beauty level of 1). In order to enhance model training, we present a new loss function by leveraging softmax probability. Then a regression manner is applied to minimize the gap between groundtruth beauty score and the expectation score. We define expectation as:

  $$\mathcal{E}_{i}=\sum_{i=1}^{\mathcal{C}}\hat{c}_{i}\cdot i$$

  (4)

  The expectation loss measures the difference between groundtruth score and expectation score with $L_{1}$ loss:

  $$L_{exp}=\frac{1}{N}\sum_{i=1}^{N}|\hat{s}_{i}-\mathcal{E}_{i}|$$

  (5)

Having defined the individual item, the ComboLoss is denoted as $\mathcal{L}_{combo}=\alpha\cdot L_{reg}+\beta\cdot L_{exp}+\gamma\cdot L_{cls}$, we set $\alpha=2,\beta=1,\gamma=1$ in our experiments.

Since we insert classification module into a regression network, the groundtruth for classification should be defined for model training. The groundtruth beauty scores in relevant benchmark dataset [4, 10, 3] are continuous values, which cannot be directly utilized to train a classification task. Therefore, attractiveness score discretization should be conducted firstly. How to discrete continuous scores and how many score ranges should be discreted remain a open problem. In this paper, we take the simplest fashion for easy implementation. Formally, in SCUT-FBP [4] and SCUT-FBP5500 [10] datasets, the rounded integer of $\lceil s-\frac{1}{2}\rceil$ is regarded as classification label, in HotOrNot [3], the intervals are set to 3 as in [6]. More advanced discretization methods may also be adopted, which is left to our future work.

It is worth noting that despite we take the similar discretization method like CRNet [6], but the imbalance problem after applying discretization is neglected in [6]. The capacity of category 0 is 3 times bigger than category 1, which is harmful for model training. In this paper, we solve the imbalanced learning problem in loss level. Namely, the hyper-parameter in Equation 3 is introduced to balance the loss and gradient of each category. The hyper-parameter $w_{c}$ of category $c\in\mathcal{C}$ is determined as:

$$w_{c}=\frac{max(|m|)_{m\in\mathcal{C}}}{|c|}$$

(6)

Inspired by [15], we incorporate squeeze-and-excitation module [15] into the ResNeXt50 [16] to form SEResNeXt50 as the backbone network for feature representation learning. Other architecture (such as ResNet [17], EfficientNet [18]) may also be utilized, but it’s beyond the scope of this paper.

Squeeze operation can embed global spatial information with a channel descriptor through utilizing global average pooling to generate channel-wise statistics. Namely, by global average pooling the feature map, we get:

$$z_{c}=\frac{1}{H\times W}\sum_{i=1}^{H}\sum_{j=1}^{W}u_{c}(i,j)$$

(7)

where $H$ and $W$ represent the height and width of the feature map, $u_{c}(i,j)$ denotes the pixel value at $(i,j)$ in channel $c$. After obtaining the statistics $z_{c}\in\mathbb{R}^{C}$, an excitation operation is applied. This is achieved by the following equation:

$$s=\sigma(W_{2}\cdot ReLU(W_{1}\cdot z))$$

(8)

where $\sigma$ denotes sigmoid activation, $W_{1}\in\mathbb{R}^{\frac{C}{r}\times C}$ is a dimensionality-reduction fully connected (FC) layer with reduction ratio $r$, and $W_{2}\in\mathbb{R}^{C\times\frac{C}{r}}$ is a dimensionality-increasing FC layer with increasing ratio $r$. The output of the squeeze-and-excitation block is calculated by rescaling the transformation output $U$ with activations:

$$\tilde{x}_{c}=s_{c}\cdot u_{c}$$

(9)

where $\tilde{X}=[\tilde{x}_{1},\cdots,\tilde{x}_{c}]$, $u_{c}\in\mathbb{R}^{H\times W}$ represents feature map, $s_{c}$ represents scalar of sigmoid activation output (please refer to [15] for more details). In this paper, we incorporate squeeze-and-excitation [15] module into ResNeXt50 [16], result in SEResNeXt50 architecture. The network architecture is shown in Fig 1.

We use 3 popular datasets (SCUT-FBP [4], HotOrNot [3] and SCUT-FBP5500 [10]) to evaluate the effectiveness of our proposed methods. SCUT-FBP [4] contains 500 female images with beauty scores ranged from 1 to 5. HotOrNot dataset [3] contains 2056 facial images collected from Internet with variant postures, cluttered background, overexposure and low resolution problems. SCUT-FBP5500 [10] contains 5500 facial images with beauty scores in $[1,5]$. Each image is annotated by 60 volunteers, and the average is used as the groundtruth to remove personal preference bias. All experiments are conducted by 5-fold cross validation, and the average is reported for comparison with other models.

As noted in related works [4, 3, 10], we adopt mean absolute error (MAE), root mean squared error (RMSE) and pearson correlation (PC) to evaluate the performance of different models on SCUT-FBP5500 [10], and PC is used as metric on SCUT-FBP [4] and HotOrNot [3] datasets. A computational model with lower MAE, lower RMSE and higher PC denotes better performance.

We conduct experiments with PyTorch [19] on two NVIDIA K80 GPUs with cuDNN acceleration. The learning rate starts from 0.01 and is divided by 10 per 50 epochs. Weight decay and batch size are set as 0.001 and 64, respectively. The model is trained via SGD with 0.9 momentum for 200 epochs. The images are resized and randomly cropped to $224\times 224$ patches, color jittering and random rotation [19] are applied for data augmentation. The network is initialized with ImageNet pretrained weights to avoid overfitting due to the limited capacity of current public research dataset. We do not adopt additional facial dataset for pretraining.

The performance comparison with other methods on SCUT-FBP [4], HotOrNot [3] and SCUT-FBP5500 [10] datasets are shown in Table 1, Tabel 2 and Table 3, respectively. The proposed ComboLoss achieves state-of-the-art performance and outperforms prior arts on SCUT-FBP [4], HotOrNot [3] and SCUT-FBP5500 [10] datasets by 1.13%, 2.1% and 0.57% on PC, respectively. We list 6 precisely predicted samples and 6 inaccurately output instances on SCUT-FBP5500 [10] in Fig 2.

To validate the effectiveness of each component of our proposed methods, we perform extra ablation experiments on SCUT-FBP5500 dataset [10]. Unlike the experimental settings used in the previous performance comparison section, we take another data splitting strategy as defined in SCUT-FBP5500 provision [10]. Namely, the 60% of the images are used as training set and the remaining images are test set. We adopt 60%/40% splitting setting [10] in ablation experiments to reduce huge computation of 5-fold cross validation.