FA-GANs: Facial Attractiveness Enhancement with Generative Adversarial Networks on Frontal Faces

We extract 2D facial landmarks for each of the training face images. For each face, $60$ landmarks are detected, and the detected landmarks are located on the outlines of left eyebrow, right eyebrow, left eye, right eye, nose bridge, nose tip, top lip, bottom lip, and chin. Fig. 3 shows an example of extracted landmarks. To normalize the scale, all faces are cropped and resized to the size of $224\times 224$ pixels. Instead of learning from facial landmarks directly, we prefer to enhance facial attractiveness via adjusting the relative distances among important facial components, such as eyes, nose, lip, and chin. Landmarks are more sensitive to small errors than relative distances. Therefore, the detected landmarks are used to construct a distance embedded mesh containing $150$ edges through Delaunay triangulation. We further reduce the dimension of the distance vector to $32$ by applying principal component analysis (PCA) and reserve $99.01\%$ of total variants.

Geometry branch aims to learn the mapping $\phi_{geo}:L_{X}\to L_{Y}$, while no samples in $L_{X}$ and $L_{Y}$ are paired. We achieve this by constructing the geometry adjustment GANs including only full connected (FC) and ReLU [30] layers. Unlike traditional GANs which train the generator until it fools the discriminator, the feature maps of $l_{y}\in L_{Y}$ are also utilized to help to learn the geometry mapping. The generator takes $l_{x}\in R^{32}$ as input and outputs the adjusted distance vector $l_{y}\in R^{32}$. Fig. 3 shows the architecture of our geometry adjustment branch. Apart from the last layer, the discriminator has the same architecture with the generator. The details of the architectures are shown in Table I.

Following the classic GANs [14, 31], the process of training discriminator amounts to minimizing the loss:

where $D_{geo}$ and $G_{geo}$ are the discriminator and the generator, respectively. $D_{geo}$ takes both the unattractive samples and generated samples as negative samples and regards the attractive samples as positive samples. The loss of discriminator can be rewritten as:

Considering that samples coming from the same category have similar feature maps, the feature loss is introduced and the loss of generator is defined as:

where $fc$ is the FC layer. $i\in\{1,2,3,4\}$ indicates the i-th layer in the generator $G_{geo}$.

Geometry branch converts $l_{x}$ to $l_{y}$ with $l_{x}$ being the distance vector derived from the landmark points $\mathbf{p}_{x}$. To adjust facial geometry, the enhanced points $\mathbf{p}_{y}$ corresponding to $l_{y}$ are estimated by minimizing the following energy function for the best fit,

where $e_{ij}$ is the element of the connectivity matrix of our constructed facial mesh. The distance term $d_{ij}$ is the entry in $l_{y}$ corresponding to $e_{ij}$. Minimization of the energy function is performed by the Levenberg-Marquardt [32] algorithm. Then the geometry enhanced face is generated by mapping the original face texture from the mesh constructed by $\mathbf{p}_{x}$ to the mesh of $\mathbf{p}_{y}$.

The process of appearance enhancement only requires a forward pass through the generator $G_{app}$. The generator $G_{app}$ is designed with the U-Net [34]. The discriminator $D_{app}$ is introduced to output the indicators $D_{app}(x)$ suggesting the probability that $x$ comes from the attractiveness category. Different from the classic discriminator of GANs, we implement our discriminator with the pyramid architecture [26] to estimate high-level attractiveness-related cues in a fine-grained way.

Specifically, a facial attractiveness classifier is pre-trained with the architecture of VGG-16 [35] to classify attractive and unattractive faces. The hierarchical layers of VGG-16 endow our network with the ability to capture image features from the pixel level to semantic level. The generator $G_{app}$ is optimized until the discriminator $D_{app}$ is confused about $G_{app}(x)$ and $y\in Y$ for all the pixel and semantic level features. Consequently, the feature maps of the 2nd, 4th, 7th, and 10th convolutional layers in VGG-16 are integrated into the discriminator $D_{app}$ for adversarial learning. The generator $G_{app}$ not only transfers $x\in X$ to $y\in Y$ but also preserves the identity. We achieve this by leveraging the network of deep face descriptor [6] $\psi_{id}$ to measure the identity similarity between $x$ and $G_{app}(x)$. The deep face descriptor $\psi_{id}$ is trained on a large face dataset containing millions of facial images by recognizing $N=2622$ unique individuals. We remove the classification layer and take the last FC layer as the identity preserving output, forming an identity descriptor $\psi_{id}(x)\in R^{4096}$. Both the original face $x$ and enhanced face $G_{app}(x)$ are fed into $\psi_{id}$ to generate $G_{app}(x)$ with small margin between $\psi_{id}(G_{app}(x))$ and $\psi_{id}(x)$.

The appearance improved face should be recognized as the same individual as the original one. To this end, the loss concerning appearance enhancement should not only take effect in helping improve the facial attractiveness, but also facilitate the preservation of identity. In addition, the difference between the improved face and the input should be constrained. Considering all these aspects, four types of loss for training the appearance branch are defined. They are the adversarial loss, identity loss, pixel loss, and total variation loss.

a) Adversarial loss. Similar to the geometry adjustment branch, both unattractive faces and the generated faces are deemed as negative faces and the attractive faces are deemed as positive ones. We also adopt the adversarial loss of LSGAN [31] to train the appearance adjustment branch. The adversarial loss is defined as:

b) Identity loss. To ensure that appearance enhanced face $G_{app}(x)$ and the original face $x$ are recognized as the same individual, the identity loss is introduced and is defined as:

where $\psi_{id}$ is the identity descriptor derived from the deep face descriptor [6].

c) Pixel loss. The generated face $G_{app}(x)$ is more attractive than the input one $x$. However, the gap between $G_{app}(x)$ and $x$ in pixels should be constrained. The pixel loss $\mathcal{L}_{px}$ enforces the enhanced face $G_{app}(x)$ to have small difference with the original face $x$ in the raw-pixel space. In addition, in experiences of other image generation tasks, $L1$ regularization performs better with less blurring than $L2$ regularization. The pixel loss is thus formulated as:

where $W$ and $H$ are the width and height of the image, respectively, and $C$ is the number of channels.

d) Total variation loss. Total variation loss $\mathcal{L}_{tv}$ is defined as total variation regularizer [36] in order to encourage spatial smoothness of the enhanced face area $G_{app}(x)$.

The comparison results are shown in Fig. 7, and are arranged into six groups. They are original, geometry adjustment, appearance adjustment, two-step adjustment, the state-of-the-arts, and FA-GANs, respectively.

In Fig. 7, Leyvand et al. [2] adjusts the geometry structure effectively compared to the original faces, and it also tends to generate portrait with smaller cheeks and bigger eyes. Li et al. [40] adjust the appearance mainly depending on the reference image, and the generated face looks homogeneous on the face area. BeautyGAN [5] only adjusts the facial image in the appearance aspect, and it has no effects in the geometry aspect. Moreover, BeautyGAN adjusts the facial image depends on the reference image and transfer the makeup style to the non-makeup image. Compared to BeautyGAN, “One-click beauty” of Meitu adjusts the face in both aspects. All of the instances generated by Meitu shrink the cheek slightly and have whiter skins while the colors of mouth and eyebrows are diluting. Our appearance branch promotes the original faces greatly and also generates whiter skin and has the makeup style on the mouth, eyes, and eyebrows. The geometry branch adjusts the geometry structure and tends to generate smaller cheeks and bigger eyes in a data-driven manner. FA-GANs adjust the face in both appearance and geometry aspects. The enhanced faces have whiter skin and the makeup style as appearance branch and have adjusted geometry structure as geometry branch.

To assess the results by different methods quantitatively, the facial attractiveness is assessed by Face++ API²²2https://www.faceplusplus.com/beauty/, which provides users with accurate, efficient, and reliable face-based services. It gives two scores: $f_{1}\%$ and $f_{2}\%$ indicating that males and females generally think this face is more attractive than $f_{1}\%$ and $f_{2}\%$ persons, respectively. $100$ groups of the original faces and their results are evaluated. The averaged attractiveness scores for each method are shown in Fig. 8.

All these methods get higher attractiveness scores than the original faces. The statistical results suggest that FA-GANs achieve the best performance over all these methods. The two-step adjustment method of applying our geometry branch followed by appearance branch to the input faces achieves the second best. FA-GANs promote the attractiveness score from $62.21\%$ to $70.23\%$ and $61.44\%$ to $70.70\%$ in the female and male views, respectively. It suggests that FA-GANs enhance the attractiveness effectively. Similarly, the two-step adjustment method achieves the second best, getting scores of $68.76\%$, $68.23\%$, respectively and outperforms BeautyGAN ($67.72\%$, $67.28\%$) and Meitu ($66.77\%$, $65.94\%$). Comparing the geometry adjustment results of our geometry branch in FA-GANs and [2], geometry branch outperforms [2] with a small margin in both assessing aspects of female and male. On the other hand, the appearance branch performs better than [40] no matter directly applying these appearance adjusting methods directly to the original faces or to the intermediate results obtained by other geometry adjusting methods. Comparing the results between appearance and geometry adjustment, appearance always achieves higher scores than geometry indicating that people can enhance their attractiveness by paying more attention to makeups than facelifts.

For identity preservation, we measure it using the cosine distance between $\psi_{id}(x)$ and $\psi_{id}(x’)$ where $x$ and $x’$ are the original face and the adjusted face, respectively, and $\psi_{id}$ is the deep face descriptor. The evaluated similarity scores are shown in Table III with respect to the listed methods in Fig. 8. As can be seen, all of the methods preserve the identity with the similarity greater than $80.00\%$. Meitu preserves the identity best with the similarity of $97.73\%$. Our geometry branch achieves the second best with the similarity of $96.34\%$, and FA-GANs also performs well with the similarity of $93.91\%$.

To build an effective and efficient method for enhancing the attractiveness of faces, we further compare these methods in time consuming. All the time consuming experiments are performed on the aforementioned computer. Geometry adjustment can be recognized as three steps, which contain extracting distance vector $l_{x}$ from original face $x$, enhancing $l_{x}$ to $l_{y}$, and mapping $l_{y}$ to the enhanced face $y$. The time consuming analysis of these steps is shown in Table IV. As can be seen in Table IV, Leyvand et al. [2] performs faster than our geometry branch in $0.1832$ seconds on our geometry adjustment dataset. The geometry adjustment methods spend most time on mapping $l_{y}$ to enhanced face $y$. Appearance branch enhances the attractiveness with only a forward pass through the generator. However, Li et al. [40] adjusts the appearance with an extra reference image, and requires image parsing in order to get a better result. The time consumption of appearance adjustment is shown in Table V. It suggests that appearance branch only requires $0.0599$ seconds to make adjustment on average. At last, we further compare the time consuming in the two-step adjustment method and FA-GANs in Table VI. It demonstrates that FA-GANs is the fastest. An input face can be adjusted in only $0.0552$ seconds. Furthermore, we compare FA-GANs with BeautyGAN in adjusting an input facial image. FA-GANs also performs faster than BeautyGAN, which takes $0.0794$ seconds on average.