High Resolution Face Age Editing

Let $\mathbf{x}_{0}$ be an image drawn randomly from a face dataset. We denote by $\alpha_{0}$ the age of the person in $\mathbf{x}_{0}$. Our goal is to transform $\mathbf{x}_{0}$ so that the person in this image looks like someone at $\alpha_{1}$ years old. We want the aged version of $\mathbf{x}_{0}$ to share many age-unrelated characteristics with $\mathbf{x}_{0}$: identity, emotion, haircut, background, etc. That is to say: the facial attributes not relevant to age, as well as the background, need to be preserved during age transformation. Therefore, we assume that a face aging model and a face de-aging model can share most of their parameters. In this setting, we consider a single age transformer $G$ and assume that $G$ can transform any face image to any target age. The inputs of our model are the face image $\mathbf{x}_{0}$ and the target age $\alpha_{1}$. The output is denoted by $G(\mathbf{x}_{0},\alpha_{1})$, which depicts $\mathbf{x}_{0}$ at the target age $\alpha_{1}$.

The proposed age transformer shown in Figure 2 employs an auto-encoder architecture and is made of an encoder, a feature modulation block and a decoder. The encoder consists of three strided convolutional layers (the first one of stride 1, the other two of stride 2) and four residual blocks [8], while the decoder contains two nearest-neighbour upsampling layers and three convolutional layers, similar to the architecture used in [14, 40]. The main difference compared to these works is our feature modulation block, in which the output features of the encoder are modulated by an age-specific vector (see details below). This idea is inspired by recent works on style transfer [5, 10] which show the possibility to represent different styles using the parameters of normalization layers.

• •

  Encoder  The face image $\mathbf{x}_{0}$ is the input of the encoder. The output features are denoted by $\mathsf{C}\in\mathbb{R}^{n\times c}$, where $c=128$ is the number of channels and $n$ is the product of the two spatial dimensions.

• •

  Feature modulation for age selection  The target age $\alpha_{1}$ is encoded as an one-hot vector, denoted by $\mathbf{z_{1}}$, and passed to the modulating network. This network consists of a single fully connected layer whith a sigmoid activation. It outputs a modulation vector $\mathbf{w}\in[0,1]^{c}$, which is used to re-weight the features $\mathsf{C}$ before passing them into the decoder and obtaining the face image at the desired age. The modulated features are $\mathsf{C}\,{\rm diag}(\mathbf{w})$, where ${\rm diag}(\mathbf{w})$ is the diagonal matrix with diagonal $\mathbf{w}$.

• •

  Decoder  The decoder takes the modulated features $\mathsf{C}\,{\rm diag}(\mathbf{w})$ as input and two skip connections, used to preserve the finer details of the input image. The final output is denoted by $G(\mathbf{x}_{0},\alpha_{1})$.

As illustrated in Figure 2, we train our age transformer with an age classifier that ensures age-accurate transformation and a discriminator that preserves photorealism.

The initial age $\alpha_{0}$ of $\mathbf{x}_{0}$ is easy to estimate using a pretrained age classifier, e.g., [31]. We thus do not use an age-annotated dataset for training. The original age range of the training dataset is denoted by $\mathcal{Q}\subset\mathbb{N}$. At test time, the target age can be chosen as any age in $\mathcal{Q}$. At training time, it would seem reasonable to chose any value in $\mathcal{Q}$ uniformly at random. However, we noticed that the artifacts appearing during large age transformations were better corrected when selecting a target age $\alpha_{1}$ far enough from $\alpha_{0}$ during training. We propose to sample $\alpha_{1}$ from the set $\mathcal{Q}_{\alpha_{0}}=\{\alpha\in\mathcal{Q}\ :\left|\alpha-\alpha_{0}\right|\geq\alpha_{*}\}$ at training time, where $\alpha_{*}$ is a predefined constant representing the minimum age transformation interval. We denote by $q(\alpha|\alpha_{0})$ the uniform distribution over $\mathcal{Q}_{\alpha_{0}}$.

Classification loss  To measure the age of $G(\mathbf{x}_{0},\alpha_{1})$, we use the same age classifier as the one used to estimate $\alpha_{0}$. During training, we freeze the weights of this classifier. The classifier, denoted by $V$, takes $G(\mathbf{x}_{0},\alpha_{1})$ as input and generates a discrete probability distribution over the set of ages $\{0,1,\ldots,100\}$. The classification loss satisfies

where $p(\mathbf{x})$ denotes the training image distribution over $\mathcal{X}$, $\ell$ denotes the categorical cross-entropy loss, and $\mathbf{z}_{1}$ is the one-hot vector encoding $\alpha_{1}$.

Adversarial loss  To enforce better photorealism of the modified images $G(\mathbf{x}_{0},\alpha_{1})$, we adopt an adversarial loss built using PatchGAN [13] with the LSGAN objective [22]. Unlike the latest works on face aging [9, 20, 33, 36, 39], our discriminator is used to distinguish between real and manipulated images without taking the age information into account. In our work, the aging and de-aging effects is obtained solely with the age classification loss.

The discriminator is denoted by $D$. The architecture of $D$ is the same as proposed in [13]. We use a patch size $142\times 142$ for $1024\times 1024$ images. The modified image $G(\mathbf{x}_{0},\alpha_{1})$ should be indistinguishable from real samples. Therefore, the losses we use are

when training $G$, and

when training $D$. We apply $R_{1}$ regularization [23] with $\gamma=10$ on the discriminator.

Reconstruction loss  When the age transformer receives $\mathbf{x}_{0}$ and $\alpha_{0}$ as inputs, the generated output image $G(\mathbf{x}_{0},\alpha_{0})$ should be identical to the input image. Hence, we minimize the following reconstruction loss:

Full loss  We train the age transformer and the discriminator by minimizing the full objective

where $\lambda_{\rm recon}$ and $\lambda_{\rm class}$ are weights balancing the influence of each loss.

Our training dataset is built upon FFHQ [16], a high resolution dataset which contains $70,000$ face images at $1024\times 1024$ resolution. The dataset includes large variations in age, ethnicity, pose, lighting, and image background. However, the dataset contains only unlabeled raw images collected from Flickr.

To obtain the age information, we use an age classifier pretrained on IMDB-WIKI [31]. We observe that FFHQ contains much more samples of young faces than of old ones. This data imbalance is challenging since the aging and de-aging tasks would not be treated equally during training: most of faces being young, the age transformer would be trained to perform aging much more often than de-aging, failing to yield satisfying de-aging results. To compensate this imbalance in the age distribution, we propose to perform data augmentation using StyleGAN - a state-of-the-art high resolution image generation model [16]. We use the StyleGAN model pretrained on FFHQ to generate $300,000$ synthetic images. A quick visual inspection shows that most of the generated images have no significant artifacts and are nearly indistinguishable from real images by a human. Therefore, we use them for data augmentation to obtain a quasi-uniform age distribution over $\mathcal{Q}$: for any age bin with less than $1,000$ samples in the original FFHQ dataset, we complete this bin with some of the generated synthetic face images; for any age bin with more than $1,000$ samples, we select randomly $1,000$ face images from the original FFHQ dataset. The age-equalized dataset contains $47,990$ images over the range $\mathcal{Q}=\{20,\ldots,69\}$.

Our model is implemented in PyTorch [28]. We take $95\%$ of the equalized dataset as our training set and the rest as test set. For the age transformer and the discriminator, spectral normalisation [25] is applied on all the convolution layers except the last one of the age transformer. All the activation layers use Leaky ReLU [21] with a negative slope of $0.2$.

We consider age transformation only in the age range $\mathcal{Q}=\{20,\ldots,69\}$. The constant $\alpha_{*}$ is set to $25$. We have observed that the most significant artifacts appear when the gap between the source and target age is large. By choosing $\alpha_{*}$ large enough, we force the discriminator $D$ to suppress these artifacts during adversarial training. The weights $\lambda_{\rm recon}$ and $\lambda_{\rm class}$ are set to $10$ and $0.1$, respectively. We use Adam optimizer with a learning rate of $10^{-4}$. The age transformer $G$ is updated once after each discriminator update. Our model is trained for $20$ epochs to achieve face age editing on high resolution images. The first $10$ epochs are trained on $512\times 512$ images with a batch size of $4$. The next $10$ epochs are trained on $1024\times 1024$ images, for which we reduce the batch size to $2$, learning rate to $10^{-5}$ and $\lambda_{\rm recon}$ to $1$.

Figure 9 presents age editing results on $1024\times 1024$ input images in different age groups. Our approach yields visually satisfying results with sharp details (best viewed when zooming on the results) and without introducing significant artifacts. Only the age relevant facial features are modified, while the identity, haircut, emotion and background are well preserved. This is all the more satisfying that no mask has been used to isolate the face from the rest of the image. Figure 4 presents age editing results with a smooth evolution of the target age. The difference between two adjacent results is nearly invisible, which illustrates the smoothness of the aging process.

We compare our method to the two most recent state-of-the-art methods on face aging for which the official codes are released - IPCGAN [36] and PAGGAN [38]. We also compare our results to those obtained with FaderNet [18], which allows one to manipulate several facial attributes including the age.

Figure 5 present the face aging results of IPCGAN, PAGGAN and our method on CACD [2]. The output size of each method is: $128\times 128$ for IPCGAN, $224\times 224$ for PAGGAN, $256\times 256$ for our method. IPCGAN generates satisfying aging results and preserves well the identity of input images. However, as can be seen e.g. in Figure 5(a) row 1 column 4, the generated image presents noticeable artifacts. PAGGAN generates impressive aging effects but also introduce colored artifacts as shown in Figure 5(b) row 1 column 2. IPCGAN and PAGGAN both degrade the quality of input images. Our method is able to generate consistent aging effects, and preserve well the fine details of the input images.

Generalisation capacity for images in unseen dataset  For fair comparison and also to reduce the possible effect of overfitting on the training data, we evaluate all methods on a dataset not viewed at training time by any of the methods. We chose CelebA-HQ [15], a high resolution version of the CelebA dataset. The input images are at $1024\times 1024$ resolution, and are further downsampled at the resolution at which each method was trained using their official codes. The output size of each method is: $224\times 224$ for PAGGAN, $128\times 128$ for IPCGAN, $256\times 256$ for FaderNet, and $1024\times 1024$ for our method. We compare only the face aging results from young age group to old age group, since PAGGAN and IPCGAN are trained only for aging. Figure 6 shows the results obtained with the different methods. FaderNet [18] introduces little modifications. PAGGAN [38] generates satisfying age progression effects. However, noticeable artifacts are present on the face edges and hairs. IPCGAN [36] is limited to low resolution and thus introduces a strong degradation on the quality of the image. In comparison to these results, our approach introduces much less artifacts and preserves the fine details of the face and the background better.

Quantitative evaluation of image-to-image translation tasks is still an open question and there is no universal metric to measure photorealism or quantify artifacts in an image. The recent works [9, 20, 38] on face aging use an online face recognition API to estimate the age and the identity preservation accuracy of the modified images. We thus employ a similar evaluation process.

In our evaluation, the first $1,000$ images with true “Young” label of the CelebA-HQ dataset are extracted as test images. Using this test set, we make a quantitative comparison with FaderNet [18], IPCGAN [36] and PAGGAN [38]. Each image is transferred to the oldest age group using their official released models. For IPCGAN and PAGGAN, the oldest age group refer to $50+$ and $[51,60]$ respectively. For FaderNet, the old attribute is set to be the default largest value for aging in their official code. To have a fair comparison with groupwise methods, and since $50+$ is considered as the oldest age group, we choose a target age of $60$ (the mean of the age range $\{51,\ldots,69\}\subset\mathcal{Q}$) for our age transformer.

Thus we get 1000 modified images for each method. We further evaluate these output images using the online face recognition API of Face++ [12]. From the detect API, we obtain the following interesting metrics: age, gender, blurriness (whether the face is blurry or not, larger values means blurrier), smiling and emotion estimation. The emotion estimation contains a series of emotions: sadness, neutral, disgust, anger, surprise, fear and happiness. With a preliminary analysis on the results, $94.20\%$ of the input images are classified as neutral or happiness. Thus we just keep these two terms for emotion preservation comparison. We have also compared the identity preservation rate using the API to compare the modified images with the original inputs. However, since all methods achieve a nearly $100\%$ accuracy, this metric is not reported here.

Table 1 shows the quantitative evaluation results. All the methods are given the oldest age group as aging target, and we notice that our method has the highest average predicted age. The gender preservation rate is calculated by comparing the estimated gender with the original CelebA annotations. Using this metric, FaderNet achieves the best performance, followed by our method. For expression preservation (smiling) and emotion preservation (neutral, happiness), our approach yields the best results. It is to be noted however that all methods have similar results. For the blur evaluation, results are much more contrasted. Our method performs much better in generating sharper images, which is in agreement with the visual comparisons.

Ablation study on discriminator  We have explored three different types of discriminators to train the age transformer. Figure 7 presents the face age editing results corresponding to the different settings.

• •

  Conditional discriminator. We adopt a patch discriminator [13] with a label projection applied on the features before the last convolutional layer, similar to the settings in [26]. The discriminator is conditioned on four age groups: $20$-$35$, $35$-$45$, $45$-$55$, $55$-$70$. At the training stage we find it essential to give the same number of real and fake images from each class to the discriminator to make the training successful. If we sample a target age $\alpha_{1}$ from the set $\mathcal{Q}_{\alpha_{0}}=\{\alpha\in\mathcal{Q}\ :\left|\alpha-\alpha_{0}\right|\geq\alpha_{*}\}$ at training time, the discriminator will receive more manipulated images in the youngest and oldest group. Thus it tends to classify all the images in these two groups as fake. The conditional discriminator is very sensitive to the original data distribution and needs much more hyper-parameter fine-tuning to converge. Figure 7(a) presents the age editing results with conditional discriminator. Strong artifacts can be observed in the aging results.

• •

  Two separate discriminators. One discriminator receives manipulated and real images with a desired age lies in the old age group ($45$-$70$), while the other one takes manipulated and real images in the young age group ($20$-$44$). With this setting, the task of generating aging/de-aging effects is shared among the classifier and the discriminators. Although the results in 7(b) are better than those in 7(a), over-smoothing artifacts are perceived in the de-aging results and colored artifacts appear in the aging results.

• •

  One single discriminator. This is our proposed method. The discriminator can be considered as a regularizer which imposes photorealism, as it takes all the manipulated and real images as input. The generation of aging/de-aging effects is solely dictated by the age classifier. We are able to achieve high resolution results only with this last setting.

Image reconstructed from a latent code optimization  As mentioned in Section 2, the recent work of Shen et al. [32] proposes an effective way to manipulate the latent code of an image generator to achieve high visual quality manipulation of synthesized images. It is therefore tempting to manipulate the latent code directly to produce face manipulation (and thus age editing) on natural images with this approach. However, finding such a latent code for an arbitrary face image is still a challenging problem. According to our experiments using StyleGAN [16], only a fraction of natural face images can be accurately reconstructed from the latent code ¹¹1The latent code is obtained through optimization in the latent space by finding a latent code that minimizes the distance between the generated image and the input image. by [27]. Consequently, this type of method is impractical until a better StyleGAN encoder is made available. Figure 8 is meant to support this claim, where reconstruction results of natural face images can be assessed. We notice that the reconstructed images have painting-like artifacts, blurry backgrounds, and sometimes fail to preserve the identity of the person in the input image. Indeed, StyleGAN is much more efficient at sampling random faces from the latent space than at approximating a given face image. This is due to the fact that a GAN is not necessarily invertible. Hence, an editing method based on this latent code reconstruction will struggle to handle correctly natural images and to achieve the high visual quality of our method.

Weakly supervised training  To the best of our knowledge, our work is the first to use unlabeled data for training among recent face aging studies [9, 20, 33, 36, 39]. A classifier pretrained on IMDB-WIKI [31], a low resolution face dataset, is used to provide age information. Moreover, the discriminator in our method is used only to distinguish real and manipulated images. Relying solely on the classifier, we successfully extract the age specific features and further realize age transform on high resolution images. This reveals the capacity of the classifier, even trained on low quality images. Our method could be potentially generalized to other face attributes manipulation tasks, by using a separate pair of modulating network and classifier for each attribute.