An Unsupervised Approach towards Varying Human Skin Tone Using Generative Adversarial Networks

The objective of skin segmentation network is two fold. Given an image as input, it first classifies each pixel in the image in two classes $-$ skin, non-skin, followed by estimating a rgb value indicating the skin tone. Here the method predicts one skin tone irrespective of the number of persons present in the image. During training all the images used contains single person in it therefore no question of confusion arises. However it might be confusing to relate this to images with multiple persons present which might occur during testing. To maintain better flow, we will explain it in detail later.

The skin detection network consists of two sub-networks. The first half of this network is dedicated to skin segmentation objective. This part of the network is trained using the following loss function,

$$L_{seg}=L_{c}(\hat{x}_{seg},x_{seg})+L_{p}(\hat{x}_{seg},x_{seg})+L_{s}(\hat{x}_{seg},x_{seg}),$$

(1)

where $L_{c}$ (. , .) represents count loss and ${L}_{p}$ (. , .), $L_{s}$ (. , .) represents the perceptual [20] and SSIM [36] loss respectively between the ground truth (say, $x_{seg}$) and the predicted segmentation result (say, $\hat{x}_{seg}$).

Count loss measures the absolute difference between the counts of skin pixels of the predicted and the ground truth image. Since in general neural networks can not predict binarized output which is required in case of segmentation problems, therefore we try to control this constraint using this loss. It is observed that this loss improves the result in terms of predicting close to binarized output.

SSIM [36] is a popular image metric to measure the structural similarity between two images. However as the value of loss is minimized in neural networks therefore instead of SSIM we employ DSSIM as the loss function which is related to SSIM the following way, DSSIM (. , .) = 1 - SSIM (. , .). Throughout the paper we refer DSSIM loss as SSIM loss.

VGG perceptual loss [20] is a $L$2 loss between the features of the generated and the groundtruth images, obtained from different layers of pre-trained classification network (VGG-19) [12]. Instead of exactly matching the pixel values of the generated and groundtruth images this loss matches their feature representations. This encourages the network to produce images which are perceptually similar to their corresponding target images.

To train the other half of this network which is responsible for the skin color estimation we used MS-SSIM [37] loss, which is a variant of SSIM [36]. As this method is unsupervised therefore we do not use any ground truth skin color annotations for training this network. Instead we fill the detected skin areas with the predicted skin color and minimize the structural dissimilarity between the input and the manipulated image. The reason for designing such objective is to learn the skin color by maximizing perceptual similarity between the input and the predicted image in an unsupervised way. Some results can be seen in Fig. 3. The importance of this half of the network lies in learning to extract skin color from the person image itself, which is useful for the next part of the proposed work.

This section elaborates the main contribution of this work. The problem we are trying to attempt in this section is to synthesize images of varying human skin tone given a source image as input. We formulate this problem as a conditional image generation problem, where the source image, along with its skin segmentation (obtained from the skin segmentation network discussed in the previous section) and a control variable $z$ is given as input to a cGAN. $z$ controls the amount of change of skin tone. The value of $z$ = 0 indices no change of skin color while, values less than zero and above zero indicates the amount of change towards darkness and fairness of skin respectively. Therefore $z$ here plays the role of a skin tone regulator.

We employ a CNN as our generator network in the cGAN and the discriminator network is a patchGAN discriminator [18] which is also a CNN. The objective of this generator is to learn a distribution over the input data set. It basically builds a mapping with some conditional information from a prior noise distribution to the data space; while the discriminator predicts the probability of an input given a conditional information, to be coming from the data distribution rather than the generator distribution. In general discriminator predicts a single probability value corresponding to a given input image. However inspired from [18] we employ a patchGAN discriminator for this objective which in contrast to classical CNN based discriminator classifies each patch of the image, where the patch size is much smaller than the input image size; which implies pixels separated by more than a patch diameter gets modelled independently. As discussed in [18] such a discriminator function acts as texture/style loss and plays significant role in keeping better texture in the generated image.

The interesting part of this cGAN lies in the design of the loss function. Lets denote the generator network as $f_{g}(.)$, the input image as $x$, $\hat{x}_{seg}$ as the skin segmentation output corresponding to $x$ (we denote complement of $\hat{x}_{seg}$ as ${\hat{x}^{\prime}}_{seg}$ representing the predicted non-skin regions). We formulate the objective function in the following way,

$$L_{cGAN}=l^{1}+l^{2}+\lambda(m\times z+l^{3}-\epsilon)+L_{ADV}.$$

(2)

Where considering $\hat{x}_{z=0}=f_{g}(x,z=0,\hat{x}_{seg})$ and $\hat{x}_{z\neq 0}=f_{g}(x,z\neq 0,\hat{x}_{seg})$, we define,

$$\begin{split}l^{1}&=L_{p}(\hat{x}_{z=0},x)\\ l^{2}&=L_{p}(\hat{x}_{z=0}\times{\hat{x}^{\prime}}_{seg},x\times{\hat{x}^{\prime}}_{seg})\\ l^{3}&=\log(0.5-L_{color})\\ L_{color}&=L_{p}^{color}(\hat{x}_{z\neq 0},x).\end{split}$$

(3)

Here $\lambda$, $m$ and $\epsilon$ are parameters. $L_{ADV}$ denotes the adversarial loss. The function $L_{p}(.,.)$ indicates VGG-perceptual loss and $L_{p}^{color}(.,.)$ indicated a loss similar in concept to perceptual loss but the underlying network is the skin color estimation network which is discussed in the previous section. The whole idea behind this loss is to ensure the skin color of the image synthesized by the cGAN with $z$ = 0 is close to that of the original image (this is ensured by the term $l^{1}$ in the loss function); while that with changing value of $z$ should differ from the original image (ensured by the term $l^{3}$), preserving the details in regions other than the skin intact (ensured by the term $l^{2}$). The term $\lambda\times m\times z$ ensures the change in skin color goes in accordance with the value and sign of $z$. During training the cGAN, we sample random values of $z$ from a specified distribution (the noise prior).

Observe from Fig 2 that this method works in case of images containing people with difference in skin tone. This eradicates the confusion regarding single skin tone prediction by the skin tone estimator sub-network in Sec. III-A. We want to clarify that the objective of that part of network was to be able to learn features related to skin color which came into use in training the cGAN. In images containing multiple person, the estimated skin color value basically indicates an average skin tone of all the persons present in the image.

We evaluate our model based on the following datasets, Category and Attribute Prediction Benchmark, In Shop Cloth Retrieval dataset of DeepFashion [24] and MPV [13]. DeepFashion is a large scale fashion dataset where images are taken under variety of illumination condition, clothing category, pose and gender. Deepfashion contains multiple subset of images each with different types of annotations. In this work we used two subsets. However our method is applicable to all other subsets of images also. The In Shop Cloth Retrieval dataset contains in total 52,712 images of multiple views of each person (front, side, back and full) while the Category and Attribute Prediction Benchmark dataset contains 289,222 number of clothing images, where the images are mostly of models wearing the clothing. The MPV dataset contains in total 35,687 images of multiple views of each person. Our model is trained on the Category and Attribute Prediction Benchmark dataset of DeepFashion. For quantitative analysis we tested our model on 2000 randomly selected images from each of the mentioned datasets. For each of the test sample image, the result is obtained with the values of $z$ drawn from $N$(0 , 0.05). The qualitative analysis is done on In Shop Cloth Retrieval dataset of DeepFashion, with the cGAN trained on Category and Attribute Prediction Benchmark dataset. For the skin detection part of work we trained the model on LIP [14] and MPV datasets, where LIP is a popular dataset for human parsing and MPV also contains annotations related to different body parts.

For quantitative analysis we have reported the scores on the following metrics, Inception Score [30] (IS) and Frechet Inception Distance [17] (FID), SSIM [36]. Note that both IS and FID are evaluation measure for GANs, which measures perceptual quality of synthesized outputs, however FID is a more preferred metric [17]. We report SSIM score for the training set of images, since groundtruth is not available for test set of images. Here ground truth implies the images of same person in different skin color. We report the value of Kolmogorov-Smirnov test [10] statistic which is a goodness of fit test.

Inception Score (IS) measures the classifiability and diverseness in the generated images where the generated images are classified using the inception v3 model [33] to predict the class probability. IS is defined as,

$${}IS(.)=exp(\underset{\textbf{a}\sim p_{g}}{\mathbb{E}}[D_{KL}(p(b|\textbf{a})||p(\textbf{a}))]),$$

(4)

where a is a generated image sampled from the learned distribution $p_{g}$, $\mathbb{E}$ is the expectation over the set of generated images, $p(b|\textbf{a})$ is the conditional class ($b$) distribution estimated for image a using Inception model [33] pre-trained with ImageNet [12], $p(b)$ = $\underset{\textbf{a}\sim p_{g}}{\mathbb{E}}p(b|\textbf{a})$ is the marginal distribution. $D_{KL}$ is the KL-divergence between the conditional class distribution and the marginal class distribution. IS has a lowest value of 1 and highest value of the number of classes in inception model. Here as we are using inception v3 model which has 1000 number of classes therefore IS has a maximum value of 1000.

Although IS is observed to correlate well with human perception [30] but it does not consider real data at all therefore it can not estimate how well the generator approximates the real data distribution. To address the issues of IS another metric for evaluating the performance of GAN is proposed. Below we discuss that.

Frechet Inception Distance (FID) is a measure of similarity between two sets of images. It extracts the features embedded in both real and the generated images from a layer of inception v3 model pre-trained with ImageNet [12]. Considering the embedding as continuous multivariate Gaussian, the mean and covariance are estimated for both the generated ($\mu_{g}$ , $\sigma_{g}$) and the real data ($\mu_{r}$ , $\sigma_{r}$). Then the FID is calculated as: $\left\lVert\mu_{r}-\mu_{g}\right\rVert_{2}^{2}+Tr(\sigma_{r}+\sigma_{g}-2(\sigma_{r}\sigma_{g})^{1/2})$. A lower value of FID is better.

Kolmogorov-Smirnov test (KS test) is a Goodness-of-fit test that measures the compatibility of random samples against some theoretical probability distribution function. In other words, it is a non-parametric test statistic which defines the largest absolute difference between two cumulative distribution functions as a measure of disagreement. Let us consider, $F_{obs}$ as the empirical distribution function of the data, and $F_{exp}$ as the cumulative distribution function (CDF) associated with the null hypothesis, then the KS test statistic is defined by,

$$D_{n}=\operatorname*{arg\,max}|F_{exp}(.)-F_{obs}(.)|.$$

(5)

Since KS test statistic is a measure of distance therefore, a lower value of it indices better distribution similarity (Maximum value of this statistic is 1 and minimum is 0). The classical KS test is based on one dimensional data, however since we are dealing in images therefore, we used the 2D variant of the KS test.

We present the values of IS, FID and SSIM in Table. I and the values of KS test statistic in Table II. The values of SSIM are based on the results of the cGAN generator with $z$ = 0. The scores of IS and FID, suggests that our method synthesizes quite good quality images which can also be verified visually from the results presented in the qualitative analysis section. Note that values of IS is low in the results. This is because, this problem is related to human images only therefore there is no question of class diverseness in the generated images, where class diverseness relate to the number of classes of InceptionV3 model. The values of KS statistic from Table. II (at 5% level of significance) shows the generator’s distribution is quite similar to the original data distribution, as the difference between the distributions of original and predicted skin tone is statistically insignificant. Which suggests that the predicted skin tones lie within the range of feasible human skin color range.

We present our qualitative results in this section. Fig. 3 shows the results of skin segmentation and the estimated skin color. It is observed that the mean predicted color of skin is visually quite similar to the original tone of skin. The final results on varying skin tone is shown in Fig. 4. It is worth noticing that the color of skin varies under different values of the control variable, while the skin becomes fairer as the value $z$ increases, it becomes darker with decreasing values of $z$.

It is observed experimentally that the value of $z$ within $\pm{0.13}$ gives perceptually coherent images on average. We also present result on in-the-wild image (Fig. 5). This shows this method is unconstrained to background clutter, variation of skin colors in different persons in the image.

We present a visual comparison with the results of SC-FEGAN [19] in Fig. 6. SC-FEGAN is a method for attribute manipulation which can not be directly applied for problems like skin tone change. However this method can reconstruct faces from its free-form input e.g., sketch of the face with color details etc. In Fig. 6 we have presented a visual comparison of our results with the results of SC-FEGAN generated from free-form inputs. Note that in case of our method the idea is to relatively make the skin fairer or darker, hence we do not explicitly provide any skin color as input. Here, for the purpose of better visual comparison we generated two results showing relatively darker and fairer (skin tones) reconstructions of the input faces. It is observed from the figures that our results look visually much convincing in terms of realism compared to that of SC-FEGAN.

In Fig. 7 we have presented a visual comparison between our results with the result generated by a professional in Photoshop photo editor. Here the editor has made the person’s skin tone fair. For the purpose of comparison we also synthesized our results towards fairness. Notice the difference between the skin colour in our results and that in the image generated by the editor. This happens because our method changes the tone gradually while keeping coherence with the skin tone in the previous image. Another thing that might be interesting is that as shown in [6] it took almost 9 minutes to generate the result in photoshop. However compared to this our results can be generated in much less time subject to the deep learning framework and GPU used.

Coming to the limitations of this method, we would like to mention that this method is sensitive to irregularities in skin segmentations. As observed from Fig. 8 improper segmentation may result in uneven skin tone (leftmost and rightmost persons).