Makeup Extraction of 3D Representation
via Illumination-Aware Image Decomposition

Facial makeup recommendation and makeup transfer methods have been developed in the computer graphics and computer vision research fields. Makeup recommendation methods can provide appropriate makeup for faces. The method in [SRH^∗11] captured 56 female faces with and without facial makeup. Suitable makeup was recommended by analyzing the principal components of the makeup and combining them with the facial appearance. An examples-rules guided deep neural network for makeup recommendation has been designed [AJWF17]. Professional makeup knowledge was combined with the corresponding before and after makeup data to train the network. However, it is challenging to collect a large-scale makeup dataset by following the existing methods. Recently, we proposed BareSkinNet [YT22], which can remove makeup and lighting effects from input face images. However, BareSkinNet cannot be used for subsequent makeup applications because it discards the makeup patterns and specular reflection. In contrast, our method can automatically extract the makeup layer information from portrait image inputs. As a result, a large-scale makeup dataset can be constructed.

The aim of makeup transfer is to transfer the makeup of a person to others. Deep learning technologies have significantly accelerated makeup transfer research. BeautyGAN  [LQD^∗18] is a method based on CycleGAN [ZPIE17] that does not require a before and after makeup image pair. A makeup loss function was also designed to calculate the difference between makeups. The makeup loss is a histogram matching loss that approximates the color distribution of the relative areas of different faces. Subsequent approaches that have used BeautyGAN as the baseline can solve more challenging problems. LADN [GWC^∗19] achieves heavy makeup transfer and removal, whereas PSGAN [JLG^∗20] and SCGAN [DHC^∗21] solve the problem of makeup transfer under different facial expressions and head poses. CPM [NTH21] is a color and pattern transfer method and SOGAN [LDP^∗21] is a shadow and occlusion robust GAN for makeup transfer. EleGANt [YHXG22] is a locally controllable makeup transfer method. Makeup transfer can also be incorporated into the Virtual-Try-On applications [KGPB20, KJB^∗22]. However, existing methods do not consider illumination and can only handle 2D images. Our method takes advantage of the makeup transfer technique. The extracted makeup is disentangled into bare skin, facial makeup, and illumination in the UV space. Furthermore, the extracted makeup is editable.

We mainly review the recent intrinsic image decomposition methods relating to the portrait image input. An intrinsic image decomposition method [LZL15] has been employed, thereby enabling accurate and realistic makeup simulation from a photo. SfSNet [SKCJ18] is an end-to-end network that decomposes face images into the shape, reflectance, and illuminance. This method uses real and synthetic images to train the network. Inspired by SfSNet, Relighting Humans [KE18] attempts to infer a light transport map to solve the problem of light occlusion from an input portrait. The aforementioned methods can handle only diffuse reflection using spherical harmonic (SH) [RH01] lighting. Certain methods use a light stage to obtain a large amount of portrait data with illumination [SBT^∗19, POL^∗21, WYL^∗20]. The global illumination can be inferred by learning these data. As data collection using a light stage requires substantial resources, several more affordable methods have been proposed [TKE21, LSY^∗21, JYG^∗22, WWR22, TFM^∗22, YNK^∗22]. In this study, we focus on makeup with the aim of decomposing a makeup portrait into bare skin, makeup, diffuse, and specular layers without using light stage data.

3D face reconstruction from a single-view portrait is challenging because in-the-wild photos always contain invisible areas or complex illumination. Existing 3D face reconstruction methods [BV03, THMM17, TZK^∗17, TZG^∗18, GZC^∗19, GCM^∗18, SBFB19, DYX^∗19, SSL^∗20, DBA^∗21, FFBB21, DBB22, ZBT22] generally use a parametric face model (known as a 3DMM) [BV99, LBB^∗17, GMB^∗18, BRP^∗18] to overcome this problem. In general, 3D face reconstruction is achieved by fitting the projected 3DMM to the input image. These methods estimate the SH lighting while inferring the shape and texture. We recommend that readers refer to  [EST^∗20, SL09], as these surveys provide a more comprehensive description of the 3DMM and 3D face reconstruction.

The 3DMM-based 3D face reconstruction method estimates coarse facial materials and cannot achieve a high-fidelity facial appearance. A detailed 3D face reconstruction method was proposed in [DBA^∗21], which can reconstruct the roughness and specular compared to the previous methods. This method involves the setup of a virtual light stage of illumination and the use of a two-stage coarse-to-fine technique to refine the facial materials. Inspired by  [DBA^∗21], we employ coarse facial materials and take specular into consideration. Furthermore, we extend and train a deep learning network using the method in  [DYX^∗19] as the backbone with a large-scale dataset. We optimize the textures in UV space to refine facial materials, to solve the problem of self-occlusion and obstacles of the face. The completed UV texture is advantageous because the complete makeup can be extracted to achieve high-fidelity makeup reconstruction.

The generation of a completed facial UV texture is a technique for obtaining refined facial materials. Several methods use UV texture datasets to train an image-translation network for the generation of the completed texture via supervised learning [SWH^∗17, YSN^∗18, DCX^∗18, GPKZ19, LL20, LL20, LMG^∗20, BLC^∗22]. Other methods [CHS22, GDZ21] use the GAN inversion [XZY^∗22] technique to generate faces in different directions for completion. However, both of these methods are limited by the training dataset. We adopt a state-of-the-art UV completion method known as DSD-GAN [KYT21] to obtain a completed high-fidelity face texture, which is to be used as the objective for the optimization of the refined facial materials. This method employs self-supervised learning to fill the missing areas without the need for paired training data.

We obtain the coarse facial materials using a 3D face reconstruction method based on regression-based inverse rendering. We use the FLAME [LBB^∗17] model with specular albedo from AlbedoMM [SSD^∗20]. The diffuse and specular albedo of FLAME are defined in the UV texture space. We only use the facial skin region in our study. Compared to existing methods, we extend the capability of the 3D face reconstruction network [DYX^∗19] ($FRN$) to estimate the shape, diffuse albedo, and diffuse shading, as well as the specular albedo and specular shading. Inspired by [DBA^∗21], a simplified virtual light stage with regular icosahedral parallel light sources is set up to infer the specular shading. The intensity of 20 light sources is predicted during the reconstruction process, and the direction of the light sources can be adjusted slightly. Furthermore, In order to eliminate the limited color range of the skin tone of FLAME, we estimate the skin tone adjustment parameters to ensure a diffuse albedo that is similar to the original image. The skin tone ablation study is depicted in Fig. 9. This process can improve the diversity of the diffuse albedo representation capability of FLAME.

The coarse shape geometry $\mathbf{G}_{c}$, diffuse albedo $\mathbf{D}_{c}$, and specular albedo $\mathbf{S}_{c}$ of the 3DMM are defined as follows:

where $\bar{\mathbf{G}}$, $\bar{\mathbf{D}}$, and $\bar{\mathbf{S}}$ are the average geometry, diffuse albedo, and specular albedo, respectively. ${\odot}$ denotes the Hadamard product. The subscript $c$ indicates coarse facial materials. Moreover, $\mathbf{B}_{id}$, $\mathbf{B}_{ex}$, $\mathbf{B}_{d}$, and $\mathbf{B}_{s}$ are the PCA basis vectors of the identity, expression, diffuse albedo, and specular albedo, respectively, whereas $\mathbf{\alpha}\in\mathbb{R}^{200}$, $\mathbf{\beta}\in\mathbb{R}^{100}$, $\mathbf{\gamma}\in\mathbb{R}^{100}$, and $\mathbf{\delta}\in\mathbb{R}^{100}$ are the corresponding parameters for controlling the geometry and reflectance of a 3D face. Finally, $\mathbf{C}_{gain}$ and $\mathbf{C}_{bias}$ are the skin tone adjustment parameters.

Our reconstructed texture can be formulated as follows:

where $\mathbf{R}_{c}^{d}$, $\mathbf{R}_{c}^{s}$, $\mathbf{D}_{c}^{S}$, and $\mathbf{S}_{c}^{S}$ are the diffuse reconstruction, specular reconstruction, diffuse shading, and specular shading, respectively. In this paper, a geometrically-derived shading component is referred to as “shading,” wheras a multiplication with reflectance is dubbed “reconstruction.” Using the normal $\mathbf{N}_{c}$ and second-order SH lighting coefficients $\mathbf{L}_{c}^{sh}\in\mathbb{R}^{27}$, the diffuse shading is calculated following the Lambertian reflectance model [BJ03]. The normal $\mathbf{N}_{c}$ is computed from the geometry $\mathbf{G}_{c}$. The specular shading is calculated following the Blinn–Phong reflection model [Bli77]. We define the 20 light sources of the virtual light stage with the light intensity $\mathbf{L}_{i}\in\mathbb{R}^{20}$ and light direction $\mathbf{L}_{d}\in\mathbb{R}^{60}$. $\mathbf{\rho}\in\mathbb{R}^{20}$ is the exponent that controls the shininess. We employ the 3D face reconstruction network implementation of  [DYX^∗19], and extend it to regress the parameters $\mathbf{\chi}=(\mathbf{\alpha},\mathbf{\beta},\mathbf{\gamma},\mathbf{\delta},\mathbf{C}_{gain},\mathbf{C}_{bias},\mathbf{r},\mathbf{t},\mathbf{L}_{c}^{sh},\mathbf{L}_{i},\mathbf{L}_{d},\mathbf{\rho})$, where $\mathbf{r}\in\mathbb{R}^{3}$ is the face rotation, and $\mathbf{t}\in\mathbb{R}^{3}$ is the face translation. Using the geometry $\mathbf{G}_{c}$, reconstructed texture $\mathbf{R}_{c}$, rotation $\mathbf{r}$, and translation $\mathbf{t}$, the reconstructed image ${I}_{rec}$ can be rendered.

We refer to [DYX^∗19] to set up the loss functions for the model training as follows:

where $\mathcal{L}_{photo}(\mathbf{\chi})$ is the L1 pixel loss between the skin region of input image ${I}_{in}$ and reconstructed image ${I}_{rec}$. $\mathcal{L}_{lan}(\mathbf{\chi})$ is the L2 loss between detected landmarks from ${I}_{in}$ and the projected landmarks of the 3DMM. $\mathcal{L}_{skin}(\mathbf{C}_{gain},\mathbf{C}_{bias})$ is the L1 loss for computing the mean color error between the skin region of ${I}_{in}$ and diffuse albedo $\mathbf{D}_{c}$. This is our specially designed loss term to adjust the skin tone. $\mathcal{L}_{reg}(\mathbf{\alpha},\mathbf{\beta},\mathbf{\gamma},\mathbf{\delta},\mathbf{L}_{i},\mathbf{L}_{d})$ is the regulation loss for preventing a failed face reconstruction result. In contrast to the previous method [DYX^∗19], we extend constraints on the specular related coefficients of $\mathbf{\delta}$, $\mathbf{L}_{i}$, and $\mathbf{L}_{d}$. $\mathcal{L}_{reg}(\mathbf{\alpha},\mathbf{\beta},\mathbf{\gamma},\mathbf{\delta},\mathbf{L}_{i},\mathbf{L}_{d})$ is determined as follows:

where $\|\cdot\|_{2}$ denotes the L2 norm. The balance weights are set to $\omega_{photo}=19.2$, $\omega_{lan}=5$, $\omega_{skin}=3$, $\omega_{reg}=3\times 10^{-4}$, $\omega_{\alpha}=1.0$, $\omega_{\beta}=0.8$, $\omega_{\gamma}=1.7\times 10^{-2}$, $\omega_{\delta}=1.0$, and $\omega_{L}=1.0$ in all experiments.

The goal of this step is to obtain the disentangled refined facial materials. We use the geometry $\mathbf{G}_{c}$ to sample the colors from the input image ${I}_{in}$ and project them to the UV texture space $\mathbf{T}_{u}$. $\mathbf{T}_{u}$ contains the missing area owing to self-occlusion or obstacles. As opposed to the 3D vertex color sampling approach used in related work [NTH21, CHS22], we adopt the image-to-UV rendering approach used in DSD-GAN [KYT21] to obtain a high-quality texture. The direct use of incomplete textures will cause many problems, such as noise and error. Thereafter, we fill the missing areas following DSD-GAN to obtain the completed UV texture $\mathbf{T}_{f}$. The subscript $f$ indicates refined facial materials. The difference between our method and DSD-GAN is that we use the FLAME model for implementation, whereas DSD-GAN uses the BFM model [GMB^∗18]. The results of the UV completion are presented in Fig. 5.

Using the completed UV texture as an objective with facial details, the optimization-based refinement module (see Fig. 3) is designed. Given the target texture $\mathbf{T}_{f}$, the coarse prior $\mathbf{D}_{c}$, $\mathbf{N}_{c}$, $\mathbf{L}_{c}^{sh}$, and $\mathbf{R}_{c}^{s}$ are used for initialization, and we optimize the refined materials of $\mathbf{D}_{f}$, $\mathbf{N}_{f}$, $\mathbf{L}_{f}^{sh}$, and $\mathbf{R}_{f}^{s}$ to reconstruct the refined texture $\mathbf{R}_{f}$. The diffuse shading is calculated following the Lambertian reflectance model [BJ03]. For the specular, we directly optimize the specular reconstruction $\mathbf{R}_{f}^{s}$, rather than the specular albedo and specular shading, which is not restricted by the light source settings of the virtual light stage in Sec. 3.1. Note that the specular reconstruction $\mathbf{R}_{c}^{s}$ is three channels image due to the specular albedo $\mathbf{S}_{c}$ of 3DMM, while $\mathbf{R}_{f}^{s}$ is converted to one channel image for stable and efficient optimization.

The loss functions for the optimization are calculated as follows:

where $\Psi=(\mathbf{R}_{f},\mathbf{D}_{f},\mathbf{N}_{f},\mathbf{R}_{f}^{s},\mathbf{L}_{f}^{sh})$ is a new parameter set for optimization. $\mathcal{L}_{recons}(\mathbf{R}_{f})$ ensures L1 consistency between $\mathbf{R}_{f}$ and $\mathbf{T}_{f}$. $\mathcal{L}_{vgg}(\mathbf{R}_{f})$ is the perceptual loss [JAF16] that aims to preserve the facial details. $\mathcal{L}_{tv}(\mathbf{D}_{f},\mathbf{N}_{f},\mathbf{R}_{f}^{s})$ is the total variation loss [GEB16] that encourages spatial smoothness in the optimized textures $\mathbf{D}_{f}$, $\mathbf{N}_{f}$, and $\mathbf{R}_{f}^{s}$.

where $\mathcal{L}_{prior}(\mathbf{D}_{f},\mathbf{N}_{f},\mathbf{R}_{f}^{s},\mathbf{L}_{f}^{sh})$ regulates the optimized texture to be similar to the coarse prior. We compute the L1 loss over the diffuse albedo, normal, and specular reconstruction as $\mathcal{L}_{D}(\mathbf{D}_{f})$, $\mathcal{L}_{N}(\mathbf{N}_{f})$, and $\mathcal{L}_{R}^{s}(\mathbf{R}_{f}^{s})$, respectively. The coarse textures are resized to the same resolution as that of refined textures. We note that the coarse and refined textures are not exactly equal; they differ in clarity, and with or without makeup. Therefore, before calculating the losses between the coarse and refined textures, we simply use a Gaussian blur filter $K$ to blur the refined textures in each iteration. We use the blurred refined texture to calculate the loss, while the original refined texture is not changed. $\mathcal{L}_{sh}(\mathbf{L}_{f}^{sh})$ is the L2 loss over the SH lighting. We normalize $\mathbf{N}_{f}$ to [-1, 1] following each optimization iteration to guarantee the correctness of the normal. The parameters $\omega_{recons}=40$, $\omega_{vgg}=5$, $\omega_{tv}=10$, $\omega_{prior}=1.0$, $\omega_{D}=4$, $\omega_{N}=1.0$, $\omega_{R}^{s}=1.0$, and $\omega_{sh}=1.0$ balance the importance of the terms. The refined textures of our coarse-to-fine optimization step are illustrated in Figs. 5,  6, and  7.

In this step, we only use the refined diffuse albedo $\mathbf{D}_{f}$, and decompose the texture into the makeup, bare skin, and an alpha matte. To train the network, we create two diffuse albedo datasets with and without makeup following the previous process; the makeup albedo $\mathbf{D}_{m}$ and non-makeup albedo $\mathbf{D}_{n}$, respectively.

As shown in Fig. 4, we design a makeup extraction network based on alpha blending. Our network consists of a makeup extractor ${E}$, a makeup discriminator ${M}$, and a bare skin discriminator ${B}$. ${M}$ and ${B}$ attempt to distinguish the makeup and non-makeup image. The core idea is that we regard the diffuse albedo as a combination of makeup and bare skin that is achieved by alpha blending. Therefore, ${E}$ extracts the bare skin $\mathbf{D}_{m}^{b}$, makeup $\mathbf{D}_{m}^{m}$, and alpha matte $\mathbf{A}$. $\mathbf{A}$ is used to blend the extracted makeup $\mathbf{D}_{m}^{m}$ and a non-makeup albedo $\mathbf{D}_{n}$ to generate a new makeup albedo $\mathbf{D}_{n}^{m}$ which has the identity from $\mathbf{D}_{n}$ and makeup from $\mathbf{D}_{m}$. The reconstructed makeup albedo $\mathbf{D}_{m}^{r}$ and the reconstructed bare skin $\mathbf{D}_{n}^{r}$ should be consistent with their original input. The reconstructed $\mathbf{D}_{m}^{r}$ and generated $\mathbf{D}_{n}^{m}$ are formulated as follows:

where $(1-\mathbf{A})$ is an inverted version of $\mathbf{A}$ with pixel values of $[0,1]$. The discriminators ensure that the generated bare skin $\mathbf{D}_{m}^{b}$ and makeup $\mathbf{D}_{n}^{m}$ are reliable. As the network takes advantage of the uniformity of the UV space, the makeup transfer is straightforward, and the problem of face misalignment does not need to be considered.

The loss function is given by:

where $\Phi=(\mathbf{D}_{m}^{r},\mathbf{D}_{n}^{r},\mathbf{D}_{m}^{b},\mathbf{D}_{n}^{m})$, $\mathcal{L}_{m}^{cyc}(\mathbf{D}_{m}^{r},\mathbf{D}_{n}^{r})$ and $\mathcal{L}_{m}^{vgg}(\mathbf{D}_{m}^{r},\mathbf{D}_{n}^{r})$ are the L1 loss and perceptual loss for the reconstruction, respectively. $\mathcal{L}_{m}^{adv}(\mathbf{D}_{m}^{b},\mathbf{D}_{n}^{m})$ is the adversarial loss for the discriminators and generators. $\mathcal{L}_{m}^{tv}(\mathbf{D}_{m}^{b},\mathbf{D}_{n}^{m})$ is the total variation loss for $\mathbf{D}_{m}^{b}$ and $\mathbf{D}_{m}^{m}$ to provide smooth texture generation, whereas $\mathcal{L}_{m}^{make}(\mathbf{D}_{m}^{b},\mathbf{D}_{n}^{m})$ is the makeup loss introduced by BeautyGAN [LQD^∗18]. We defined the corresponding face region ${F}$ of the brows, eyes, and lips in the UV texture space to compute makeup loss, and the compared textures are $\mathbf{D}_{m}$ and $\mathbf{D}_{n}^{m}$, and $\mathbf{D}_{n}$ and $\mathbf{D}_{m}^{b}$. Note that, in contrast with previous makeup transfer methods, the skin region is not calculated for the makeup loss because we believe that the difference in skin tone between individuals cannot be considered as makeup, while it can also be a restriction if the foundation of the makeup changes the entire face color. This is a trade-off between skin tone and makeup color. In this work, we assume that the makeup will not drastically change the skin tone. Moreover, as indicated in Fig. 6, even without the makeup loss of the skin region, our network precisely extracts the makeup of the cheeks using the alpha blending approach because we use two discriminators to distinguish the makeup and non-makeup textures. The details are discussed in Sec. 5.

We use $\omega_{m}^{cyc}=20$, $\omega_{m}^{vgg}=2$, $\omega_{m}^{adv}=5$, $\omega_{m}^{tv}=8$, and $\omega_{m}^{make}=1$ as the balancing terms. Our makeup extraction results are presented in Figs. 5,  6, and  7.

We followed the training strategy of  [DYX^∗19] and use the same datasets for approximately 260K face images. We used [BT17] to detect and crop the faces for alignment. The image size was $256\times 256$ and the resolution of FLAME albedo textures was $256\times 256$. We initialized the network with the weights of the pre-trained [RDS^∗15] and modified the last layer to estimate our own 3DMM parameters. The batch size was 8, and the learning rate was $1\times 10^{-4}$ using an Adam optimizer with 20 training epochs. For the shininess parameter $\mathbf{\rho}$, we set the initial value to 200 to achieve highlight effects.

We sampled approximately 100K images from FFHQ [KLA19] and CelebA-HQ [KALL18] in the UV space to train DSD-GAN for the UV completion, and the resolution of the UV textures was $512\times 512$. Prior to sampling, the face images were segmented using the method of  [YWP^∗18] and only the skin region of the face was used. Subsequently, we performed a manual cleanup to remove the low-quality textures. Eventually, 60,073 textures remained for training.

The optimization-based refinement process was executed with 500 iterations for each texture. The learning rate of the Adam optimizer was set to $1\times 10^{-2}$ with a 0.1 learning rate decay. The kernel size of the Gaussian blur filter $K$ was set to 11.

We combined two makeup datasets, namely the MT dataset [LQD^∗18] and LADN dataset [GWC^∗19], which consisted of 3,070 makeup images and 1,449 non-makeup images. A total of 300 makeup images were randomly selected for testing. By implementing the previous steps, both the makeup and non-makeup images were processed into albedo textures to train our network. We trained the network with 40 epochs and batch size of 1. The Adam optimizer used a learning rate of $1\times 10^{-4}$. The makeup extractor ${E}$ had the same architecture as the generator of DSD-GAN, and PatchGAN  [IZZE17] was used for the discriminators $M$ and $B$.

We used Nvdiffrast [LHK^∗20] for the differentiable renderer and trained the networks using a single NVIDIA GeForce RTX 2080 Ti GPU. Our training required approximately 3 days for the coarse facial reconstruction network and approximately 1 day for the makeup extraction network. Approximately 1 minute was required to process a texture in the refinement step.

First, we demonstrate how the extracted makeup dataset can be used to enhance the 3D face reconstruction of makeup. The same process as the diffuse albedo model construction of FLAME [LBB^∗17] was followed, and the collected makeup textures $(1-\mathbf{A})\odot\mathbf{D}_{m}^{m}$ were used to construct a PCA-based statistical model of makeup. The randomly sampled textures from the makeup model are depicted in Fig. 10. The makeup model is an extension of the diffuse albedo model, and the new model $\mathbf{D}_{c}^{{}^{\prime}}$ can be formulated as:

where $\mathbf{D}_{c}^{m}$ is the makeup model. We used an optimization-based manner to reconstruct the 3D face from the makeup portraits because no large-scale makeup dataset is available to train a neural network. A comparison of the 3D makeup face reconstruction using different albedo models $\mathbf{D}_{c}$ and $\mathbf{D}_{c}^{{}^{\prime}}$ is presented in Fig. 11. $\mathbf{D}_{c}^{{}^{\prime}}$ could recover the makeup, and improve the accuracy of the entire reconstruction, especially for lipsticks and eye shadows. The shape of the lips and eyes were also matched more effectively by extending the ability of the diffuse albedo.

We believe that this makeup database can be explored further; for example, by using advanced image generation techniques such as StyleGAN/StyleGAN2 [KLA19, KLA^∗20] or diffusion model [HJA20]. The accuracy of the reconstruction results also requires further quantitative evaluation, and we consider these as future research topics.

We used the extracted makeup for makeup transfer by employing the same method as that for the makeup extraction network. Equation (10) was followed to blend a new makeup face, which was subsequently projected onto the original image. The functionalities of our method and previous methods are summarized in Tab. 2. Similar to the approach of CPM [NTH21], the makeup textures are UV representations to solve the misalignment of faces. The face region that is used for transfer can be specified, and the makeup is editable. In addition to these advantages, our approach extracts makeup and uses a completed UV representation, which can handle occlusion and illumination.

Our makeup transfer results are depicted in Fig. 12. The reference images contained various complex factors, including occlusion, face misalignment, and lighting conditions. We extracted the makeup from the reference image. Subsequently, we transferred the makeup to the source image while maintaining the original illumination of the source image.

A qualitative comparison with state-of-art makeup transfer methods is depicted in Fig. 13, in which the source and reference images have different illumination. As existing makeup transfer methods do not consider illumination, they transfer not only the makeup, but also the effect of the illumination. The first two rows present the exchanged makeup results of two identities, one was evenly illuminated, whereas the other had shadows on the cheeks. The existing methods could not retain the illumination from the source image, which resulted in a mismatch between the face and surrounding environment. Note that our method preserved the illumination and shade of the original images in the cheeks. The final row shows an example in which the source and reference images have contrasting illumination. Our method enabled a natural makeup transfer, whereas the other methods were affected by the illumination, resulting in undesirable results. Note that the pioneering method known as BeautyGAN [LQD^∗18] (a) was relatively stable. However, as it is not suitable for transferring eye shadow, the makeup in the source image was not cleaned up.

As illustrated in Fig. 14, compared to existing makeup transfer methods, our method could achieve makeup interpolation and removal without a reference image. Moreover, our makeup interpolation maintained constant illumination conditions and achieved natural makeup interpolation. The first two rows present the interpolation and removal results of the face images. The specular and diffuse shading were not changed while the makeup was adjusted from heavy to light. The third row shows how the alpha matte changed in the makeup interpolation process. The alpha matte $\mathbf{A}$ is adjusted to $\mathbf{A}_{\sigma}$ as follows:

where $\mathbf{A}_{\sigma}(p)$ and $\mathbf{A}(p)$ are values of $\mathbf{A}_{\sigma}$ and $\mathbf{A}$ at pixel $p$, respectively, and $\sigma\in[0,1]$. $\mathbf{A}_{\sigma}$ was eventually clipped to between 0 and 1. It can be observed that for the change in $\mathbf{A}_{\sigma}$, the makeup was mainly lipstick and eye shadow in this sample. Thus, the original $\mathbf{A}$ was the black color around the lips and eyes, which means that this part of the skin color was not used, while the makeup is mainly applied. With the increase in $\\ sigma$, $\mathbf{A}_{\sigma}$ became whiter; thus the use of makeup was reduced and makeup interpolation was achieved. As the illumination was disentangled from the makeup, it was possible to relight the face while adjusting the makeup. The results are presented in the final row.

In addition to the above applications, the extracted makeup can be used for makeup recommendations, please refer to [SRH^∗11, AJWF17]. The makeup textures can also facilitate the subsequent processing of traditional graphics pipelines such as physically-based makeup rendering.