FFHQ-UV: Normalized Facial UV-Texture Dataset for 3D Face Reconstruction

To extract high-quality texture maps from in-the-wild face images, we first derive multi-view, normalized face images from single-view images, where the resulting face images have even illumination, neutral expression, and no occlusions (e.g., eyeglasses, hairs). Specifically, we employ StyleFlow [1] and InterFaceGAN [37] to automatically edit the image attributes in $\mathcal{W}+$ latent space of StyleGAN2 [21]. For each in-the-wild face image $I$, we first obtain its latent code $w$ in $\mathcal{W}+$ space using the GAN inversion method e4e [38], and then detect the attribute values of the inverted image $I_{inv}=G(w)$ from StyleGAN generator $G$, so that we can normalize these properties in the following semantic editing. The attributes we intend to normalize include lighting, eyeglasses, hair, head pose, and facial expression. The lighting condition, represented as spherical harmonic (SH) coefficients, are predicted using DPR model [44], and the other attributes are detected using Microsoft Face API [28].

For the lighting normalization, we set the target lighting SH coefficients to keep only the first dimension and reset the rest dimensions to zeros, and then use StyleFlow [1] to get evenly illuminated face images. As in the SH representation, only the first dimension of the SH coefficients represents uniform lighting from all directions, while the other dimensions represent lighting from certain directions which are undesired. After the lighting normalization, we normalize the eyeglasses, head pose, and hair attributes by setting their target values to $0$, and obtain the edited latent code $w^{\prime}$. For the facial expression attribute, similar to InterFaceGAN [37], we find a direction $\beta$ of editing facial expression using SVM, and achieve the normalized latent code $\hat{w}$ by walking along the direction $\beta$ starting from $w^{\prime}$. To avoid over-editing, we further introduce an expression classifier to decide the stop condition for walking. Here, we obtain the normalized face image $I_{n}=G(\hat{w})$. Finally, two side-view face images $I_{n}^{l}$ and $I_{n}^{r}$ are generated using StyleFlow [1] by modifying the head pose attribute.

The process of extracting UV-texture from a face image, also termed “unwrapping”, requires a single-image 3D face shape estimator. We train a Deep3D model [13] with the recent 3DMM basis HiFi3D++ [8] to regress the 3D shape in 3DMM coefficients, as well as the head pose parameters, from each normalized face image. Then the facial UV-texture is unwrapped by projecting the input image onto the 3D face model. In addition, we employ a face parsing model [45] to predict the parsing mask for the facial region, so that non-facial regions can be excluded from the unwrapped texture UV-map. In this way, for each face we obtain three texture maps, $T_{f}$, $T_{l}$, and $T_{r}$ from frontal, left, and right views, respectively. To fuse them together, we first perform a color matching between them to avoid color jumps. The color matching is computed from $T_{l}$ and $T_{r}$ to $T_{f}$ for each channel in YUV color space:

$$T_{a}^{\prime}=\frac{T_{a}-\mu(T_{a})}{\sigma(T_{a})\times\omega}\times\sigma(T_{b})+\mu(T_{b}),$$

(1)

where $T_{b}$, $T_{a}$, and $T_{a}^{\prime}$ are the target texture, source texture, and color-matched texture, respectively; $\mu$ and $\sigma$ denote the mean and standard deviation; and $\omega$ is a hyper-parameter (set to $1.5$ empirically) used to control the contrast of the output texture. Finally, the three color matched texture maps are linearly blended together using pre-defined visibility masks (see Fig. 2) to get a complete texture map $T$.

The obtained texture map $T$ usually contains artifacts near eyes, mouth, and nose regions, due to imperfect 3D shape estimation. For example, eyeball and mouth interior textures^***In our texture UV-map, only eyelids and lips are needed since the eyeballs and mouth interiors are independent mesh models similar to the mesh topology used in HiFi3D++ [8]. that are undesired would appear in the texture map if the eyelids and lips between the estimated 3D shape and image are not well aligned. While performing local mesh deformation according to image contents [44] could fix the artifacts in some cases, we find it would still fail for many images when processing large-scale dataset. To handle these issues, we simply extend and exclude error-prone regions and fill the missing regions with a template texture UV-map $\bar{T}$ (see Fig. 2). Specifically, we extract eyeball and mouth interior regions predicted from the face parsing result, and then unwrap them to the UV coordinate system after a dilation operation to obtain the masks of these artifacts $M_{leye}$, $M_{reye}$, and $M_{mouth}$ on its texture UV-map. As for the nose region, we extract the nostril regions by brightness thresholding around the nose region to obtain a mask $M_{nostril}$, since nostril regions are usually dark. Then the regions in these masks are filled with textures from template $\bar{T}$ using Poisson editing [32] to get a texture map $\hat{T}$.

Finally, to obtain a complete texture map beyond facial regions (e.g., ear, neck, hair, etc.), we fill the rest regions using template $\bar{T}$, with a color matching using Eq. (1) followed by Laplacian pyramid blending [7]. The final obtained texture UV-map is denoted as $\hat{T}_{final}$.

We apply the above pipeline to all the images in FFHQ dataset [20], which includes 70,000 high-quality face images with high variation in terms of age and ethnicity. Images with facial occlusions that cannot be normalized are excluded using automatic filters based on face parsing results. The final obtained UV-maps are manually inspected and filtered, leaving 54,165 high-quality UV-maps at $1024\times 1024$ resolution, which we name as FFHQ-UV dataset. Tab. 1 shows the statistics of the proposed dataset compared to other UV-map datasets.

Fig. 3 shows an example of the UV-maps obtained with different baseline methods. The baseline “w/o editing” (see Fig. 3(a)) yields a UV-map that has significant shadows and occlusions of hair. The UV-map generated by the baseline “w/o multi-view” (see Fig. 3(b)) contains smaller area of actual texture and heavily relies on template texture for completion. For the baseline method “naive blending”, there are obvious artifacts near eyes, mouth, and nose regions (Fig. 3(c)). In contrast, our pipeline is able to produce high-quality UV-maps with complete facial textures (Fig. 3(d)).

We expect FFHQ-UV would inherit the data diversity of FFHQ dataset [20]. To verify this, we compute the identity vector using Arcface [12] for each face, and then calculate the standard deviation of these identity vectors to measure the identity variations of the dataset. Tab. 2 shows the identity standard deviation of the original dataset (FFHQ), the dataset inverted to the latent space (FFHQ-Inv), the normalized dataset using our StyleGAN-based facial image editing (FFHQ-Norm), the rendered face images using our facial UV-texture dataset (FFHQ-UV), where FFHQ-UV preserves the most identity variations in FFHQ (over $90\%$). Furthermore, our dataset has a higher identity standard deviation value compared to Facescape dataset [42], indicating that FFHQ-UV is more diverse. To analyze the identity preservation from FFHQ-Norm to FFHQ-UV, we compute the identity similarity between each image in FFHQ-Norm and the rendered image using the corresponding UV-map in FFHQ-UV with a random head pose. Tab. 3 shows that the average identity similarity score over the whole dataset achieves $0.78$, which is pretty satisfactory considering that the rendered images are in different views (with random head poses) from real images. The table also shows that our result is superior to the results obtained by the baseline variants of our UV-map creation pipeline.

Having even illumination is one important aspect for measuring the quality of a UV-map [26]. To quantitatively evaluate this aspect, we present a new metric, namely Brightness Symmetry Error (BS Error) as follows

$$BS\_Error(T)=\left\|\mathcal{B}_{\alpha}(T^{Y})-\mathcal{F}_{h}(\mathcal{B}_{\alpha}(T^{Y}))\right\|_{1},$$

(2)

where $T^{Y}$ denotes the $Y$ channel of $T$ in YUV space; $\mathcal{B}_{\alpha}(\cdot)$ denotes the Gaussian blurring operation with the kernel size of $\alpha$ (set to $55$ empirically); $\mathcal{F}_{h}(\cdot)$ denotes the horizontal flip operation. The metric is based on the observation that an unevenly illuminated texture map usually has shadows on the face which makes the brightness of UV-map asymmetrical. Fig. 4 shows two examples of the computed BS Error on UV-maps. Tab. 4 shows the average BS Error computed over the whole dataset, which demonstrates that the StyleGAN-based editing step in our pipeline effectively improves the quality in terms of more even illumination. In addition, the BS Error of our dataset is competitive with that of Facescape [42], which is captured under controlled conditions with even illumination, indicating that our dataset is indeed evenly illuminated.