DeepFacePencil: Creating Face Images from Freehand Sketches

Given an input image from one domain, an image-to-image translation model outputs a corresponding image from another domain and preserves the content in the input image. Existing image-to-image translation models are based on generative adversarial networks conditioned on images. Pix2pix (Isola et al., 2017) is the first general image-to-image translation model which can be applied to different scenarios according to paired training images, such as, semantic maps to natural images, day images to night images, image colorization, and edge maps to images. (Karacan et al., 2016) utilizes semantic label maps and attributes of outdoor scenes as input and generates the corresponding photo-realistic images. In order to model multi-modal distribution of output images, BicycleGAN (Zhu et al., 2017b) encourages the connection between the output and the latent code to be invertible. CycleGAN (Zhu et al., 2017a), DualGAN (Yi et al., 2017), and DiscoGAN (Kim et al., 2017) propose unsupervised image translation models with a common idea named cycle consistency, which is borrowed from language translation literature. Pix2pixHD (Wang et al., 2018) is proposed as a high-resolution image-to-image translation model for generating photo-realistic image from semantic label maps using a coarse-to-fine generator and a multi-scale discriminator. It can also be applied to edge-to-photo generation when trained on paired edge maps and photos. However, the large gap between synthesized edge maps and hand-drawn sketches challenges the generalization of these models.

Sketch-based image generation is a hot topic in multimedia and computer graphics. Given a sketch which describes the desired scene layout with text labels for objects, traditional methods, such as Sketch2Photo (Chen et al., 2009) and PhotoSketcher (Eitz et al., 2011), search image patches from a large-scale image dataset and fuse the retrieved image patches together according to the sketch. These methods are not able to ensure the global consistency of the resultant image and fail to generate totally new images. Nevertheless, it is challenging for these methods to ensure global consistency of the resultant images. Thus they frequently fail to generate totally new images. After the breakthroughs made by deep neural networks (DNNs) in many image understanding tasks, a variety of DNN-based models have been proposed for sketch-based image generation. The general image-to-image translation models mentioned above can be easily extended to sketch-based image generation once sketches and their corresponding images are available as training data. Besides, a few other models are designed specially for sketch inputs. SketchyGAN (Chen and Hays, 2018) aims to generate real images from multi-class sketches. A novel neural network module, called mask residual unit (MRU), is proposed to improve the information flow by injecting the input image at multiple scales. Edge maps are extracted from real images and utilized as training sketches. However, the resultant images of SketchyGAN are still not satisfied. LinesToFacePhoto (Li et al., 2019) employs a conditional self-attention module to preserve the completeness of global facial structure in generated face images. However, this model cannot be generalized to hand-drawn sketches directly due to distinct stroke characteristics.

Recently, studies on face image generation and editing have made tremendous progress. Generative adversarial network (GAN) (Goodfellow et al., 2014), which generates images from noise, is widely used in a wide range of applications (Ledig et al., 2017; Zhang et al., 2017; Zhang et al., 2019; Huang et al., 2020; Liu et al., 2019; Zha et al., 2020). Using GAN, realistic face images can be generated from noise vectors. DCGAN (Radford et al., 2016) introduces a novel network to stabilize training of GAN. PGGAN (Karras et al., 2018) utilizes a progressively growing architecture to generate high-resolution face images. Inspired by style transfer literature, StyleGAN (Karras et al., 2019) introduces a novel generator which synthesizes plausible high-resolution face images and learns unsupervised separation of high-level attributes and stochastic variation in synthesized images. On the other side, a number of works focus on face image editing through different control information. StarGAN (Choi et al., 2018) designs a one-to-many translation framework which switches face attributes assigned by an attribute code. FaceShop (Portenier et al., 2018) and SC-FEGAN (Jo and Park, 2019) treat sketch-based face editing as a sketch-guided image inpainting problem where stroke colors are also applied as guidance information. In this work, we focus more on synthesizing realistic face images from imperfectly drawn sketches.

Paired face sketch-photo dataset is required for supervised sketch-to-face translation methods. Since there exits no large-scale paired sketch-image dataset, the training sketches used by existing methods (Isola et al., 2017; Li et al., 2019) are generated from face image dataset, e.g. CelebA-HQ face dataset, using edge detection algorithm such as HED (Xie and Tu, 2015). However, the level of details in edge maps rely heavily on the value of a threshold in the edge detection algorithm. An edge map with a large threshold contains too many redundant edges while an edge map with a small threshold fails to preserve the entire global facial structure (Li et al., 2019). Pix2pixHD (Wang et al., 2018) introduces another method to generate sketches from face images. Given a face image, the face landmarks are detected using an off-shelf landmark detection model. A new kind of sketch, denoted as face contour, is obtained by connecting the landmarks in a fixed order. However, since the pre-defined face landmarks mainly depict the facial area while ignoring details, a sketch-to-face model trained by face contours fails to generalize to hand-drawn sketches with hair, beard, or ornaments .

Based on the discussion above, we utilize a new kind of generated sketches with the assist of semantic maps. The CelebAMask-HQ dataset (Lee et al., 2020) provides a face semantic map for each face image in CelebA-HQ dataset. We use the boundaries of the semantic map as a synthesized sketch of the corresponding face image. Figure 1 compares different styles of synthetic sketches, including an edge map (b), a face contour (c), region boundaries of semantic maps (d), and deformed boundary map (e) from the same real image (a).

A sketch-to-image model trained with edge-aligned sketch-image pairs tends to generate images whose edges strictly align with the strokes of the input sketch. When an input hand-drawn sketch is not well-drawn, line distortions in the input sketch damage the quality of the generated face image. There is a trade-off between the realism of the generated face image and the conformance between the input sketch and the output face image. In order to alleviate the edge alignment between the input sketch and the output face image, we propose to relax thin strokes to a tolerance region with various width. A straightforward way is to smooth the strokes to multi-pixel width by image smoothing or dilation. However, the capacity of this hand-crafted way is limited, because the uniform smoothness for all positions of the whole sketch violates the unevenness of hand-drawn sketches on depicting different facial parts. We argue that the balance between the realism and the conformance differs from one position to another across the face image. Even in the same input sketch, the user might draw strokes in some parts (such as eyes and mouth) carefully with little distortions, while draw strokes in other parts (such as chin and hair) roughly. The balance moves towards conformance at a well-drawn part with realistic shape to meet the user’s intention, while it moves towards realism at a poorly-drawn part to ensure the quality of the generated image. Therefore, the relaxation degree should be spatially varying.

We propose a new module, called spatial attention pooling (SAP), to adaptively relax strokes in the input sketch to spatially varying tolerance regions. A stroke with a larger width indicates less restriction between this stroke and the corresponding edge in the synthesized image. The stroke widths are controlled by the kernel sizes of pooling operators. However, the kernel size of an pooling operator is not trainable using back propagation. Instead, we apply multiple pooling operators with different kernel sizes to get multiple relaxed sketches with different stroke widths. The relaxed sketches are then fused by a spatial attention layer which spatially adjusts the balance between realism and conformance.

The architecture of SAP is shown in Figure 3. Given an input deformed sketch $\mathcal{S}_{dfm}\in{\mathbb{R}}^{H\times W}$, we first pass it through $N_{r}$ pooling branches with different kernel sizes of $\{r_{i}\}_{i=1,\ldots,N_{r}}$ to get $P_{i}=Pooling(\mathcal{S}_{dfm};r_{i})$. Then we utilize convolutional layers to extract feature maps of $P_{i}$ separately. These feature maps are concatenated to get a relaxed representation of $\mathcal{S}_{dfm}$, denoted as $R$:

where $Conv_{i}()$ indicates convolutional layers for each pooling branch, and $Concat$ is a channel-wise concatenation operator.

In order to control the relax degrees of all positions, we compute a spatial attention map $A$ to assign different attention weights to $R$. A stroke with more distortion is supposed to be assigned with larger relax degree. Hence, $A$ is supposed to adaptively pay more attention (a larger weight) to a $Conv_{i}(P_{i})$ that comes from a larger pooling kernel size in the areas with more distortions. In order to perceive the distortion degree of different regions in a hand-drawn sketch, we employ a pre-trained binary classifier $C$ with three convolutional layers to distinguish edge-aligned $\mathcal{S}_{syn}$ sketches from deformed sketches $\mathcal{S}_{dfm}$. We utilize this classifier to extract features of the input sketch $\mathcal{S}_{dfm}$ to get ${F^{C}_{i}(\mathcal{S}_{dfm}),i=1,2,3}$, where $F^{C}_{i}()$ denotes the feature maps extracted by the $i^{th}$ convolution layers of $C$. These feature maps emphasize the differences between edge-aligned sketches and deformed sketches. We resize and concatenate these feature maps, and pass them through three convolutional layers to get the spatial attention map:

where $Up_{2}$ and $Up_{4}$ indicates $2\times$ and $4\times$ upsampling, $Conv()$ indicates three cascaded convoutional layers, and $softmax()$ is a softmax layer computed over channels to ensuring that for each position of $A$, the sum of weights of all channels equals to $1$.

At last, the output SAP is computed as:

where $*$ is element-wise multiplication.

The goal of SAP is to sense stroke distortions in different regions of a hand-drawn sketch and then relax the edge alignment accordingly in the synthesized face image. However, directly adding the SAP module at the front end of a general image translation network does not guarantee that the SAP can be effectively trained under the loss with regard to the synthesized image only. Following a general image translation network architecture, the realism of synthesized images relies on the coincidence of the embedded structure from the input sketch and real face images. In our dual-generator framework shown in Figure 2, we enforce the embedded features from the deformed sketch with SAP to be consistent with the synthesized edge-aligned sketches by introducing a generator feature matching loss $\mathcal{L}_{GFM}$ between the embedded features from the deformed sketch $\mathcal{S}_{syn}$ and the edge-aligned sketch $\mathcal{S}_{dfm}$. The main generator $G_{m}$ for deformed sketches, the auxiliary generator $G_{a}$ for edge-aligned sketches, and the discriminator $D$ are trained with the following objectives.

Before showing experimental results, we first introduce details in our network implementation and training.

In order to verify the generalization ability of our model, we compare our model with state-of-the-art image translation models by testing with synthesized sketches of different levels of deformation, well-drawn sketches and poorly-drawn sketches by common users.