Joint Geometric-Semantic Driven Character Line Drawing Generation

The image-to-image translation (Mao et al., 2022)(Wang et al., 2022)(Richardson et al., 2021)(Pang et al., 2021) can be formulated as an image generation function that maps the given source domain image into the desired target artistic style. It has a wide range of applications, including image cartoonization (Dong et al., 2021)(Wang and Yu, 2020), flat filling (Zhang et al., 2021b)(Wu et al., 2021), face sketch synthesis (Li et al., 2021)(Cao et al., 2022) and image inpainting (Zhao et al., 2020)(Liu et al., 2021). Particularly, the generative adversarial network based image translation has been receiving substantial interest. Based on conditional adversarial network, Pix2pix (Isola et al., 2017) develops a general image translation framework for different tasks, like colorization, sketch-to-portrait. However, the training instability makes it difficult to apply pix2pix to high-resolution image generation issue. Wang et al. (Wang et al., 2018) introduced the multi-scale generator and discriminator models together with a new feature match loss to form pix2pixHD to achieve high-resolution results. Qi et al. (Qi et al., 2021) developed a semantic-driven GAN with pix2pix as backbone, and designed an adaptive re-weighting loss to balance semantic classes’ contributions. In contrast with learning from paired data, unsupervised image translation has recently attracted more and more attention. CycleGAN (Zhu et al., 2017) designs a cycle consistency constraint to conduct learning with unpaired inputs. Similarly, DualGAN (Yi et al., 2017) also proposes a general-purpose image translation framework using a cyclic mapping. While DiscoGAN (Kim et al., 2017) couples two different GANs for bidirectional mapping between source and target domains. Based on the assumption that corresponding images from different domains share the same intermediate representation in a latent space, Liu et al. (Liu et al., 2017) integrated the core ideas of GANs and variational autoencoders (VAEs) forming the UNIT framework to model each image domain. MUNIT (Huang et al., 2018) assumes that the representation of source and target images can be decomposed into a shared content space but different domain-specific style spaces. To achieve image translation, it performs a swapping combination of these content and style codes.

The deep learning methodology has already pushed significant steps in a wide variety of visual processing tasks, which makes great contributions to line drawing related applications, including line drawing colorization (Yuan and Simo-Serra, 2021), artistic shadow creation (Zhang et al., 2021a), manga structural line extraction (Li et al., 2017), and face portrait line drawing (Yi et al., 2020). Zhang et al. (Zhang et al., 2021b) proposed a split filling mechanism to perform line art colorization. They first spilt the input user scribbles into several groups for influence areas estimation, then a data-driven color generation process is conducted for each group. These outputs are finally merged to form the high-quality filling results. Zheng et al. (Zheng et al., 2020) first designed a ShapeNet to learn the 3D geometric information from line drawings, then a RenderNet is performed to produce 3D shadow. The SmartShadow (Zhang et al., 2021a) application consists of three data-driven tools, a shadow brush used to determine the areas the user want to create shadow, a shadow boundary brush to control the shadow boundaries, and a global shadow generator for the entire image shadow generation based on the estimated global shadow direction. Li et al. (Li et al., 2017) took advantage of the ideas of residual network and symmetric skipping network to build their CNN based method to solve problem of structural line extraction from manga image. To generate portrait drawings in multiple styles and unseen style, Yi et al. (Yi et al., 2022) designed a novel portrait drawing generation architecture using a asymmetric cycle structure, where a regression network is first adopted to calculate the quality score for an APDrawing. Based on this model, they defined a quality loss to guide the network to generate high-quality APDrawings. Im2Pencil (Li et al., 2019) and (Zhang et al., 2021c) treat the procedure of line drawing generation as an independent subtask to facilitate manga and Illustration synthesis, respectively.

Different from these methods, we apply the generative adversarial network to a new problem, i.e. our P2LDGAN mainly concentrates on finding cross-domain relations between character images/photos and hand-drawn character line drawings using paired data.

We attempt to generalize the application of generative adversarial framework to the photo-to-line drawing style conversion problem, i.e. automatic artistic character line drawings generation from real images/photos. Given the well-aligned source-reference pair $\{p_{i},l{i}\}_{i=1}^{N}$, where $p_{i}$ and $l_{i}$ belong to character photo domain $\mathcal{P}$ and line drawing domain $\mathcal{I}$, respectively. $N$ denotes the number of image pairs. The goal of our model is to learn a mapping function $\mathcal{F}$ that automatically discover the domain relations between line drawing $\hat{l}_{i}$ and corresponding character photo.

Figure 3 illustrates the details of our developed P2LDGAN for this cross domain corresponding learning, which consists of (1) a joint geometric-semantic driven generator G, and (2) a character line drawing discriminator D. In the following, we give a detailed description of our method.

The character line drawings describe the characters’ clear and critical features using structural lines representation. Therefore, both geometric and semantic information play a fairly important role in synthesizing vital details in drawings. Previous state-of-the-art image translation models mainly adopt U-net framework with skip connections. However, these methods only combine the feature maps of the same scale from encoding and decoding stage, lacking geometric and semantic information fusion, which limits the generation quality.

Therefore, in order to progressively propagate the geometric and semantic information into the output line drawing, we development a simple but effective framework, named cross-scale dense skip connections module, which is the core of our joint geometric-semantic driven generator.

Our generator fundamentally is an encoder-decoder architecture, where the encoding stage compresses the rich information in character photo into latent representation, while the decoding network constructs the desired line drawings from encoded representation. As shown in the bottom-left part of Figure 3, we adopt the pre-trained ResNeXt-50 as encoder because of its simpleness, modularization, efficiency and higher learning capability. Specifically, we extract the stages conv2-conv5 from ResNeXt as our encoding layer (i.e. ResNeXt Layer), which can be formulated as,

Where $i$ indicates the $i$th layer of the encoder, $\mathcal{F}_{i-1}$ and $\mathcal{F}_{i}$ denoted the input and output. $C$ in the number of groups. $\mathcal{T}$ means transformation function, here we use a sequence of $1\times 1$, $3\times 3$ and $1\times 1$ convolutions.

We input a character image/photo of size $512\times 512$ into encoder, and extract feature maps from each ResNeXt layer together with our decoding layer to build the cross-scale skip connections model to fuse and propagate information of different levels of abstractness. To be specific, suppose our generator has $n$ layers in total, the input of the current decoding layer $n-i+1$ is the combination between the outputs of the previous layer and corresponding encoding layers having the same or larger resolutions. The formulation is defined as,

Where $\mathcal{F}_{n-i}$ represents the previous $n-i$th layer’s output, $\mathcal{F}_{1}$ to $\mathcal{F}_{i}$ denote the outputs of the 1st to $i$th encoding layers with scales larger than or equal to that of the $n-i+1$ layer. PDS() refers to progressive scaling operation to scale feature maps using convolutions. $\oplus$ is the element wise sum operation. Subsequently, the fused multi-scale features flows through the decoding layer to perform geometric and semantic information propagation into a higher-resolution maps.

Here, UP() operation is used to upsample the feature maps by a factor of 2 using nearest neighbor algorithm. Through our cross skip connection mechanism, feature maps from each encoding layer can be embedded into different decoding layers to strengthen semantic and geometric information integration and improve feature propagation across encoder and decoder, so that we can directly train our generator to learn character line drawing modality and translation the real character image/photo into line drawing with details preservation.

For discriminator network, its task is to discriminate the generated character line drawings from ground truth. Here, we adopt the PatchGAN exploited in (Isola et al., 2017) to classify whether the $32\times 32$ patches are real or not. The bottom-middle part of Figure 3 displays discriminator architecture. We can observe that each discriminator block is composed of $4\times 4$ Convolution, InstanceNorm and LeakyReLU with a slop of 0.2. Such a discriminator contributes to high-quality line drawings generation since it makes our P2LDGAN pay more attention to detailed content in patches, and can process images of any size with fewer parameters (Zhu et al., 2017).

Adversarial Loss. The generator $G$ creates character line drawing $\hat{l}$ from image $p$ in character photo domain $\mathcal{P}$, which can not be distinguished by discriminator $D$, while the goal of discriminator $D$ is to discriminate between translated drawings and real samples $l$ from $\mathcal{I}$. Motivated by the significant stability and high-quality generation capability of relativistic average GANs (Jolicoeur-Martineau, 2018), we introduce the following Adversarial loss to supervise our P2LDGAN for more realistic line drawing synthesis.

Where $f_{1}$ is mean square error loss. $f_{2}$ performs mean operation. $f_{4}$ patches the input photo with the same size as outputs of discriminator $D$, $f_{3}$ and $f_{5}$ are used to fill tensor with scalar value 1 and 0, respectively.

Pixel-wise Loss. Since we use paired samples to train our model, we can introduce the $L_{1}$ loss (Chen and Hays, 2018) (compared to $L_{2}$ loss, it introduce less blurriness (Yi et al., 2017)) to measure the error between the generated and ground truth character line drawings, which can make translated drawings follow the domain $I$ distribution. The function is defined as,

The final objective loss function of our P2LDGAN is given by,

In our experiments, we set the weights $\lambda_{1}=1$, $\lambda_{2}=0.5$, and $\lambda_{3}=100$, respectively.

We implement the photo-to-line drawing translator using PyTorch. All experiments are conducted on a single NVIDIA GeForce RTX 3090 GPU. For both the generator and discriminator network, we use the Adam optimizer with $\beta_{1}=0.5$ and $\beta_{2}=0.999$. The initial learning rate is set to 0.0002. We train the model for 200 epochs with a mini batch size of 1. All input images and ground truth are aligned and resized to 512 $\times$ 512 pixels.

To measure the quality of synthesized line drawing images, three commonly-used similarity metrics, namely Fr$\acute{e}$chet Inception Distance (FID) (Heusel et al., 2017) and Structural Similarity Metric (SSIM) (Wang et al., 2004) and Peak Signal-to-noise Ratio (PSNR) (Chen et al., 2020), are adopted to quantitatively evaluate the performance of previous methods and our proposed P2LDGAN. Where the FID calculates the distribution distance between the set of line drawing images created from input photos and the corresponding ground truth drawings (the smaller the FID value is, the better the drawing quality will be ), while SSIM describes the similarity between the generated image and ground truth (Higher SSIM value indicates better result). PSNR computes the intensity difference between predicted and ground truth images (Larger PSNR scores means smaller difference between images) (Pang et al., 2021).

To demonstrate the superior performance of our P2LDGAN model, we make the quantitative and qualitative comparisons with five state-of-the-art networks, including one existing neural style transfer method (i.e. Gatys (Gatys et al., 2016)) as well as four general image-to-image translation approaches, CycleGAN (Zhu et al., 2017), DiscoGAN (Kim et al., 2017), UNIT (Liu et al., 2017), pix2pix (Isola et al., 2017), MUNIT (Huang et al., 2018). It is worth noting that we also use paired data to train unsupervised methods.

We replace the sum operation in cross-scale skip connections modules with concatenation. From Figure 5, we can report that the usage of concatenation achieves acceptable results but having unclear boundaries, some roughtness and noiseness. The numerical values in Table 4 also demonstrate the influence of fusion strategies had on drawings quality.

In order to validate the effectiveness of our cross-scale skip connections model in keeping fine details, We perform quantitative and qualitative comparisons between our P2LDGAN with/without using cross-scale skip connections model (our baseline). From the visual results, it can be reported that without our connection model, the generated line drawings have many structural details loss (e.g., hair area in the second row), resulting in poor visual effects, while our P2LDGAN could restore these delicate structures, producing robust and better looking results.

We also replace our decoding layer with a DeConvolution + Upsample + Convolution block. As can be found in Figure 5 and Table 4, the network using replaced decoder produces the character line drawings that are similar to or even better than our proposed models. However, it requires more parameters and computation consumption than P2LDGAN.

Summarily, our well-designed P2LDGAN shows superior performance in translating character images/photos into high-quality line drawings with better detailed structure, clear lines, and fewer artifacts.