StrokeGAN: Reducing Mode Collapse in Chinese Font Generation
via Stroke Encoding

In recent years, many generation methods of stylish Chinese fonts have been suggested in the literature (Tian 2017; Chang, Gu, and Zhang 2017; Chang et al. 2018; Chen et al. 2019; Wu, Yang, and Hsu 2020; Jiang et al. 2017, 2019; Zhang et al. 2020) with the development of deep learning. In (Tian 2017), the authors adapted pix2pix model developed in (Isola et al. 2017) for the image style translation problem to Chinese font generation and then suggested zi2zi method with paired training data, that is, there is a one-to-one correspondence between the characters in the source (input) style domain and target (output) style domain. Similar idea was extended to realize the Chinese character generation from one font to multiple fonts in (Chen et al. 2019). Besides (Tian 2017) and (Chen et al. 2019), some other paired data based Chinese font generation methods were suggested in (Chang, Gu, and Zhang 2017; Jiang et al. 2017; Wu, Yang, and Hsu 2020). However, it is usually human-intensive to build up the paired training data. In order to overcome this challenge, (Chang et al. 2018) adapted CycleGAN developed in (Zhu et al. 2017) for the image style translation to Chinese font generation based on unpaired training data. Yet, the CycleGAN based method suggested in (Chang et al. 2018) (called CCG-CycleGAN) may suffer from the mode collapse issue (Goodfellow et al. 2014). When mode collapse occurs, the generator produces fewer patterns for different inputs, and thus significantly degrades the diversity and quality of generated results.

Motivated by the observation from traditional Chinese character generation and recognition methods (see (Xu et al. 2005; Kim, Liu, and Kim 1999)) that the explicit stroke feature can provide much mode information for a Chinese character, in this paper, we incorporate such stroke information into the training of CycleGAN for Chinese font generation (Chang et al. 2018) to tackle the issue of mode collapse, via introducing a one-bit stroke encoding and certain stroke-encoding reconstruction loss. The very recent papers (Wu, Yang, and Hsu 2020; Jiang et al. 2019; Zhang et al. 2020) also incorporated some stroke or radical information of Chinese characters into Chinese font generation. Their main idea is firstly to utilize a deep neural network to extract the strokes or radicals of Chinese characters and then merge them by another deep neural network, which is very different to our idea of the use of a very simple one-bit stroke encoding. According to our later numerical experiments, our introduced one-bit stroke encoding is very effective.

The rest of this paper is organized as follows. In Section 2Preliminary Work, we present some preliminary work. In Section 3StrokeGAN for Chinese Font Generation, we introduce the proposed method in detail. In Section 4Numerical Experiments, we provide a series of experiments to demonstrate the effectiveness of the proposed method. We conclude this paper in Section 5Conclusion.

Generative adversarial networks (GAN) (Goodfellow et al. 2014) have achieved great achievements in the task of high-quality image synthesis. The classic GAN model consists of two parts: a generator $G$ and a discriminator $D$. The generator $G$ generates fake images, and the discriminator $D$ judges whether the images generated by $G$ are fake or real. Mathematically, GAN can be formulated as the following two-player minimax game between $G$ and $D$,

$\displaystyle\min_{G}\max_{D}\ \mathbb{E}_{x\sim\mathbb{P}_{data}}[\log D(x)]+\mathbb{E}_{z\sim\mathbb{P}_{z}}[\log(1-D(G(z)))]$

where $\mathbb{P}_{data}$ and $\mathbb{P}_{z}$ are the distributions of data $x$ and the input noise variable $z$ for the generator, respectively. In practice, the generator $G$ and discriminator $D$ are generally represented by some deep neural networks.

The conditional GAN (cGAN) was suggested in (Mirza and Osindero 2014) mainly to embed some conditional information such as the category information of samples. Such idea was later exploited in (Isola et al. 2017) for the image style translation. For cGAN, besides the original input $z$, the conditional information $c$ is also a part of input for $G$. Thus, the objective for GAN should be slightly modified for cGAN, shown as follows:

	$\displaystyle{\cal L}_{adv}(D,G)$	$\displaystyle=\mathbb{E}_{x\sim\mathbb{P}_{data}}[\log D(x)]$
		$\displaystyle+\mathbb{E}_{z\sim\mathbb{P}_{z},c\sim\mathbb{P}_{c}}[\log(1-D(G(z,c)))],$

where $\mathbb{P}_{c}$ represents the distribution of the referred conditional information $c$.

The training of cGAN (Isola et al. 2017) is based on the paired data, of which the collection is usually time-consuming and laborious. In order to overcome this challenge, CycleGAN was proposed in (Zhu et al. 2017) for the image style translation based on the unpaired data. The main idea of CycleGAN is to preserve key attributes between the source and target domains by utilizing a cycle consistency loss. Specifically, let $x\in{\cal X}$ and $x^{\prime}\in{\cal X}^{\prime}$ be two images from two different style domains conditioned on respectively some conditional information domains $c\in{\cal C}$ and $c^{\prime}\in{\cal C}^{\prime}$. In order to realize the bidirectional translation between them, two generators $G:{\cal X}\times{\cal C}\rightarrow{\cal X}^{\prime}$ and $G^{\prime}:{\cal X}^{\prime}\times{\cal C}^{\prime}\rightarrow{\cal X}$ are exploited. With these, the cycle consistency loss can be defined as follows:

	$\displaystyle{\cal L}_{cyc}(G,G^{\prime})$	$\displaystyle=\mathbb{E}_{x\sim\mathbb{P}_{x},c\sim\mathbb{P}_{c},c^{\prime}\sim\mathbb{P}_{c^{\prime}}}[\|x-G^{\prime}(G(x,c),c^{\prime})\|_{1}]$
		$\displaystyle+\mathbb{E}_{x^{\prime}\sim\mathbb{P}_{x^{\prime}},c^{\prime}\sim\mathbb{P}_{c^{\prime}},c\sim\mathbb{P}_{c}}[\|x^{\prime}-G(G^{\prime}(x^{\prime},c^{\prime}),c)\|_{1}],$

where $x\in{\cal X}$, $x^{\prime}\in{\cal X}^{\prime}$, $c\in{\cal C}$, $c^{\prime}\in{\cal C}^{\prime}$, and $\mathbb{P}_{*}$ represents the distribution of the associated domain $*$. The generators $G$ and $G^{\prime}$ are trained to make the cycle consistency loss ${\cal L}_{cyc}(G,G^{\prime})$ small.

From Figure 4, the stroke encoding of character is taken as a part of input of CycleGAN. To realize this, we introduce a simple one-bit encoding way to yield the associated stroke encoding for a given Chinese character. Specifically, according to Figure 2, there are in total 32 kinds of strokes to make up Chinese characters. Thus, for any given Chinese character $x$, we define its stroke encoding as a 32-dimensional vector $c\in\{0,1\}^{32}$ with the $i$-th entry being $1$ if the $i$-th kind of stroke is included in $x$ and otherwise $0$ for all $i$ from $1$ to $32$. In this paper, we only use the indicator function of such kind of stroke instead of its exact number for a given Chinese character mainly in consideration of the robustness of StrokeGAN. This is in general sufficient according to our later experiments (see, Figure 7).

The training loss for StrokeGAN consists of three parts, that is, the general adversarial loss, cycle consistency loss and stroke-encoding reconstruction loss, where the stroke-encoding reconstruction loss is firstly introduced in this paper for the generation of stylish Chinese fonts.

A. Adversarial loss. The first part of loss is the adversarial loss defined commonly as follows,

	$\displaystyle{\cal L}_{adv}(D,G)$	$\displaystyle=\mathbb{E}_{x}[\log D_{src}(x)]$		(1)
		$\displaystyle+\mathbb{E}_{x}[\log(1-D_{src}(G(x)))],$

where the generator $G$ generates the fake character $G(x)$ conditional over the input character $x$, and the first part of discriminator $D_{src}$ attempts to distinguish the generated character is real or fake.

B. Cycle consistency loss. The second part of loss is the cycle consistency loss, which is introduced to let the generator reconstruct the character in the source font domain from the generated fake character, and thus avoid using the paired data. Specifically, such part of loss can be defined as follows:

$\displaystyle{\cal L}_{cyc}(G)=\mathbb{E}_{x}[\|x-G(G(x))\|_{1}],$

(2)

where $G(x)$ represents the generated fake character according to the input pair $x$ and $G(G(x))$ represents the reconstructed character from the character $G(x)$ generated by $G$.

C. Stroke-encoding reconstruction loss. Notice that in both adversarial loss and cycle consistency loss, the characters are regarded as images during the training, while the stroke information is not paid much attention. Actually, as discussed before, the stroke information embodies amount of mode information of a Chinese character, and thus is some kind of very important information that should be preserved during the training. Also, the stroke information is very special for Chinese characters and makes the generation of Chinese characters very different from the style translation of natural images. Specifically, the stroke-encoding reconstruction loss can be defined as follows:

$\displaystyle{\cal L}_{st}(D)=\mathbb{E}_{x,c}\left[{\|D_{st}(G(x))-c\|_{2}}\right].$

(3)

where $D_{st}(G(x))$ is the stroke encoding yielded by discriminator for the generated fake character $G(x)$. Such stroke-encoding reconstruction loss is used to guide the networks to reconstruct the stroke encodings as accurately as possible so that the modes of characters can be preserved much better.

D. Total training loss. Combining the above three parts of loss, i.e., (1)-(3), the total training loss of StrokeGAN is presented as follows:

	$\displaystyle{\cal L}_{strokegan}(D,G)$	$\displaystyle={\cal L}_{adv}(D,G)+\lambda_{cyc}{\cal L}_{cyc}(G)$		(4)
		$\displaystyle+\lambda_{st}{\cal L}_{st}(D),$

where $\lambda_{cyc}$ and $\lambda_{st}$ are two penalty parameters. Based on the above defined loss ${\cal L}_{strokegan}(D,G)$, the discriminator $D$ attempts to maximize it while the generator $G$ tries to minimize it, shown as follows:

$\displaystyle\min_{G}\max_{D}\ {\cal L}_{strokegan}(D,G).$

(5)

A. Collection of datasets. The dataset used in this paper consists of 9 sub-datasets with different fonts divided into three categories, i.e., a handwriting font, three standard printing fonts $\{$Black, Regular Script, Imitated Song$\}$, and five pseudo-handwriting fonts $\{$Huawen Amber, Shu, Hanyi Lingbo, Hanyi Doll, Hanyi Thin Round$\}$, where the pseudo-handwriting fonts are personalized fonts designed by artistic font designers.

The first kind of dataset related to the handwriting Chinese characters is built up from CASIA-HWDB1.1 ¹¹1http://www.nlpr.ia.ac.cn/databases/handwriting/Home.html, which was collected by 300 people. There are in total 3755 different commonly used Chinese characters written by everyone, and thus there are $3755\times 300$ handwriting characters in this dataset. To build up our handwriting dataset, for each character, we randomly selected one sample from these 300 samples. Therefore, the size of the handwriting font dataset used in this paper is 3755. Except the handwriting Chinese characters, the other font datasets were collected by ourselves from the internet ²²2say, http://fonts.mobanwang.com/ and made automatically via TTF. Specifically, the second kind of datasets contain three printing font datasets, of which the sizes are 2560, 3757 and 2506 respectively for the Black, Regular Script and Imitated Song fonts. The third kind of datasets consist of five pseudo-handwriting fonts, of which the sizes are 3596, 2595, 3673, 3213 and 2840 respectively for the Huawen Amber, Shu, Hanyi Lingbo, Hanyi Doll and Hanyi Thin Round fonts. The size of each character was resized to $128\times 128\times 3$. In our experiments, we used 90% and 10% of the samples respectively as the training and test sets.

B. Network architectures and optimizer. The network structure of the generator in StrokeGAN is the same as CycleGAN (Zhu et al. 2017), including 2 convolutional layers in the down-sampling module, 9 residual modules with 2 convolutional layers of residual networks for each residual module and 2 deconvolutional layers in the up-sampling module, as presented in Table 3 in Appendix. The network structure of the discriminator in StrokeGAN is similar to PatchGAN (Isola et al. 2017) with 6 hidden convolutional layers and 2 convolutional layers in the output module, as presented in Table 4 in Appendix. Moreover, the batch normalization (Ioffe and Szegedy 2015) was used in all layers.

In our experiments, we used the popular Adam algorithm (Kingma and Ba 2014) as the optimizer with the associated parameters $(0.5,0.999)$ in both the generator and discriminator optimization subproblems. The penalty parameters of the cycle consistency loss and stroke reconstruction loss were fine-tuned at 10 and 0.18, respectively.

C. Evaluation metrics. To evaluate the performance of StrokeGAN, we introduced three evaluation metrics. The first one is the commonly used content accuracy (Zhu et al. 2017), which was suggested to justify the quality of the contents of generated characters. Specifically, a pre-trained HCCG-GoogLeNet (Szegedy et al. 2014) was exploited to calculate the content accuracy. Besides the content accuracy, we also suggested the recognition accuracy and stroke error particularly for the Chinese character generation task. The recognition accuracy is defined as the ratio of those generated characters that can be correctly recognized by people to all the generated characters for testing, via a crowdsourcing way. Specifically, we randomly invited five Chinese adults to recognize the generated Chinese characters, and then took their average as the recognition accuracy. The stroke error is defined as the ratio of the number of missing and redundant strokes in the generated Chinese characters to its true total number of strokes. Thus, the smaller stroke error means the strokes are preserved better.

Our experiments consist of three parts, where the first two parts were conducted to respectively show the effectiveness and sufficiency of the introduced one-bit stroke encoding, and the third part was conducted to demonstrate the effectiveness of StrokeGAN via comparing with the state-of-the-art methods including CycleGAN (Chang et al. 2018), zi2zi (Tian 2017) and Chinese typography transfer (CTT) method (Chang, Gu, and Zhang 2017), where the latter two methods are based on paired training data.

A. Effectiveness of one-bit stroke encoding. In these experiments, we verified the feasibility and effectiveness of our idea that incorporating the stroke information into CycleGAN can preserve the modes of Chinese characters better and thus alleviate the issue of mode collapse. In order to verify this, we implemented nine one-to-one Chinese character style translation experiments as listed in Table 1. For each experiment, we selected one font domain (say Regular Script) from these nine different font styles as the source font domain and another font domain (say Shu) as the target font domain, and then trained StrokeGAN as well as CycleGAN as the baseline. The performance of the suggested StrokeGAN is presented in Table 1. Some examples of the generated Chinese characters by StrokeGAN are shown in the previous Figure 3 and Figure 5, which demonstrate that StrokeGAN can generate very realistic Chinese characters for all these nine fonts.

From Table 1, StrokeGAN outperforms CycleGAN (without using the stroke encoding) in most of the tasks except the style translation task from Imitated Song to Regular Script. In terms of the stroke error presented in the last two columns, StrokeGAN consistently improves the performance of CycleGAN (Chang et al. 2018). This shows the feasibility and effectiveness of the suggested StrokeGAN.

Moreover, as demonstrated in the previous Figure 1, CycleGAN sometimes suffers from the issue of mode collapse (particularly, when applied to the generation task from Black font to Shu font), while the suggested StrokeGAN can significantly alleviate this issue. Besides the issue of mode collapse, as shown in Figure 6, CycleGAN also sometimes suffers from another issue, that is, missing some key strokes of Chinese characters generated, which may result in the recognition issues of these characters, while the suggested StrokeGAN can preserve these strokes much better.

B. Sufficiency of one-bit stroke encoding. Notice that there are many very similar Chinese characters having the same stroke encodings. In order to show the sufficiency of the introduced one-bit encoding, we tested the performance of StrokeGAN on some very similar Chinese characters, as shown in Figure 7, which shows that the generated characters can be well distinguished with well preserved strokes. This demonstrates that such one-bit encoding way is in general enough to preserve the key modes of Chinese characters.

C. Comparison with the state-of-the-art methods. In this part of experiments, besides CycleGAN (Chang et al. 2018), we compared StrokeGAN with two paired data based methods, i.e., zi2zi (Tian 2017) and CTT method (Chang, Gu, and Zhang 2017). In order to implement both zi2zi and CTT methods, we manually built up two paired datasets based on the Regular Script font and Shu font datasets. Since the collection of paired training data is very costly, in these experiments, we only considered the generation task from the Regular Script font to Shu font for all these four methods, while for the other generation tasks, it can be implemented similarly. The performance of these four methods is presented in Table 2, and some examples of generated characters are shown in Figure 8. From Table 2, the suggested StrokeGAN outperforms all these existing methods in terms of the suggested three evaluation metrics, and also generates the Chinese characters with the highest quality among all these methods. These experiment results demonstrate the effectiveness of the proposed StrokeGAN.