ZiGAN: Fine-grained Chinese Calligraphy Font Generation
via a Few-shot Style Transfer Approach

Generative Adversarial Networks (GAN) (Goodfellow et al., 2014) has attracted a lot of interest since it was proposed. It has been successfully applied in many different fields and achieved impressive results, such as image generation (Martin Arjovsky and Bottou, 2017; Karras et al., 2017, 2019), image completion (Iizuka et al., 2017; Yu et al., 2018, 2019), image editing (Zhu et al., 2016), transfer learning (S. et al., 2018, 2019), image translation (Zhu et al., 2017; Isola et al., 2017; Kim et al., 2019; Choi et al., 2018), etc. The key to the success of GAN is that the discriminator tries to distinguish the generated images from the realistic images, while the generator tries to confuse the judgment of the discriminator. In this paper, our model is based on GAN and only uses a few reference data to learn the Chinese calligraphy character style translation.

Image-to-image translation aims to learn a mapping function that can transform an image from the source domain to the target domain. It has been widely used in many applications, for example, for artistic style transfer (Johnson et al., 2016; Chen and Schmidt, 2016), semantic segmentation (Long et al., 2015; Noh et al., 2015; S. et al., 2020), photo enhancement or object replacement.

A great quantity of GAN-based methods have been proposed, quite a few of them condition on images (Isola et al., 2017; Choi et al., 2018, 2020; Wang et al., 2018). Pix2pix (Isola et al., 2017) is the pioneering method to figure out image-to-image translation. It follows the idea of conditional GAN, applying adversarial loss and L1-loss, and achieves impressive results. After that, high-resolution version is proposed to reinforce pix2pix in image synthesis and semantic manipulation (Wang et al., 2018). But those paired training data are hard to obtain for some applications such as artistic style transfer. To alleviate this pain point, unpaired image-to-image translation frameworks have been proposed where no paired data are available anymore (Zhu et al., 2017; Liu et al., 2017; Liu and Tuzel, 2016). It is a remarkable fact that CycleGAN (Zhu et al., 2017) proposes the cycle-consistent adversarial network, where two GANs interact in a cycle and learn source and target image distributions simultaneously. Based on CycleGAN, U-GAT-IT (Kim et al., 2019) proposes a novel method for unsupervised image-to-image translation, which incorporates a new attention module and a new learnable normalization function called AdaLIN in an end-to-end manner. It is effective in the task of animating faces. In summary, the aforementioned paired methods all require a lot of data for training, otherwise the results will be unsatisfactory. Meanwhile, the unpaired methods often cause missing or redundant construction. But in the task of Chinese calligraphy character style translation, calligraphers’ handwriting is often difficult to obtain, and we cannot tolerate the inconsistency of character structure.

Chinese font generation has been studied for a long time (Wong and Ip, 2000; Wu et al., 2006). The image-based methods  (Xu et al., 2005; Du et al., 2016) split and reorganize the corresponding strokes and radicals in the dataset to generate the characters we want. But these methods contain too much human intervention, which is very inconvenient. With the development of deep learning, people have paid more attention to GAN-based character translation. Since character translation requires higher accuracy according to its complex strokes and style, it is more difficult than classic image-to-image translation problems. Transferring the styles of the alphabet is quite helpful and efficient for English translation (Azadi et al., 2018). While it is not simple like this for Chinese character style transfer because each Chinese character has its own glyph shape and there is a large number of existed characters. The style of the strokes in a certain character may quite different from the same strokes in other characters (Wen et al., 2019), which makes the problem harder.

The first way to generate Chinese characters is following image-to-image methods, like zi2zi (Tian, 2017), an open-source project that was never published as a paper. It’s based on pix2pix, trying to translate character images from source style to various target styles. Based on zi2zi, DCFont (Jiang et al., 2017) and PEGAN (Sun et al., 2018) have made improvements and achieved better results. The second way to synthesize Chinese characters often separates a character into two parts, which are content and style (Gao and Wu, 2020; Wen et al., 2019). EMD (Zhang et al., 2018) and SA-VAE (Sun et al., 2017) use two different encoders to process content and style respectively. After absorbing the advantages of the above methods, CalliGAN (Wu et al., 2020) adds an extra component code of the character to train a conditional GAN, exploiting prior knowledge to maintain the structure information. While it needs a dictionary for each Chinese character to save its component code, this is a complicated preprocessing work. Unlike the aforementioned methods, ChiroGAN (Gao and Wu, 2020) uses erosion and dilation operations to obtain the basic skeleton of characters, then transfers style from source to target at the skeleton level. The output of this module is the skeleton image so it has to use another network to render the skeleton into the target character. Moreover, it relies on the effects of corrosion and expansion algorithms so that it often crashes on complex characters with numerous strokes or irregular glyph styles.

In order to save the cost of multiple Chinese characters selection, several recent methods aim to generate new glyphs with few numbers style references. DMfont (Cha et al., 2020) disassembles Korean or Thai glyphs to stylize components and then reassembles them. But it cannot handle complex Chinese characters. RD-GAN (Huang et al., 2020) aims to generate unseen characters in the fixed style, but it still requires a lot of prior knowledge of radicals, which will be very troublesome to process. Other earlier few-shot methods also have fatal shortcomings, such as being unable to generate complex glyphs (Azadi et al., 2018) or failing to capture local styles (Sun et al., 2017; Gao et al., 2019).

To sum up, part of the methods require lots of data, but the handwritings of many ancient Chinese calligraphers are not handed down so we cannot obtain them. The other parts of the methods are doped with too much manual processing. Moreover, they utilize too much intricate prior knowledge, which makes the preprocessing work complicated and can only adapt to a single task. To overcome these challenges, in this paper we propose a novel ZiGAN that can learn an intact and delicate character style and structure when only a few target characters are provided. ZiGAN is an end-to-end framework, which can be easily and conveniently applied to any character style translation task, and is capable of generating a complete and consistent font library.

Figure 2 shows the architecture of our network. We encode $x_{p}$ and $x_{r}$ into the feature space through an image encoder $E_{s}$, and then decode image features by an image decoder $G_{s}$ to generate the stylized character images $\hat{y}_{p}$ and $\hat{y}_{r}$. After that, we model on CycleGAN (Zhu et al., 2017) and set up a reversing generator, including encoder $E_{t}$ and decoder $G_{t}$.

We define four losses in total. The loss items of $x\rightarrow{y}$ can be written as:

GAN loss. GAN loss is divided into main and auxiliary parts. In the main part, we impose adversarial loss to match the distribution of the translated images and target images. We use the Least Squares GAN (Mao et al., 2017) objective to train our model.

In addition, we add an auxiliary classifier $\eta_{D_{t}}$ based on Class Activation Map(CAM) (Zhou et al., 2016) to the discriminator $D_{t}$. Let $y{\in\left\{Y\},G_{s}(E_{s}\left(X\right)\right)}$ represent a sample from the target domain and the translated source domain. The discriminator $D_{t}$ consists of an encoder $E_{D_{t}}$, a classifier $C_{D_{t}}$, and an auxiliary classifier $\eta_{D_{t}}$. The auxiliary classifier is trained to learn the weight of the $k$-th feature map for the target domain, $w_{t}^{k}$, by using the global average pooling and global max pooling, i.e., ${\eta_{D_{t}}(y)=\sigma\left(\Sigma_{k}w_{t}^{k}\Sigma_{ij}E_{D_{t}}^{k_{ij}}(y)\right)}$. By exploiting $w_{t}^{k}$, we can calculate a set of domain specific attention feature map $a_{D_{t}}(y)=w_{D_{t}}*E_{D_{t}}(y)=\left\{w_{D_{t}}^{k}*E_{D_{t}}^{k}(y)\mid 1\leq k\leq n\right\}$, where $n$ is the number of encoded feature maps. Then, our discriminator $D_{t}(y)$ becomes equal to $C_{D_{t}}(a_{D_{t}}(y))$. By doing so, the discriminator can better distinguish the differences in the details of different character styles, while $E_{s}$ and $G_{s}$ can make improvements in the most important regions.

On the whole:

Consistency loss. We constrain the consistency of the model from two parts. First, the model must have the ability to cycle back. It means that after $x$ is translated to $\hat{y}$, it must be successfully translated back to the original domain:

Second, identity loss is used to constrain the color and shape of the characters to not be distorted. Given an image $y$, after the translation of $Es$ and $Gs$, it should be the same character in the same style.

So the total consistency loss is:

Alignment loss We align the content and feature levels of $x_{p}$ and $y$ to leverage the paired samples to attain fine-grained structural correspondence. In the font style translation task, the job of the discriminator is still to distinguish which is generated or which is real, but the generator is tasked to not only fool the discriminator, but also to be as similar to the ground truth at the content level as possible. We use the L1 loss to constrain the output of paired data $x_{p}$,

And in order to constrain the features of the generated image and the real image to the same space, we apply constancy loss:

Therefore, the total alignment loss can be formulated as:

where $\alpha=5$.

Style loss For better understanding of coarse-grained character content and a maturer style translation, we have introduced style loss to take advantage of multiple unpaired samples $x_{r}$. Unlike paired data, unpaired data cannot simply be restricted by L1 or L2 losses. Therefore, with comprehensive consideration of time complexity and computational cost, we utilize MK-MMD (Gretton et al., 2007; S. et al., 2021) to match the feature distributions to retain style information. Denote by $\mathcal{H}_{k}$ be the reproducing kernel Hilbert space (RKHS) endowed with a characteristic kernel $k$. The mean embedding of distribution $p$ in $\mathcal{H}_{k}$ is a unique element $\mu_{k}(p)$ such that $\mathbf{E}_{\mathbf{x}\sim p}f(\mathbf{x})=\left\langle f(\mathbf{x}),\mu_{k}(p)\right\rangle_{\mathcal{H}_{k}}$ for all $f$ $\in\mathcal{H}_{k}$. The MK-MMD $d_{k}(x,y)$ between probability distributions $x$ and $y$ is defined as the RKHS distance between the mean embeddings of $x$ and $y$. The squared formulation of style loss is defined as:

where $\phi$ is the corresponding feature map. And it’s worth noting that when $x=y$, $\mathcal{L}_{style}$=0. Here we choose Gaussian kernel function as the kernel function:

Full objective Finally, the full objective function is defined as:

where $\lambda_{1}=5$, $\lambda_{2}=10$, $\lambda_{3}=10$, $\lambda_{4}=10$. Here $\mathcal{L}_{GAN}$ = $\mathcal{L}_{GAN}^{x\rightarrow y}$ + $\mathcal{L}_{GAN}^{y\rightarrow x}$ and the other losses are defined in the similar way.

To better show our model’s performance, we use the same datasets with CalliGAN (Wu et al., 2020). The datasets could be downloaded from a Chinese calligraphy character website¹¹1http://163.20.160.14/~word/modules/myalbum/, where there are more than 20 kinds of brush-written calligraphy sets belonging to different Chinese ancient experts. And 7 styles belonging to regular script are used to complete our experiment. The 3rd and the 4th style sets are the same ancient calligraphic expert’s masterpieces created in different periods of his life. They are treated as two different style sets due to the differences between them, which is also the rule of thumb in the Chinese calligraphy community. The 5th and the 6th style sets are the same situation as above. In addition to the above-mentioned dataset which is the same as CalliGAN, we also test our model in other more irregular and challenging Chinese character fonts like XING and CAO to prove that our method is highly adaptable and robust to any font style. So our data set consists of 9 fonts in total which are shown in Table 2. Figure 3 shows example characters in 9 different styles.

We collect 61151 target images that cover 6560 characters in the 9 styles in all. And we use TTF of font style Song to render source image $x$. To explore the ability of our few-shot method, we create two configurations for the dataset: 100-shot and 200-shot. Each style has 100 or 200 randomly selected training examples respectively as input $y$, while the remaining images as the test set. Such a training set size is much smaller than other methods which often require thousands of training images. Specific information is listed in Table 3. Given ${y}$, we can easily get the same character image ${x_{p}}$ in the source domain. In the meantime, we randomly sample and render 6000 unpaired images with font style Song which cover a large number of characters as input $x_{r}$.

The images in this repository have various shapes depending on the character’s shapes. We follow the preprocessing steps of CalliGAN (Wu et al., 2020), but the only difference is that we process the images into three-channel RGB images. So we get $256\times 256\times 3$ images as our ground truth $y$. All images are converted to tensors linearly with a value range between -1 and 1 by our network.

We use standard methods to optimize our network: first optimize on D, then on E and G together. Similarly, we also alternate training on $x_{r}$ and $x_{p}$. All models are trained using Adam with $\beta_{1}=0.5$, $\beta_{2}=0.999$. The learning rate is initialized to $0.0003$ and drops by half every $500$ epochs. Because the number of training samples is small, we train our model in $1500$ epochs.

To prove the advancement of our method in the field of few-shot font style transfer, we have extensively compared various methods. Six classic methods are used as the baselines, including zi2zi (Tian, 2017), pix2pix (Isola et al., 2017), U-GAT-IT (Kim et al., 2019), CycleGAN (Zhu et al., 2017), StarGAN (Choi et al., 2018), CalliGAN (Wu et al., 2020). Among them, zi2zi, pix2pix and CalliGAN need paired data for training. We use the same number of paired images as ours to train their model in corresponding configurations. CycleGAN, U-GAT-IT and StarGAN are unsupervised methods. We use 6200 images in font style Song as their source domain and 100 or 200 calligraphic images as their target domain for different configurations so that the size of their training set is not smaller than ours. Figure 4 shows the comparison of generation results.

CycleGAN not only did not fully learn the style of the characters but also lost some strokes. StarGAN has lost the structural information of the character and is completely unable to do this job. Pix2pix barely maintains the structure of the characters, but there are too many fuzzy and damaged places. U-GAT-IT seems to have learned the style of the font, but there are still many erroneous and missing strokes in the result. Although zi2zi and CalliGAN are professional in font style translation, they produce unsatisfactory results which contain too many meaningless blanks and blurs when few targets are referenced. Only ZiGAN has found the law of calligraphy from limited target characters.

For quantitative evaluation, we evaluate it from two aspects: style and content. For the former, we use Intersection Over Union(IOU) to measure whether our results have completed the style translation. IOU calculates the ratio of the intersection and union between the generated character images and the real character images. The higher the value indicates that the distribution of the generated images is closer to the distribution of the real images, and the result is better. As above, we compare six classic methods, and Table 4 shows that ZiGAN achieves the highest IOU scores.

Similarly, for content evaluation, we train a Resnet18 model as a Chinese character recognizer on the ground truth of all styles. And test it on the images generated by our test set. As we can see from Table 5, the top-1 accuracy achieved by our method is significantly ahead of other methods, which proves that our method can effectively retain the structure and content information of the character.

We implement user study to verify that our results are not only better in the calculated indicators. 40 people who are familiar with Chinese characters participate in the experiment. We randomly select 65 characters in the test set, then use the compared methods and the proposed method to generate images. Therefore, the participants see a total of 520 images, including 390 images generated by the compared methods, 65 images generated by our ZiGAN, and 65 of ground truth. At each selection, participants will see 7 images generated by 7 different methods and ground truth. Overall, the participant’s goal is to find the image that is most similar to ground truth. In detail, participants are asked to prioritize finding the images that are semantically consistent with the ground truth, which means that the generated characters cannot have wrong radicals or missing strokes. On this basis, consider which image style is closer to the ground truth and has better details. Table 6 shows the respondents’ vote rates for each method.

Meanwhile, we build the Turing test set and make a Turing test. As shown in Table 7, each sample contains 6 fake glyph images generated by ZiGAN and 6 real glyph images. We ask 50 professional Chinese users to identify which images are generated in 30 sets of samples. ZiGAN achieves an accuracy of 51.6%, which is very close to random selection.