SGCE-Font: Skeleton Guided Channel Expansion for Chinese Font Generation

As discussed before, using what kind of information as the guidance information and how to use it are very important for the Chinese font generation. Motivated by the superiority of the skeleton on representing the global and local structure information of Chinese characters and the advantage of channel expansion on directly using the information, we propose a skeleton guided channel expansion module as certain a information guidance module for the Chinese font generation, as depicted in Figure 4. From Figure 4, given a character image $x$ as the input, we firstly extract its skeleton $sx$ with single channel via a skeletonization strategy denoted as Ske for short and described later, and then combine the single-channel skeleton information with the original (R, G, B) three-channel character image information to yield the four-channel image information $cex$ as the input of the generator through the channel expansion way.

In the next, we describe the skeletonization strategy used in this paper to extract the skeleton of a Chinese character, inspired by [38]. Given an image of a Chinese character, we firstly perform a binarized operation on it, and then yield the skeleton from the binarized image. Specifically, for a given pixel $P_{1}$ of the binarized image, we define a $3\times 3$ image patch with $P_{1}$ as the center, and number the other eight pixels according to Table I. For those pixels lying on the edge of the binarized image, we obtain the associated $3\times 3$ image patches using the zero-padding way. For the defined $3\times 3$ image patch as shown in Table I, let $p_{i}$ be the value of $P_{i}$ ($i=1,\ldots,9$), ${\cal N}(P_{1})$ be the number of non-zero points in this $3\times 3$ image patch, and ${\cal P}(P_{1})$ be the number of $(0,1)$ pairs among $P_{2}$ to $P_{9}$ by sequence, then we delete $P_{1}$ if the following hold:

1. (a)

   $2\leq{\cal N}(P_{1})\leq 6,\ {\cal P}(P_{1})=1$; and either

2. (b)

   $p_{2}\cdot p_{4}\cdot p_{6}=0,\ p_{4}\cdot p_{6}\cdot p_{8}=0$; or

3. (c)

   $p_{2}\cdot p_{4}\cdot p_{8}=0,\ p_{2}\cdot p_{6}\cdot p_{8}=0$.

We eventually yield the skeleton of a Chinese character through doing the above operation over all pixels.

According to the above description, the proposed SGCE module does not require any additional neural network to implement it. This is very different to existing works [26, 36, 37, 17, 11] struggling to increase the model complexity to exploit the component information (say skeleton) for the Chinese font generation.

In the following, we describe the proposed model for Chinese font generation through integrating the suggested SGCE module (called SGCE-Font), where we take the well-known CycleGAN [23] as the base model as an example. The specific model architecture is presented in Figure 5. As depicted in Figure 5, the proposed SGCE-Font is very similar to that of the existing model (say, CycleGAN), except that the input of the generator is replaced by the four-channel skeleton-image information yielded by the suggested SGCE module, instead of the original (R, G, B) three-channel image information. The specific workflow of the proposed model can be described as follows. We first input the Chinese character image $x$ with (R, G, B) three channels in the source font domain into SGCE module to yield $cex=SGCE(x)$ enriched by the skeleton information into the generator $G_{y}$ to realize the style translation from the source font style to the target font style. After $G_{y}$, we yield the generated character $\tilde{y}=G_{y}(cex)$ in the target font domain. On one hand, we simultaneously feed the generated characters and the real Chinese characters in the target font domain into the discriminator $D_{y}$ to distinguish whether they are real or fake, and on the other hand, we feed the generated character $\tilde{y}=G_{y}(cex)$ into the SGCE module and then input the enriched information into another generator $G_{x}$ (realizing the font generation task from the target font domain to the source font domain), and finally yield a reconstructed character $\tilde{x}=G_{x}(SGCE(\tilde{y}))$ in the source font domain. According to the above workflow, the training loss of the proposed model consists of four parts including two adversarial losses ${\cal L}_{adv_{x}}$ and ${\cal L}_{adv_{y}}$ related to $(D_{x},G_{x})$ and $(D_{y},G_{y})$ respectively, the cycle consistency loss ${\cal L}_{cyc}$ between the original Chinese character $x$ and its reconstructed character $\tilde{x}$ in the source font domain, the skeleton consistency loss ${\cal L}_{ske}$ between the skeleton $sx$ of the input $x$ and the skeleton $\widetilde{sx}=Ske(\tilde{x})$ of the reconstructed Chinese character $\tilde{x}$ in the source font domain, where the skeleton consistency loss is imposed to further supervise the generation procedure. Specifically, these losses are defined as follows:

where ${\cal X}$ and ${\cal Y}$ respectively represent the sets of real Chinese characters in the source and target domains, $\tilde{\cal X}$ represents the set of reconstructed Chinese characters in the source font domain by $G_{x}$, $\tilde{\cal Y}$ represents the set of generated characters in the target font domain by $G_{y}$. Thus, the training model of SGCE-Font is shown as follows:

where

and $\lambda_{cyc}$, $\lambda_{ske}$ are two tunable hyperparameters.

A. Datasets. We considered ten datasets with different fonts including three standard printing fonts {Heiti (ht), Fangsong (fs), Zhengkai (zk)}, the handwriting (hw) font, three pseudo-handwriting fonts {Hupo (hp), Shuti (st), Lishu (ls)}, and three calligraphy fonts {Huangcao (hc), Zhuxi (zx), SuShi(ss)}. The specific sizes of datasets were presented in Table II and some samples were shown in the left part of Figure 3. The handwriting dataset was randomly collected from the CASIA-HWDB1.1 dataset, the other font datasets were collected by ourselves from the internet. The sizes of the last two calligraphy font datasets, i.e., Zhuxi and SuShi are very small. We use them mainly to show the performance of the proposed model over the few-shot font generation task. For each sample, we reset the image size of each character to 128×128×3. In all experiments, we used 80% and 20% samples as the training and test sets respectively.

B. Baselines. In this paper, we considered the following six state-of-the-art models as baselines.

• •

  CycleGAN [23]: A typical model based on GAN and unpaired data.

• •

  SQ-GAN [24]: An improved CycleGAN model equipped with a square-block geometric transformation based self-supervised learning scheme.

• •

  StrokeGAN [25]: A refined CycleGAN model incorporated with the one-bit stroke encoding for the Chinese font generation to mitigate the mode collapse.

• •

  UGATIT [20]: An effective GAN model using unsupervised generative attention network with adaptive layer instance normalization and unpaired data.

• •

  FUNIT [39]: An effective GAN model based on the disentangled representation learning and unpaired data.

• •

  AttentionGAN [40]: An effective GAN model based on the attention mechanism and unpaired data.

Besides the above baselines, we also suggested another model called SkeGAN as a baseline for better comparison with the proposed model. In the SkeGAN, we impose the skeleton information to the discriminator in the similar way of StrokeGAN [25].

C. Network structures. The network structure of the generator of SGCE-Font is almost the same as that of CycleGAN [29], including 2 convolutional layers in the down-sampling module, 9 residual modules with 2 convolutional layers of residual networks for each residual module, and 2 deconvolutional layers in the up-sampling module, except that the input of the generator was modified from 3 channels to 4 channels. The network structure of the discriminator of SGCE-Font is the same as CycleGAN [29], with 6 hidden convolutional layers and 2 convolutional layers in the output module. Moreover, batch normalization [41] was used in all layers.

We used the popular Adam algorithm [42] as the optimizer with the associated parameters (0.5, 0.999) in both the generator and discriminator optimization subproblems. The hyperparameters of the cycle consistency loss and skeleton consistency loss were empirically set as 1 and 0.001, respectively.

D. Evaluation metrics. We used four important evaluation metrics to measure the performance. The first two evaluation metrics were the commonly used SSIM (Structural Similarity) [43] and PSNR (Peak Signal to Noise Ratio), which were employed to measure how the pixel-level details are preserved. The third evaluation metric is the well-known MSE (Mean Square Error). The last evaluation metric is FID (Frechet Inception Distance) [44], which was used to measure how the generated results of the model match the distribution of real data. Larger SSIM and PSNR values, and smaller MSE and FID values generally imply better generation performance.

In this section, we implemented extensive experiments over fourteen generation tasks to demonstrate the superiority of the proposed SGCE module for the Chinese font generation. The quantitative comparisons between the proposed SGCE-Font and these baselines including CycleGAN [23], SQ-GAN [24], StrokeGAN [25] and SkeGAN are presented in Table III, and some visual comparisons on the generated characters are presented in Figure LABEL:fig:modecollapse and Figure 6.

From Table III, the proposed SGCE-Font model outperforms these concerned baselines in terms of four evaluation metrics in average. Specifically, when regarding the FID, the proposed SGCE-Font achieves the best results in nine font generation tasks and the second best results in three font generation tasks, and is significantly better than the baselines. Similar claims can be concluded in terms of the other three evaluation metrics.

When comparing the performance among different models, the refined CycleGAN models using different guidance information such as the square-block transformation in SQ-GAN [24], the stroke encoding in StrokeGAN [25] and the skeleton in SkeGAN outperform the original CycleGAN model for the Chinese font generation. Considering the performance of these models using different guidance information, the performance of SkeGAN using the skeleton information is significantly better than that of SQ-GAN and StrokeGAN in average, in terms of these four evaluation metrics. This show clearly that the skeleton information is much better than the square-block transformation and stroke encoding as the guidance information for the Chinese font generation. Moreover, when comparing the performance of SkeGAN and the proposed SGCE-Font, it can be observed that the proposed SGCE-Font outperforms SkeGAN in most of cases. This shows that the suggested way of using the skeleton information by the channel expansion as the direct input of the generator is much better than the way of imposing the skeleton information onto the discriminator as used in SkeGAN. These claims can be also verified by the visualization results as depicted in Figure 6. From Figure 6, the quality of generated characters of the proposed SGCE-Font is better than SkeGAN, while the performance of SkeGAN is much better than that of StrokeGAN, SQ-GAN as well as CycleGAN.

In particular, as shown in Figure LABEL:fig:modecollapse, the well-known CycleGAN suffers from the mode collapse issue when applied to these three font generation tasks, i.e., {Hupo $\rightarrow$ Zhengkai, Fangsong $\rightarrow$ Heiti, Fangsong $\rightarrow$ Lishu}. Such mode collapse phenomena of CycleGAN can be also observed from Table III, where the FID values of CycleGAN in these three font generation tasks are 218.18, 304.34 and 282.81, which are abnormally large. Although existing square-block transformation and stroke encoding based guidance schemes can reduce the mode collapse to some extent as shown by the third and fourth rows of Figure LABEL:fig:modecollapse, the SQ-GAN [24] still suffers from the mode collapse in the font generation task {Hupo $\rightarrow$ Zhengkai} and there is stroke-missing or stroke-redundancy phenomenon in the generated characters of StrokeGAN, while from the fifth rows of Figure LABEL:fig:modecollapse, the mode collapse issue of CycelGAN can be significantly alleviated by the suggested SGCE module, and the quality of the generated characters of the proposed SGCE-Font is much better than that of the other three baselines. These show clearly the superiority of the suggested SGCE module on reducing mode collapse.

In this section, we conducted twelve printing or handwriting font generation tasks to demonstrate the effectiveness of the proposed model through comparing with the state-of-the-art models including FUNIT [39], UGATIT [45], and AttentionGAN [40]. The comparison results between the proposed model and these existing models for the twelve standard character generation are presented in Table IV.

From Table IV, the proposed SGCE-Font achieves the best performance in seven of these twelve font generation tasks and performs the second best in four of these twelve font generation tasks in terms of FID, as presented in the fifth row of Table IV. The average FID value of the proposed model over these fourteen standard Characters generation tasks is much better than those of state-of-the-art models, as shown in the last column of Table IV. Similar claims can be concluded in terms of the other three evaluation metrics from Table IV.

Some generated characters of the proposed model as well as these three state-of-the-art models for some Chinese characters in the test set are shown in Figure 7. It can be observed from this figure that SGCE-Font can produce Chinese characters with higher quality than the concerned existing models. Specifically, FUNIT [39] does not perform very well when applied to the generation tasks {Fangsong$\rightarrow$Zhengkai, Zhengkai$\rightarrow$Hupo}, that is, the styles of generated fonts seem closer to the styles of source fonts than that of target fonts, as shown in the second row of Figure 7. Besides these, there are some flaws (say, missing some strokes) in the generated characters of existing models for some font generation tasks such as {Zhengkai$\rightarrow$Handwriting} and {Zhengkai$\rightarrow$Shuti}, while these flaws can be remarkably reduced by the suggested SGCE-Font model. These demonstrate the effectiveness of the proposed model.

We also present some generated characters of the proposed model and these state-of-the-art models for some unseen Chinese characters, i.e., excluded by the concerned datasets, in Figure 8. It can be observed that the proposed SGCE-Font model can generate more realistic and better Chinese characters than existing models for these unseen Chinese characters. This shows that the proposed model has the better generalization performance.

Besides the previous twelve printing or handwriting font generation tasks, we particularly considered four calligraphy font generation tasks to show the effectiveness of the proposed model. As presented in II, the total sizes of the Zhuxi (zx) and SuShi (ss) are only 920 and 166, which are much fewer than those of printing or handwriting font datasets. The quantitative comparison results over these four font generation tasks are presented in Table V.

From Table V, the proposed SGCE-Font achieves the best performance in average in terms of FID, MSE and PSNR, while acheives the second best performance in average in terms of SSIM. Specifically, in terms of FID, the proposed SGCE-Font model achieves the best results in the font generation tasks {ls$\rightarrow$hc, zk$\rightarrow$ss, zk$\rightarrow$zx}, while achieves the second best result in the font generation tasks {hc$\rightarrow$ls}. Similar claims can be also claimed in terms of the other three evaluation metrics. We also present some visualization comparisons between the proposed model and five state-of-the-art models over two calligraphy font generation tasks. The comparison results for the characters in the test set and the unseen characters excluded by the datasets are presented in Figure 9. It can be observed from Figure 9 that the quality of generated characters of the proposed model is better than the baselines. These show clearly the effectiveness of the proposed model in these calligraphy font generation tasks.

As pointed out before, the proposed SGCE module can be used as a plug-and-play module for the Chinese font generation models. In the following, we adapted the proposed SGCE module to five existing models including UGATIT [45], AttentionGAN [40], StrokeGAN [25], SQ-GAN [24] and FUNIT [39] to further enhance their performance. We incorporated SGCE into each base model by a similar way in Figure 5. For UGATIT, AttentionGAN, SQ-GAN and FUNIT, we implemented eleven font generation tasks including eight printing or handwriting font generation tasks and three calligraphy font generation tasks to evaluate the effectiveness of the proposed SCGE module, while for StrokeGAN, we did not tested its performance in these three calligraphy font generation tasks since it was expensive to build up the stroke encodings of the traditional Chinese characters involved in the calligraphy font generation tasks.

The quantitative comparison results are presented in Table VI and Table VII. From Table VI and Table VII, the performance of these five concerned models can be substantially improved over most of font generation tasks, through equipping with the proposed SGCE module in terms of these four evaluation metrics. We also present some visualization comparison results in Figure 10. It can be observed from Figure 10 that the quality of generated characters can be improved by equipping with the proposed SGCE module. These show clearly the effectiveness of the proposed SGCE module.