Few-Shot Font Generation by Learning Fine-Grained Local Styles

Image-to-image (I2I) translation refers to the task of learning a mapping function between the source domain and target domain, which preserves the content of the source image while merging the style of the target domain at the same time. According to many I2I methods [14, 34, 6, 7, 17, 32, 18], I2I methods have developed towards multi-mapping [7, 32, 17] and few-shot learning [18]. CycleGAN [34] introduces the cycle-consistency into generative models, which enables I2I methods to train cross-domain translation without paired data. FUNIT [18] accomplish the style transfer task by encoding content and style respectively and combine them with adaptive instance normalization (AdaIN) [12]. From an intuitive thought, font generation is a typical I2I translation task, since it tries to keep the content information of the source font and maps it into the target font. Thus, many font generation methods are based on I2I translation methods.

Early font generation methods [26, 25, 19, 24, 5, 15, 16, 9, 30, 28, 11] aim at learning a mapping function between source fonts and target fonts. When new font references are introduced, these methods use hundreds of reference glyphs to fine-tune the original mapping function. zi2zi [26] and Rewrite [25] train GANs in a supervised way with one-hot style labels. AGEN[19] proposes an model based on the auto encoder to transfer standard font images to calligraphy images. HAN [5] designs skip connection and hierarchical loss functions to improve zi2zi’s generation performance. These methods require paired data to train a mapping function, e.g,775 for [16]. Other methods [9, 11] focus on unpaired data, and use them for style extraction. Although these many-shot font generation methods have achieved remarkable performance, it is still a laborious task to collect hundreds of references for the fine-tuning process, especially when the reference font library is glyph-rich.

Most Few-shot font generation (FFG) methods focus on disentangling the content feature and style feature from the given glyphs. Based on different kinds of feature representation, FFG methods can be divided into two main categories: global feature representation [1, 33, 8, 29] and component-based feature representation [13, 4, 22, 23]. In methods that apply global feature representation, vectors related to content and style are extracted from content glyphs and reference glyphs respectively. MCGAN [1] synthesize the ornamented glyphs with stacked conditional GANs to extract features from input images. EMD[33] and AGIS-Net[8] combine a style vector and content vector together to synthesize a glyph. ZiGAN[29] matches the features to Hilbert space to better capture the structural information. Works related to component-based feature representation concentrate on devising a feature representation that is related to the glyphs’€™ components or localized features. RD-GAN [13] uses a radical encoder to extract features of glyphs’ specific components. In DM-Font [4], it disassembles glyphs to stylized components and reassembles them to new glyphs by utilizing strong compositionality prior. LF-Font [22] designs a component-conditioned reference encoder to extract component-wise features from reference images. MX-Font [23] employs multiple encoders for each reference image with disentanglement between content and style which makes the cross-lingual task possible. DG-Font [31] is an unsupervised framework based on TUNIT [2] by replacing the traditional convolutional blocks with Deformable blocks which enables the model to perform better on cursive characters which are more difficult to generate.

However, the previous FFG works fail to fully explore the style maps from $k$-shot references. When receiving $k$-shot reference images, they tend to explicitly disentangle $style$ and $content$ of images globally or component-wisely and conduct a average operation among extracted features [22, 23]. The local details extracted from each reference are significantly weakened by disentanglement and the average operation. Therefore, we design Style Aggregation Module that aims to keep the very detailed features from reference images and to fully utilize the spatial details.

The attention mechanism [27] is known to capture dependence. After its debut in machine translation, it has been applied in many vision tasks including font generation. RD-GAN[13] using attention mechanism to extract rough radicals from content characters and then render them into target style. HWT [3] uses transformer blocks to bridge the gap between image and text, making it capable of generating stylized handwriting English text images. Our Style Aggregation Module is highly motivated by the attention mechanism for fine-grained feature map re-composition.

In few-shot font generation, given a content glyph image $x_{c}$ with the content $c$ from a standard font $X=\{x_{c}\}^{N}_{c=1}$ and a $k$-shot reference glyph images with the style $s$: $R_{s}=\left\{y_{i}\right\}_{i=1}^{k}\subset Y_{s}$, our goal is to generate a stylized image $\hat{y}_{c}$ that has the content $c$ and style $s$ via a generator $G$:

where the style $s$ of $\hat{y}_{c}$ is omitted for sake of simplicity.

In training, we collect $L$ fonts with different styles $Y=\{Y_{s}\}^{L}_{s=1}$, where $Y_{s}=\{y_{c}\}^{N}_{c=1}$. Therefore, we have a paired dataset $\{x_{c},y_{c}\}_{c=1}^{N}$ for each content $c$ in $s$-th training style.

The overall framework is shown in Figure 2. The reference encoder $E_{r}$ first encodes $R_{s}$ into $k$-shot reference maps $F_{r}=\left\{f_{i}\right\}_{i=1}^{k}$, where $f_{i}$ is encoded from $y_{i}$. The content encoder $E_{c}$ extracts the content feature map $f_{c}$ from the input content image $x_{c}$. Our proposed SAM takes $F_{r}$ and $f_{c}$ as inputs, attends to the corresponding spatially local styles in the reference maps $F_{r}$ and aggregates the local styles into the target style map $f_{c,r}=SAM(f_{c},F_{r})$. In the end, the decoder $D$ decodes $f_{c,r}$ into the generated output image $\hat{y}_{c}$. A multi-task projection discriminator[21] is employed to discriminate each generated image and real image. The discriminator outputs a binary classification of fake or real for each character’s style and content category.

An overview of the SAM is depicted in Figure 3. The core in SAM is a multi-head cross attention block that attends to spatially local styles from the reference maps $F_{r}$ and aggregates the reference styles into the fine-grained style representation for the given content image. For the $m$-th attention head, SAM learns a Query map $Q^{m}$ from content feature map $f_{c}$ and Key map $K^{m}$ from reference maps $F_{r}$, which achieves the spatial correspondence matrix $A^{m}$ between the pixels of $Q^{m}$ and $K^{m}$. A value map $V^{m}$ is simultaneously learned from $F_{r}$. Multiplying the correspondence matrix $A^{m}$ with the value map $V^{m}$ aggregates the local style styles into the target style $S^{m}$. As different head captures different information, we combine all the target styles together for decoding.

Formally, we reshape $f_{c}\in\mathbb{R}^{c\times h\times w}$ into a sequence $\tilde{f_{c}}\in\mathbb{R}^{c\times(h\cdot w)}$, where $(h,w)$ is the resolution of the feature map, $c$ is the number of channels. For the $m$-th head, after applying a linear layer $L^{m}_{query}\in\mathbb{R}^{c\times c^{m}}$, we acquire the query matrix $Q^{m}\in\mathbb{R}^{c^{m}\times(h\cdot w)}$. Meanwhile, the reference maps $F_{r}=\left\{f_{i}\right\}_{i=1}^{k}$ are reshaped and concatenated along the spatial dimension $(h,w)$, forming a reference sequence $\tilde{f}_{r}\in\mathbb{R}^{c\times(k\cdot h\cdot w)}$. We multiply $\tilde{f}_{r}$ by two linear projections $L^{m}_{key},L^{m}_{value}\in\mathbb{R}^{c\times c^{m}}$ and generate a key map $K^{m}$ and a value map $V^{m}$ as follows:

We then compute a spatial correspondence matrix $A^{m}$ of which each element $A^{m}(u,v)$ is a pairwise feature correlation between content feature in position $u$ and reference feature in position $v$, calculated as follows:

where $c^{m}$ is the hidden dimension of $Q^{m}$ and $K^{m}$. The $1/\sqrt{c^{m}}$ factor follows Transformers[27] to prevent the magnitude of the dot product from growing extreme.

With the correspondence matrix $A^{m}$, we obtain the aggregated style from references by

After permuting and reshaping $S^{m}$ into $\mathbb{R}^{c^{m}\times h\times w}$, we concatenate all $S^{m}$ along the channel dimension, and employ a linear projection $L_{s}\in\mathbb{R}^{(c^{m}\cdot M)\times c}$ to obtain $S$. The target style map $f_{c,r}$ is obtained as the concatenation between $S$ and content feature $f_{c}$. The decoder $D$ decodes it into the target image $\hat{y}_{c}$ as follows:

where $\circ$ denotes concatenation operator and $M$ is the number of total attention heads.

To achieve a highly style-consistent glyph using SAM, it depends not only on the proper aggregation of styles, but also on the expressivity of the styles that depict fine and local details of the references. However, the $k$-shot font generation setup is a hard-to-learn problem, as it requires the attention learns spatial correspondence from weakly correlated content-references pairs, which is sometimes confusing even for a human expert. Thus, besides the main branch of $k$-shot learning, we introduce an easy-to-learn Self-Reconstruction (SR) branch to boost the learning process.

While the generator is trained in the above mentioned $k$-shot setup, for a given content input $x_{c}$, we have the ground truth image $y_{c}$ to supervise the model training. Self Reconstruction is a parallel branch that shares the same model as the main branch during training. It takes $y_{c}$ as the reference image $\tilde{R_{s}}=\left\{y_{c}\right\}$, and its output $\tilde{y}_{c}$ is also supervised by $y_{c}$ itself. In contrast to Eq. 1, the generation process of this branch is

The detailed training setup can be found in Sec.3.5. In the SR branch, the content and reference images are strongly correlated. The spatial correspondence matrix can be easily learned as the strokes and the components’ relationship between content and reference is clear. With the well-learned correspondence, the generator can be optimized with well-aligned gradients. As a result, the expressivity of depicting details can be further learned within our framework.

In previous works considering the decomposition of components like LF-Font[22], reference characters that contain the same components as content character are randomly selected from the training set during each iteration. The model can hardly learn how to extract the right component-wise features with varying combinations of the reference set. Thus, we introduce a strategy to select a fixed reference set whose components cover most of the commonly used characters and design a content-reference mapping that fixes the combination of reference set for each character. To establish this mapping function, we firstly decompose each character into a component tree, as shown in Figure 4, based on a commonly used decomposition table¹¹1https://github.com/cjkvi/cjkvi-ids/blob/master/ids.txt. We define the components at the level 0, 1, and 2 as the conspicuous components, which contains both radical and compositional structures that easier be transfered from the references to the target.

Reference set selection. This reference set should cover as many conspicuous-level components as possible. Initially, we select a small subset (typically including 100 characters). First, we decompose the characters into component trees. Once the character contains two or more new components, we add this character to our reference set. When the elements in the reference set reach their limits, we stop searching and obtain the style reference set and its corresponding contained components.

Content-reference mapping. After completing the reference set selection, we then establish the mapping relations between content glyphs and style references. We propose a greedy process to find $k$-shot references for a glyph. In this process, we search the reference set for $k$ times to establish a mapping relation. During every searching step, we find the reference glyph that shares the most components with the target glyph. If there are multiple solutions, we select the optimal solution that has the most components with the same structure composition. After selection, we remove the reference from the reference set and continue the next searching step. By this process, we can determine every glyph’s corresponding $k$-shot references.

We train our model to generate the image $\hat{y}_{c}$ from a content glyph image $x_{c}$ and a fixed set of reference glyph images $R_{s}$. In each iteration, the main branch and the SR branch generate $\hat{y}_{c}$ and $\tilde{y}_{c}$ simultaneously and are supervised by the same losses. FSFont learns the reference encoder $E_{r}$, content encoder $E_{c}$, Style Aggregation Module and decoder $D$ with following losses: 1) Adversarial loss with the multi-task discriminator. 2) L1 loss among $\hat{y}_{c}$, $\tilde{y}_{c}$ and a paired ground truth image ${y_{c}}$.

Adversarial loss. Since we our aim is to generate visually high-quality images, we employ a multi-head projection discriminator [22] in our framework. The loss function is represented as follows:

where $\bar{y}_{c}\in\{\hat{y}_{c},\tilde{y}_{c}\}$, $D_{s,c}(\cdot)$ refers to the logits from the character $c$ head and style $s$ head from the discriminator, $P_{g}$ denotes the set of generated images from both main branch and Self-Reconstruction branch, and $P_{d}$ denotes the set of real glyph images.

L1 loss. For learning the pixel-level consistency, we employ an L1 loss between the generated images $\hat{y}_{c}$, $\tilde{y}_{c}$ and ground truth image $y_{c}$.

Overall objective loss. Combining all losses mentioned above, we train the whole model by the following objective:

where $\lambda_{adv}$ and $\lambda_{l1}$ are hyperparameters to control the weight of each loss. We empirically set $\lambda_{adv}=1$ and $\lambda_{l1}=0.1$ in our experiments.

Datasets. We choose 407 fonts and 3396 commonly used Chinese characters as our datasets including handwriting fonts, printed fonts as well as artistic fonts. All images are $128\times 128$ pixels. We select 100 characters from datasets as our reference set and create a Content-Reference mapping with the strategy discussed in Sec.3.4. Reference set and Content-Reference mapping are fixed in both training set and testing set which means only 100 characters are needed to generate a new font library. The training set contains 397 fonts, 2896 characters. The test set are from 10 representative fonts including typewriter fonts, artistic fonts, and handwriting fonts to evaluate the generalization of our model on variant unseen fonts. We test the methods with two setups Unseen Fonts Unseen Characters(UFUC) and Unseen Fonts Seen Characters(UFSC). UFUC refers to the 500 characters in the test fonts, and UFSC refers to the 2896 characters in the test fonts.

Evaluation metrics. To evaluate the similarity between generated images and ground truth, we compare our framework with other SOTA methods in the following metrics, i.e. L1, RMSE, SSIM and LPIPS. Additionally, we conduct user studies to calculate the Character Accuracy, as CNN-based classifiers are tolerant of little defects like missing or broken strokes and blurry edges in a stroke-rich character. We hire 51 volunteers to rigorously count the correct ones from 500 generated characters of each method. A character will be counted as correct only if the volunteer can not spot a defect. For Style Consistency, the volunteers are required to evaluate which of the methods generates the most similar character given a set of reference glyph images from 30 randomly selected cases.

We compare our method with previous SOTA methods. 1) FUNIT[18] is a early work on Few-shot image translation in an Unsupervised way which introduces two different encoders and an AdaIN module to generate a new image with mixed content and style. 2) DG-Font[31] is an Unsupervised network using Deformable Convolution in Generator to achieve a better effect on cursive characters 3) MX-Font[23] adopts multiple experts to extract different local structures which makes cross-lingual generation task possible. (4) AGIS-net[8] uses two different decoders to generate images with shape and texture information which makes generated image more stable. (5) LF-Font [22] proposes localized style representation which makes it enable to extract the component-wise features. For a fair comparison, we choose the Kaiti Line Font ²²2 Font is available at https://chanind.github.io/hanzi-writer-data/ as the standard font and re-train all models on the training datasets in Sec. 4.1.

Quantitative comparison. Table 1 shows the FFG performance of our FSFont and other competitors. We conducted the experiment on UFUC and UFSC. FSFont clearly outweighs previous SOTA methods on all of the similarity metrics from pixel-level to perceptual-level. On UFSC setup, MX-Font[23] and AGIS-Net [8] generates little more correct characters than FSFont. However, FSFont still generates the most correct characters on UFUC. AGIS-Net suffers from a performance drop on Character Accuracy in UFUC. This may suggest that its generalization to new characters are limited. Meanwhile, the user study of Style Consistency of FSFont is remarkably better than other methods, which further verifies that FSFont generates the visually similar results from users’ perspective.

Qualitative comparison We illustrate the generated samples in Figure 5 for each method. We selected four different fonts including typewriter fonts, artistic fonts as well as handwriting fonts to see the generalization of all competitors. As demonstrated in Figure 5, FSFont are available to recover as many details from reference images. Though other methods like LF-Font [22] or AGIS-Net [8] are able to generate stable characters and recover coarse features such as the thickness of strokes and inclination of font, they could not recover details as explicitly as our method does.

Ablation studies. In this part, we discuss the effectiveness of each module we introduce in FSFont. We discard each module at a time and train the model with other settings unchanged. The overall evaluation results on UFUC datasets are shown in Table 2. We replace SAM with averaging features in $F_{r}$ to test its effect. For Reference Selection and Content-Reference mapping, we replace them with the strategy of LF-Font[22] by randomly selecting reference glyph images with a common component set for each content character. Both two modules have a positive effect on the final results. The Self-Reconstruction Branch has a significant impact on outputs. As shown in Table 3, the model trained without SR branch can hardly recover details from reference glyph images.

Visualization of SAM. To demonstrate the effectiveness of SAM, we visualize the attention maps for different levels in Figure 6. Specifically, given a certain spatial point $q$ in the content feature map as a query, we can obtain the corresponding correlations $A^{m}_{q}\in\mathbb{R}^{khw}$ from the spatial correspondence matrix $A^{m}\in\mathbb{R}^{hw\times khw}$, and construct an attention map by reshaping $A^{m}_{q}$ to $h\times kw$. We consider the queries from the Granular, Stroke, and Component-level, respectively, and compute the final attention map by summing over the attention maps related to the queries. It shows that our SAM module empowers the model to attend to the correct FLSs from reference images and extract a sub-component level feature representation for content images.