SAN: Structure-Aware Network for Complex and Long-tailed Chinese Text Recognition

In Chinese text recognition, early works are often character-based. Some works are based on CNN model [9, 25, 29] to design improved or integrated methods. For example, MCDNN [9] integrates multiple models including CNN, which shows advantageous performance in handwritten characters recognition. ART-CNN [25] alternatively trains a relaxation CNN to regularize the neural network during the training procedure and achieves state-of-the-art accuracy. Later, DNN-HMM [10] sequentially models the text line and adopts DNN to model the posterior probability of all HMM states, which significantly outperforms the best over-segmentation-based approach [19]. The SOTA text recognition model, ABINet [11], recommends blocking the gradient flow between the vision and language models, and introduces an iterative correction approach for the language model. These strategies promote explicit language modeling and effectively mitigate the influence of noisy input. However, these methods did not put forward effective countermeasures against the difficult problems of Chinese text recognition like many complex characters, insufficient training samples, and so on, so the performance improvement of these models is greatly limited.

Several methods attempt to design targeted optimization strategies according to the characteristics of Chinese text [26, 8, 23, 28, 22, 27]. Wu et al [26]. use MLCA recognition framework and new writing-style-aware image synthesis method to overcome large character sets and great insufficient training samples problems. In [8], the authors apply Maximum Entropy Regularization to regularize the training process to optimize the large amount of fine-grained Chinese characters and the great imbalance over class problems. Wang et al. [23] utilize the similarity of Chinese characters to reduce the total number of HMM states and model the tied states more accurately. These methods pay attention to the particularity of Chinese characters and give targeted optimization. However, they are still character-based, which makes it difficult to further explore the deep features of Chinese characters.

In recent years, radical-based approaches have shown outstanding advantages in Chinese recognition [31, 20, 2, 21, 24]. RAN [31] employs an attention mechanism to extract radicals from Chinese characters and to detect spatial structures among the radicals, which reduce the vocabulary size and can recognize unseen characters. FewShotRAN [20] maps each radical to a latent space and uses CAD to analyze the radical representation. HDE [2] designs an embedding vector for each character and learns both radicals and structures of characters via a semantic vector, which achieves superior performance in both traditional HCCR and zero-shot HCCR tasks. Inspired by these works, our method emphasizes the role of structural information of radical in visual perception and proposes to utilize the common local and structural traits between characters to optimize the recognition performance in complex and long-tailed characters.

As characters can be decomposed into basic components, we model it as a tree composed of various components hierarchically and model the label as a forest composed of character trees, which gives the label a structural representation.

As shown in Fig.2 (a), in Chinese, one popular modeling method is to decompose characters into radicals and structures [7]. Radicals are the basic components of characters, and structures describe the spatial composition of the radicals in each character. One structure is often associated with two or more radicals, and the structure is always the root node of a subtree. In this way, all Chinese characters can be modeled as a tree, in which leaf nodes are radicals, and non-leaf nodes are structures. For simplicity, we call such trees “radical trees” in this paper.

In this work, we propose a novel Tree Similarity (TreeSim) metric to measure the visual similarity between two characters represented with radical trees. In a radial tree, the upper components represent large and overall information, while the lower components represent small and local information. When comparing the similarity of two characters, humans tend to pay more attention to the upper components and less attention to the lower ones.

Correspondingly, as shown in Fig.2 (b), we first propose a weighting method for TreeSim. The method includes the following three characteristics: First, upper components in the radical tree have greater penalty weight. Second, every subtree is regarded as an independent individual, the root node and its subtrees have equal weight. Third, the total weight of the tree is 1. The calculation procedure is depicted in Algorithm 1, where $Node$ is the pointer of the root node and $w_{sub}$ is the weight of the subtree. The weight of each node can be calculated by the recursion method.

Based on this weighting method, the TreeSim calculation sample is shown in Fig.3. The calculation process is as follows: First, select any one of the two radical trees, and traverse every node in preorder. Second, judge whether it is matching for each node. The matching rules include: 1)Every ancestor of this node is matching. 2)This location in the other tree also has a node and the two nodes are the same. Last, TreeSim is equal to the sum of the weights of all matching nodes.

The proposed TreeSim is hierarchical, bidirectional, and deep-independent. Hierarchical means TreeSim pays different attention to different levels of the tree. Bidirectional means no matter select which of the two trees, the calculation result is the same. Depth-independent means the weight of nodes at a certain level is independent of the depth of the tree.

Based on this label modeling method, we proposed two loss design strategies. Considering that a rigorous calculation for the loss of two forests may be complex and nonlinear, we introduce two linear approximate calculation methods in 3.2 to supervise the loss calculation.

Inspired by the radical-based methods [20, 7], we first propose an Auxiliary Radical Branch (ARB) module to enhance the structure-awareness of visual features by using the hierarchical representation as a regularization term. Unlike the PRAB [7] method, the proposed ARB module directly decodes the feature map extracted from the Feat. stage, thus do not need individual character features and can be theoretically applied to most methods with a Feat. stage [1]. As shown in Fig.1, ARB includes the following characteristics: 1)The label representation in ARB is hierarchical, which contains component and structural information. 2)ARB is regarded as an independent visual perception optimization branch, which takes feature as input, decodes according to structured representation, and finally refines the visual feature extraction.

The role of ARB is reflected in two aspects: One is that ARB can use hierarchical label representation to supply and refine the feature extraction, improving the structure-awareness of the model. The other is that by taking advantage of the common component combination traits between Chinese characters, ARB could exploit the component information connections between tail classes and head classes, which can optimize long-tailed character recognition.

In ARB, we propose two linear loss design strategies, which reflect the hierarchical label representation while simplifying the design and calculation.

The training gradient flow by two branches is superimposed and updates the weight of the feature extractor together, which allows the radical level prediction model to supplement and refine the visual feature extraction of the character prediction model in the training process. we define the loss as follows

where $L_{base}$ is the loss of base model, which is the loss from ABINet in this work, i.e.,

Both the character prediction branch and radical structure prediction branch affect the feature extraction. For fairness, we give them the same weight of influence on visual feature extraction, to ensure that the performance of both parties is fully reflected.

The datasets used in our experiments are Web Dataset and Scene Dataset [7]. The Web Dataset contains 20,000 Chinese and English web text images from 17 different categories on the Taobao website and the Scene Dataset contains 636,455 text images from several competitions, papers, and projects. The number of radical classes is 960. We set the value of R (Fig.1) to be 33 for the web dataset and 39 for the scene dataset, covering a substantial 95$\%$ of the training samples from both datasets.

We implement our method with PyTorch and conduct experiments on three NVIDIA RTX 3060 GPUs. Each input Image is resized to $32\times 128$ with data augmentation. The batch size is set to 96 and ADAM optimizer is adopted with the initial learning rate of $1e^{-4}$, which is decayed to $1e^{-5}$ after 12 epochs.

We discuss the performance of the proposed approaches (ARB and TreeSim weighting) with two base-model configurations: Vision Model and ABINet. Experiment results are recorded in Tab.1.

First, the proposed ARB is proved useful, by significantly improving the accuracy of both base-metods. Specifically, the VM-Naive outperforms VM-Base by 1.6$\%$ accuracy and 0.011 1-NED, ABINet-Naive outperforms ABINet-Base by 1.2$\%$ accuracy and 0.009 1-NED. Second, TreeSim enhanced weighting also achieves expected improvements. VM-TreeSim boosts accuracy by 0.6$\%$ and 1-NED by 0.007 than VM-Naive, SAN boosts accuracy by 0.5$\%$ and 1-NED by 0.005 than ABINet-Naive.

The above observations suggest that the hierarchical components information is useful to the feature extractor, which can significantly improve the performance of the vision model. The proposed approaches also yields significant improvement against full ABINet indicating hierarchical components information still plays a significant part, even when the language models can, to some extent, alleviate the confusions caused by insufficiency shape awareness.

Compared with Chinese text recognition benchmarks and recent SOTA works that are trained on web and scene datasets, SAN also shows impressive performance(Tab.2). We can see from the comparison, our SAN outperforms ABINet with 1.7$\%$, 1.8$\%$ on Web and Scene datasets respectively. Also, SAN achieves the best 1-NED on both datasets. Some successful recognition examples are shown in Fig.4, which shows the complex characters and long-tailed characters that predict failure using the base model (ABINet) while predicted successfully by the full model.

To validate the optimization of ARB on complex character recognition and long-tailed character recognition, we divide them into different groups according to their complexity and frequency. For each category, to give a better understanding of the differences in recognizing performances, the character prediction samples by SAN and ABINet are studied in more detail. Specifically, we calculated the accuracy of character prediction results and the average TreeSim between the predicted results and the ground truth. TreeSim serves as a more fine-grained metric compared to accuracy, as it can indicate the awareness of structural information.