ICAL: Implicit Character-Aided Learning for Enhanced Handwritten Mathematical Expression Recognition

In traditional handwritten mathematical expression recognition, the process mainly involves symbol recognition and structural analysis. Symbol recognition requires segmenting and identifying individual symbols within expressions, utilizing techniques like pixel-based segmentation proposed by OKAMOTO et al [21]. and further refined by HA et al [8]. into recursive cropping methods. These approaches often depend on predefined thresholds. For symbol classification, methods like Hidden Markov Models (HMM)[24, 11, 1], Elastic Matching[3, 23], and Support Vector Machines (SVM)[10] have been used, where HMMs allow for joint optimization without explicit symbol segmentation, though at a high computational cost.

Structural analysis employs strategies like the two-dimensional Stochastic Context-free Grammar (SCFG)[6] and algorithms such as Cocke-Younger-Kasami (CKY)[16] for parsing, albeit slowly due to the complexity of two-dimensional grammar. Faster parsing has been achieved with Left-to-right Recursive Descent and Tree Transformation methods[29], the latter describing the arrangement and grouping of symbols into a structured tree for parsing. Some approaches bypass two-dimensional grammar altogether, using Define Clause Grammar (DCG) [4]and Formula Description Grammar[18] for one-dimensional parsing, highlighting the challenges in designing comprehensive grammars for the diverse structures of mathematical expressions.

In recent years, encoder-decoder-based deep learning models have become the mainstream framework in the field of HMER. Depending on the architecture of the decoder, past deep learning approaches can be categorized into methods based on Recurrent Neural Networks (RNNs) [33, 30, 7, 31, 26, 27, 19, 2, 12, 32, 25, 28] and those based on the Transformer model [36, 35, 34]. Furthermore, based on the decoding strategy, these methods can be divided into those based on sequence decoding [33, 30, 7, 31, 26, 27, 19, 2, 12, 36, 35, 34] and those based on tree-structured decoding  [32, 25, 28].

Similar to most of the previous work [30, 7, 31, 26, 27, 19, 2, 12, 36, 35, 34, 32, 25, 28], we continue to use DenseNet [9] as the visual encoder to extract features from the input images.

DenseNet consists of multiple dense blocks and transition layers. Within each dense block, the input for each layer is the concatenated outputs from all preceding layers, enhancing the flow of information between layers in this manner. The transition layers reduce the dimensions of the feature maps using $1\times 1$ convolutional kernels, decreasing the number of parameters and controlling the complexity of the model.

For an input grayscale image of size $1\times H_{0}\times W_{0}$, the output visual feature $\mathbf{V}_{feature}\in\mathbb{R}^{D\times H\times W}$. The ratios of $H$ to $H_{0}$ and $W$ to $W_{0}$ are both 1/16. In this work, a $1\times 1$ convolutional layer is used to adjust the number of channels $D$ to $d_{\text{model}}$, aligning it with the dimension size of the Transformer decoder.

In the decoder part, we employ a Transformer Decoder with Attention Refine Module(ARM) as the decoder [22, 35], which mainly consists of three modules: Self-Attention Module, Coverage Attention Module, and Feed-Forward Network.

The mechanism first calculates a scaled dot-product attention score $\mathbf{E}_{i}$ for the $i$-th head by multiplying the transformed queries and keys and then scaling the result by the square root of the dimension of the key vectors $\mathbf{K}$ to avoid large values that could hinder softmax computation:

A softmax function is then applied to these scores to obtain the attention weights $A_{i}$:

These weights are used to compute a weighted sum of the values, producing the output $H_{i}$ for each head:

Finally, the outputs of all heads are concatenated and linearly transformed to produce the final output of the multi-head attention module:

Since the decoder performs decoding in an autoregressive manner, the prediction of the current symbol depends on past predicted symbols and input visual information. To avoid the decoder obtaining information about future symbols when predicting the current symbol and to ensure parallelism, Self-Attention Module uses a masked lower-triangular matrix to constrain the information that the self-attention module can access at the current step.

For the target LaTeX sequence, we only retain its implicit characters, namely “^”, “_”, “{”, and “}”, replacing other characters with newly constructed ones to form a sequence corresponding to the implicit characters. For example, for the target LaTeX sequence B _ { m + 1 } , the corresponding implicit character sequence is <space> _ { <space> <space> <space> }.

The output of the Transformer Decoder with ARM serves as the input to our Implicit Character Construction Module (ICCM), which consists of a layer of masked self-attention and a Feed-Forward Network (FFN) layer. Consequently, the output of the ICCM, $\mathbf{I}_{\text{feature}}$, is calculated as follows:

To integrate the information learned by the ICCM, we introduce a weighted adjustment strategy based on the attention mechanism, capable of aggregating the output of the ICCM, thereby correcting the prediction results of the Transformer Decoder.

The Fusion Module assimilates inputs from both the ICCM’s output, denoted as $\mathbf{I}_{\text{feature}}\in\mathbb{R}^{B\times T\times d_{\text{model}}}$, and the Transformer Decoder’s output, denoted as $\mathbf{E}_{\text{feature}}\in\mathbb{R}^{B\times T\times d_{\text{model}}}$. The attention weights $\mathbf{f}_{att}$ are calculated as follows:

where a linear layer $w_{att}$ maps the concatenate feature matrix back to the original dimension $d_{\text{model}}$, and attention weights $f_{att}$ are computed through a sigmoid activation function $\sigma$.

Finally, $f_{att}$ are used to perform a weighted fusion of the two features, resulting in the final output feature for prediction.

where $\odot$ indicates element-wise multiplication.

To alleviate the issue of unbalanced output, we adhere to the bidirectional training strategy used in BTTR and CoMER, necessitating the computation of losses in both directions within the same batch.

The total loss function is divided into three parts:

where both $\mathcal{L}_{\text{Initial}}$ and $\mathcal{L}_{\text{fusion}}$ are standard cross-entropy loss functions.

$\mathcal{L}_{\text{Initial}}$ calculates the loss using the Transformer decoder’s predicted probabilities and the ground truth, consistent with the approach used in CoMER. $\mathcal{L}_{\text{fusion}}$ uses the predicted probabilities outputted by the Fusion Module and the ground truth.

Since we construct the implicit character sequence directly from the LaTeX sequence, keeping its length consistent with the original LaTeX sequence, the constructed target implicit character sequence exhibits an imbalance in the occurrence frequency between <space> and implicit characters. To address this, we use a weighted cross-entropy loss function to calculate the loss for implicit characters, as follows:

where $N$ represents the batch size, $T_{i}$ is the length of the $i$-th sequence, $y_{i,t}$ is the ground truth token at position $t$ in the $i$-th sequence, $\hat{y}_{i,t}$ is the predicted probability of the correct token at position $t$ in the $i$-th sequence, and $w_{y_{i,t}}$ is the weight associated with the ground truth token $y_{i,t}$.

The weight $w_{y_{i,t}}$ for each token is dynamically adjusted based on the occurrence frequency of the token within the entire batch of sequences. The adjustment is made using a logarithmic function to ensure a smooth transition of weights across different frequencies:

where $f_{y_{i,t}}$ is the frequency of the token in the target sequences and $\epsilon$ is set to $1e-6$.

The CROHME dataset includes data from the Online Handwritten Mathematical Expressions Recognition Competitions (CROHME) [14, 15, 13] held over several years and is currently the most widely used dataset for handwritten mathematical expression recognition. The training set of the CROHME dataset consists of 8,836 samples, while the CROHME 2014 [14]/2016 [15]/2019 [13] test sets contain 986, 1147, and 1199 samples respectively. In the CROHME dataset, each handwritten mathematical expression is stored in InkML format, recording the trajectory coordinates of the handwritten strokes. Before training, we converts the handwritten stroke trajectory information in the InkML files into grayscale images, and then carries out model training and testing.

The HME100K dataset [28] is a large-scale collection of real-scene handwritten mathematical expressions. It contains 74,502 training images and 24,607 testing images, making it significantly larger than similar datasets like CROHME. Notably, it features a wide range of real-world challenges such as variations in color, blur, and complex backgrounds, contributed by tens of thousands of writers. With 249 symbol classes, HME100K offers a diverse and realistic dataset for developing advanced handwritten mathematical expression recognition systems.

The Expression Recognition Rate (ExpRate) is the most commonly used evaluation metric for handwritten mathematical expression recognition. It is defined as the percentage of expressions that are correctly recognized out of the total number of expressions. Additionally, we use the metrics “$\leq 1$ error” and “$\leq 2$ error” to describe the performance of the model when we tolerate up to 1 or 2 token prediction errors, respectively, in the LaTeX sequence.

We employs DenseNet [9] as the visual encoder to extract visual features from expression images. The visual encoder utilizes 3 layers of DenseNet blocks, with each block containing 16 bottleneck layers. Between every two DenseNet blocks, a transition layer is used to reduce the spatial dimensions and the channel count of the visual features to half of their original sizes. The dropout rate is set at 0.2, and the growth rate of the model is established at 24.

We employ a 3-layer Transformer Decoder with ARM [22, 35] as the backbone of the decoder, with a model dimension ($d_{\text{model}}$) of 256 and head number set to 8. The dimension of the feed-forward layer is set to 1024, and the dropout rate is established at 0.3. The parameter settings for the Attention Rectifying Module (ARM) are consistent with those of CoMER, with the convolutional kernel size set to 5. Additionally, the parameters of masked self-attention and FFN in Implicit Character Construction Module(ICCM) are consistent with the aforementioned Transformer Decoder.

During the training phase, we utilize Mini-batch Stochastic Gradient Descent(SGD) to learn the model parameters, with weight decay set to 1e-4 and momentum set at 0.9. The initial learning rate is established at 0.08. We also adopt ReduceOnPlateau as the learning rate scheduler, whereby the learning rate is reduced to 25% of its original value when the ExpRate metric ceases to change. When trained on the CROHME dataset, the CROHME 2014 test set [14] is used as the validation set to select the model with the best performance. During the inference phase, we employ the approximate joint search method previously used in BTTR [36] to predict the output.

Table  1 presents the results on the CROHME dataset. To ensure fairness in performance comparison and considering that different methods have used various data augmentation techniques, many of which have not been disclosed, we have limited our comparison to results without the application of data augmentation. Given the relative small size of the CROHME dataset, we conducted experiments with both the baseline CoMER and the proposed ICAL model using five different random seeds (7, 77, 777, 7777, 77777) under the same experimental conditions. The reported results are the averages and standard deviations of these five experiments.

It is noteworthy that while GCN  [34] has attained impressive results on CROHME 2016 [15] and 2019 [13], its performance benefits from the additional introduction of category information, while ICAL constructs the implicit character sequence directly from the LaTeX sequence, eliminating the need for manually constructing additional category information. Consequently, we present the performance of the GCN solely for reference purposes and exclude it from direct comparisons. The CoMER model represents the current state-of-the-art(SOTA) method; however, CoMER[35] did not disclose their results without data augmentation in the original paper. Therefore, we have reproduced the results of CoMER without data augmentation using their open-source code, denoted by $\dagger$ in the table 1.

As shown in table 1, our method achieved the best performance across all metrics. Our method outperforms CoMER by 2.25%/1.81%/1.39% on the CROHME 2014, 2016, and 2019 datasets, respectively. Across all metrics, the ICAL method achieves an average improvement of 1.6% over the CoMER method. The experimental results on CROHME dataset prove the effectiveness of our method.

We also conducted experiments on the challenging and real-world dataset HME100K, as shown in table 3. We have replicated the performance of CoMER on the HME100K dataset, and our method surpasses the state-of-the-art(SOTA) by 0.94%, reaching an impressive 69.06%. The outstanding experimental performance on the HME100K dataset, which is more complex, larger, and more realistic compared to CROHME, further proves the superior generalization ability and effectiveness of the ICAL method.

Our research includes a series of ablation studies to corroborate the effectiveness of our proposed method. Presented in Table  4, Initial Loss refers to the cross-entropy loss computed from the discrepancy between the Transformer Decoder’s direct output and the ground truth (the intact LaTeX code). Fusion Loss is the cross-entropy loss determined by comparing the combined outputs from both the Implicit Character Construction Module(ICCM) and Decoder—synergized through the Fusion Module—with the ground truth (the intact LaTeX sequence). Implicit Loss is calculated using a cross-entropy formula with adaptive weighting, which evaluates the discrepancies between the ICCM’s output and the sequence of implicit characters. When Implicit Loss is not applied, the ICCM module is also omitted, serving as a validation of ICCM’s effectiveness.

Additionally, it should be noted that in the ablation experiments mentioned above, when utilizing Fusion Loss, we employ the output of the Fusion Module for inference. Conversely, when Fusion Loss is not implemented, inference is conducted directly using the output from the Transformer Decoder.

From the 4th and 5th rows of each dataset in the table, it is evident that compared to using only the Initial Loss(1st row), which serves as the baseline, CoMER, Implicit Loss(4th row) and the Fusion Loss (5th row) that we have introduced can both effectively enhance the model’s recognition performance. We also designed experiments that exclusively utilize Fusion Loss and Implicit Loss (the 3rd row), from which it can be observed that, relative to the baseline approach, our method still manages to achieve a notable improvement in effectiveness.

Due to time limitations, the ablation study conducted on the HME100K dataset only used a single random seed for each experiment, and thus, the results are not averaged over multiple runs. This may introduce some variability in the reported performance, and we plan to address this limitation by conducting additional experiments with multiple seeds in future work for more robust evaluation.

As shown in Table  5, we have evaluated the inference speed of our method on a single NVIDIA 2080Ti GPU. Compared to the baseline model, our method has a modest increase in the number of parameters with a negligible impact on FPS, and there is also a slight increase in FLOPs.

We provide several typical recognition examples to demonstrate the effectiveness of the proposed method, as shown in Fig. 3. Entries highlighted in red indicate cases where the model made incorrect predictions. ’ICCM’ represents the implicit character sequence predicted by the ICCM module, where <s> is the abbreviation for the <space> token.

In Case (a), the baseline CoMER model incorrectly identified the first character, $\pi$, as y_{0}. In contrast, the ICCM correctly determined that there were no implicit characters in the formula, as indicated on the fourth line of Group a. This accurate detection allowed for the correct LaTeX  sequence to be output by ICAL, as shown on the third line of Group a.

In Case (b), the ICCM’s prediction of the implicit character sequence (fourth line of Group b) was crucial. It enabled the ICAL method to correctly place both characters 3 and 4 in the subscript of q (third line of Group b), unlike CoMER, which misidentified this relationship (second line of Group b).

Case (c) demonstrates that the ICCM’s prediction of implicit characters can also alleviate the lack of coverage issue [20, 33] . Here, ICCM’s prediction indicated that there should be two explicit characters, 9 and 1, within { }, and correspondingly, ICAL also successfully predicted these two characters. However, CoMER only predicted the character 9, and missed the character 1.

Moreover, Case (d) also effectively highlights how ICCM and its ability to predict implicit characters can enhance a model’s understanding of the structural relationships within formulas.

This work is supported by the projects of National Science and Technology Major Project (2021ZD0113301) and National Natural Science Foundation of China (No. 62376012), which is also a research achievement of Key Laboratory of Science, Technology and Standard in Press Industry (Key Laboratory of Intelligent Press Media Technology).