Towards Robust Handwritten Text Recognition with On-the-fly User Participation

Optical Character Recognition (ocr) is the electronic conversion of printed or handwritten document images into a machine-readable form. ocr is an essential component for document image analysis. Typically, an ocr system includes two main modules (i) a text detection module and (ii) a text recognition module. A text detection module aims to localize all text blocks within the image, either at the word or line levels. In contrast, the text recognition module strives to understand the text image content and transcribe the visual signals into natural language tokens. The problem of handwritten text recognition is more exciting and challenging than printed text due to uneven variations in handwriting style to the writers, content, and time. A person’s handwriting is always unique, and this unique property creates motivation and interest among researchers to work in this imperative and challenging field.

In the field of English handwritten text recognition tasks, several methods are proposed; (i) methods (Chaudhary and Bali, 2021; Ptucha et al., 2019; Yousef et al., 2020) based on only convolutional neural networks, (ii) methods (Kang et al., 2022; Diaz et al., 2021; Li et al., 2021) based on transformer networks, and (iii) methods (Wigington et al., 2017; Bluche and Messina, 2017; Michael et al., 2019; Kang et al., 2018; Bluche, 2016; Chowdhury and Vig, 2018; Sueiras et al., 2018) based on cnn-rnn architectures. Among almost $7000$ languages around the world, ocr systems are mostly available for languages that are of huge importance and strong economic value like English (Graves and Schmidhuber, 2008; Pham et al., 2014; Li et al., 2021), Chinese (Xie et al., 2016; Wu et al., 2017; Peng et al., 2022), Arabic (Maalej and Kherallah, 2020; Jemni et al., 2022), and Japanese (Ly et al., 2018; Nguyen et al., 2020). Among $22$ Indic languages, most of the languages derived from Indic script appear to be at risk of vanishing due to the absence of efforts to preserve them. In many Indic scripts, two or more characters often combine to form conjuncts, considerably increasing the vocabulary to be tackled by ocr systems (cit, 2020). These inherent features of Indic scripts make Handwritten Recognizer (hwr) more challenging than Latin scripts. In contrast to the 52 unique (upper case and lower case) characters in English, most Indic scripts have over 100 unique basic Unicode characters (Pal and Chaudhuri, 2004). Several methods offer solutions to Indic handwritten text recognition tasks. These methods can be categorised into (i) segmentation-free but lexicon-dependent methods (Shaw et al., 2008, 2014; Kaur and Kumar, 2021), (ii) segmentation dependent methods (Arora et al., 2010; Labani et al., 2018; Alonso-Weber et al., 2014; Roy et al., 2016), and (iii) sequence-to-sequence i.e., cnn-rnn methods (Adak et al., 2016; Dutta et al., 2017, 2018; Gongidi and Jawahar, 2021).

All the ocr systems mentioned above for English and Indian languages perform well when the test set’s distribution is similar to the data used to train them. When the test set’s distribution deviates from the training set, the performance of these ocr systems drastically reduces. This situation often arises in a real-world scenario and often leads to a dissatisfied user experience, as shown at the top of Fig. 1. In this work, we propose a controlled framework for long-term ocr services providing high-quality output to their users at fixed budgets. Robust ocr systems must update the models that fail on the complex documents uploaded by the users. As Fig. 1 (bottom) depicts, the regular upgradation of the model collaboratively on users’ data and standard datasets can help achieve a satisfactory experience for users as well as service providers. The process also involves user selection based on the available budget for annotation and rewarding the users for the data. We expressly assume that serving a large number of users, say $N$, is the end goal of the ocr system. In our setting, we have $N_{1}=4$ users available at the first iteration of upgrading the model, $N_{2}=10$ at the second, and $N_{3}=15$ at the third iteration. We assume that $N_{3}$ represents $N$ to investigate the subject matter in a controlled setting. We fix the budget of $m=4$ users for each iteration. At the start of the second (and third) iteration, all the new users write a page, which the experts or service providers analyze to select the $m$ (new) best users based on the quality of ocr predictions on the human-readable words. Each of the selected $m$ users (or fixed $m$ users for iteration 1) writes $23$ more pages for training the model. We upgrade the model in each iteration with Curriculum Learning (cl) and test it on the held-out set consisting of one page each from all the users ($N_{3}$). We provide insights analysis on the effect of cl, user selection, and performance of our system on unseen writing styles. The key contributions of this work are as follows:

• •

  Our setup is unique compared to the previous ocr works which do not include users’ satisfaction or participation. We aim to solve the problem in a real-life scenario, where users continuously grow for an ocr service, trying to satisfy them with the limited budget of the service provider.

• •

  Our experiments show that i) cl can be effective in continuously upgrading the ocr model on users’ data, ii) user selection based on ocr predictions on human-readable user’s data can improve results over other selections with fixed budget, and iii) Observation on the dataset from users which the model has never seen (or trained upon) can help quantify the robustness of the models.

OCR Services: Tesseract (Tesseract, 2022) is an open-source ocr system that works for over $100$ languages. The first three versions of Tesseract ocr work on the principle of classical machine learning techniques. A Gaussian Mixture Model (GMM) based classifier identifies characters from features based on the vectors obtained from text boundaries. The subsequent versions of Tesseract involve line-level deep models. Google Docs ocr works for over $245$ languages (Google, 2020). Tafti et al. (Tafti et al., 2016) perform qualitative and quantitative analysis on Google Docs ocr, Tesseract, ABBYY FineReader, and Transym using $1227$ images from $15$ categories like printed, handwritten, noisy, multi-oriented, multi-lingual images, etc. While most services work well on printed text, many fail on handwritten and noisy text images. While some of the commercial service providers might be using the users’ data to improve their models, none of the ocr services mentioned above provide a transparent up-gradation process based on users’ inputs like ours with a focus on improving the overall user experience.

Handwritten Text Recognition: In the space of English handwritten text recognition, few works have used entirely convolutional neural networks without using any recurrent architectures (Chaudhary and Bali, 2021; Ptucha et al., 2019; Yousef et al., 2020). Few recent networks (Ingle et al., 2019; Coquenet et al., 2020) use gating mechanisms in cnns to compensate for the dependency on lstm cells, known as gated Convolutional Neural Networks (gcn). These types of networks outperform fully convolutional Network yet they lag behind rnn/Transformer based ocr models (Kang et al., 2022; Diaz et al., 2021; Li et al., 2021). Recurrent Neural Networks (rnns) are successfully applied to solve Handwritten Text Recognition (htr) tasks. LSTM-based models can handle long-term context in sequences. The most common architectures (Wigington et al., 2017; Bluche and Messina, 2017) are combination of cnn and rnn, where cnn is used for feature extraction from word or line images and rnn is used for modeling sequential context. Several works (Michael et al., 2019; Kang et al., 2018; Bluche, 2016; Chowdhury and Vig, 2018; Sueiras et al., 2018) use various attention mechanisms to improve the performance of cnn +lstm. Recently, Transformer based text recognizers (Kang et al., 2022; Diaz et al., 2021; Li et al., 2021) achieved state-of-the-art performance. Some of these works use a cnn-based backbone with self-attention as encoders to understand document images (Li et al., 2021).

Officially, there are $22$ languages in India, many of which are used only for communication purposes. Among these languages, Hindi, Bangla, and Telugu are the top three languages in terms of the percentage of native speakers (Krishnan and Jawahar, 2019). In many Indic scripts, two or more characters often combine to form conjuncts which considerably increasing the vocabulary to be tackled by ocr systems (cit, 2020). These inherent features of Indic scripts make Handwritten Recognizer (hwr) more challenging than Latin scripts. Compared to the 52 unique (upper case and lower case) characters in English, most Indic scripts have over 100 unique basic Unicode characters (Pal and Chaudhuri, 2004).

Three popular ways of building handwritten word recognizers for Indic scripts are available in the literature. The first one is segmentation-free but lexicon-dependent methods which train on the representation of the whole words (Shaw et al., 2008, 2014; Kaur and Kumar, 2021). Shaw et al. (Shaw et al., 2008) represent word images using a histogram of chain-code directions in the image-strips scanning from left to right by a sliding window as the feature vector. A continuous density Hidden Markov Model (hmm) recognizes handwritten Devanagari words. Shaw et al. (Shaw et al., 2014) discuss a novel combination of two different feature vectors for holistic recognition of offline handwritten word images in the same direction. The second category of approaches involves segmentation of the characters within the word images and recognition of isolated characters using an isolated symbol classifier such as Support Vector Machine (svm) (Arora et al., 2010), and Artificial Neural Network (ann) (Labani et al., 2018; Alonso-Weber et al., 2014). Roy et al. (Roy et al., 2016) segment Bengali and Devanagari word images into the upper, middle, and lower zones, using morphology and shape matching. The symbols in the upper and lower zone are recognized using an svm, while a Hidden Markov Model (hmm) is used to recognize the characters in the middle zone. Finally, the results from all three zones are combined. This category of approaches suffers from the drawback of using an error-prone script-dependent character segmentation algorithm. The third category of approaches treats word recognition as a sequence-to-sequence prediction task where both the input and output are treated as vector sequences. The aim is to maximize the probability of predicting the output label sequence given the input feature sequence (Adak et al., 2016; Dutta et al., 2017, 2018). Garain et al. (Garain et al., 2015) propose a recognizer using Bidirectional Long Short-term Memory (blstm) with Connectionist Temporal Classification (ctc) layer to recognize unconstrained Bengali offline handwriting words. Adak et al. (Adak et al., 2016) use Convolutional Neural Network (cnn) integrated with an lstm along with a ctc layer to recognize Bengali handwritten word. In the same direction, Dutta et al. proposed CNN-RNN hybrid end-to-end model to recognize Devanagari, Bengali (Dutta et al., 2017), and Telugu (Dutta et al., 2018) handwritten words. In one of the works by Gongidi (Gongidi and Jawahar, 2021), the authors use Spatial Transformer Network along with hybrid cnn-rnn with ctc layer to recognize word images in eight different Indic scripts such as Bengali, Gurumukhi, Gujarati, Odia, Kannada, Malayalam, Tamil, and Urdu. The authors use various data augmentation functions to improve recognition accuracy. This category of methods does not require character-level segmentation and is not bound to recognize a limited set of words.

Curriculum Learning: The Curriculum Learning (cl) methods are commonly used for computer vision tasks like object recognition (Hacohen and Weinshall, 2019; Mousavi et al., 2021; Wang et al., 2019) and object detection (Singh et al., [n.d.]; Goyal et al., 2022; Wang et al., 2018). The works mentioned above use cl to handle intra-class scale variations, inter-class confusions, and challenges involved in weakly-supervised or semi-supervised training. Our method utilizes the cl-based handwriting recognition ocr model to collaboratively learn from the datasets of the service providers and continuously update the dataset from the growing number of users.

This section describes the assumptions and the pipeline of our framework. Consider the problem of updating an ocr service on the complex data provided by different users. The number of users generally grows in such scenarios. The end goal of ocr service is to provide a satisfactory experience to many users, which we refer to as $N$. However, practically all the users cannot contribute much data but can contribute a few (one or two) pages. Some of the regular users will naturally contribute more data for upgrading the model at arbitrary intervals since such users are also interested in utilizing such a model repeatedly for their documents. The overall situation can be complex in real-life scenarios to the extent that users can arbitrarily grow. The budget of the service providers and the number of pages shared by each can also vary with time. Hence, we make the following assumptions to investigate the work in the controlled setting, which we discuss in the next subsection.

We use the network architecture proposed by Gongidi et al. (Gongidi and Jawahar, 2021) for this experiment and refer to its model and dataset as belonging to iteration $0$. The used network consists of four modules: Transformation Network (tn), Feature Extractor (fe), Sequence Modeling (sm), and finally Predictive Modeling (pm). The transformation Network has six plain convolutional layers with $16$, $32$, $64$, $128$, $128$, and $128$ channels. Each layer has the filter size, stride, and padding size of $3$, $1$, and $1$, followed by a $2\times 2$ max-pooling layer with a stride of $2$. The Feature Extractor module consists of ResNet architecture. The Sequence Modeling consists of a $2$ layer Bidirectional lstm (blstm) architecture with $256$ hidden neurons in each layer. The predictive Modeling consists of Connectionist Temporal Classification (ctc) to decode and recognize the characters by aligning the feature sequence and the target character sequence. We resize input images into $96\times 256$. We use Adadelta optimizer for Stochastic Gradient Descent (sgd) for all the experiments. We set the learning rate to $0.001$, batch size to $64$, and momentum to $0.09$. For every curriculum fine-tuning, we reset the learning rate to 0.001. The code and trained model are available at ²²2https://github.com/ajoymondal/Towards-Robust-Handwritten-Text-Recognition-with-On-the-fly-User-Participation.

We carry out the following experiments to study the effect of curriculum learning, and user selection:

• •

  Iter1CL: We compare the Iteration $1$ model trained with i) Curriculum Learning (cl) on dataset from iterations $[0,1]$ and ii) without cl, i.e., with just Fine Tuning Iteration $0$ model on Iteration $1$ training sets. The former is referred to as Iter1CL and latter as Iter1FT.

• •

  Iter2CLm: For Iteration $2$, we compare the model trained on i) training set from $m$ selected users (Iter2CLm) and ii) swapping the training set from one and two of the users in the selected set of $m$ users, with a training set of remaining (non-selected) users available in Iteration $2$. We refer to them as Iter2CLmS1 and Iter2CLmS2. For Iter2CLmS1 the dataset from a user with the highest ocr quality³³3We refer to the user whose validation set has the highest ocr quality among other users as the best performing user. among $m$ users is swapped with data from user with the overall highest ocr quality on validations set. For Iter2CLmS2, the dataset from two users with the highest ocr quality, among $m$ users are swapped with data from the two users with the overall highest ocr quality on validations set.

• •

  Iter3CLm: Finally, we compare the model upgraded on the data collected in the third iteration Iter3CLm, with Iter1CL and Iter2CLm on overall test set, common seen ⁴⁴4Seen/unseen throughout the paper means the model has seen/unseen the writer’s (or user’s) distribution during training. A common seen/unseen test set is the test set from users whose data is seen/unseen in all the iterations. test set, and common unseen test set, to demonstrate the generalization capability of the final model.

Effect of Curriculum Learning: As shown in the first two rows of Table 1, just fine-tuning the model by Gongidi et al. (Gongidi and Jawahar, 2021) on Iteration $1$ dataset, improve the Character Recognition Rate (CRR) and Word Recognition (WRR) significantly by $19\%$ and $38\%$ on the test set. However, Curriculum Learning (cl) on the samples collected from the first four users and the equal number of samples from the large training set of Gongidi et al. (Gongidi and Jawahar, 2021) (words) help us with further improvements, as shown in the third row of Table 1. As we will see in the next paragraphs, the cl also helps us improve results in subsequent iterations.

Effect of User Selection: The results of Iter1CL model of validation set of Iteration $2$ from users with IDs $[5,10]$ are shown in Table 2. Firstly, as shown in the last two columns of the table, the WRR on Human-Readable (WRR-HR) words is generally higher than the WRR on all the words (WRR) in the validation set. However, the sorting order of the recognition rates may change if we consider all words against only Human-Readable (HR) words. Selecting users based on the worst WRR can have an adverse effect on training the model, since we cannot expect the model to perform well on non-readable samples with spelling mistakes and genuine n-gram confusion errors that occur due to awful handwriting (refer Fig. 4). Based on WRR-HRR, users with IDs $\{5,6,7,10\}$ are selected for training the Iter2CL model.

The results of model trained on the selected users (with IDs $\{5,6,7,10\}$) are shown in the third row of Table 3. As shown, the user selection based on WRR-HR performs best as compared to the other two variants (rows $1-2$ of Table 3). Iter2CLmS1 involves swapping the best performing selected user’s (i.e., user $7$) training set with the overall best performing user’s (i.e., user $8$) training set. Iter2CLmS1 achieves slightly degraded performance with a drop of around $1\%$ in CRR and $2\%$ in WRR compared to Iter2CLm as shown in the last two rows of Table 3. As the first row of the table depicts, the performance drops further in similar proportions if we swap the training data from users $\{5,7\}$, with training data from users $\{8,9\}$.

Results on different Iterations: Since the users with IDs $\{8,9\}$ were not selected in the second iteration, we reconsider them in the third iteration. Interestingly, as shown in Table 4, the performance of Iter2CLm model on (unseen) user $8$ validation set is retained compared to Iter1CL model (Table 3 row 4, and the performance on (unseen) user $9$ data improve significantly by $9\%$ (compare row 5 of Table 3 with row 2 of Table 4). Another interesting highlight of Table 4) is that WRR-HR of Iter2CLm model on all the unseen validation sets from users with IDs $\{8,9,11,12,13,14,15\}$ is close to or above $80\%$. This shows that Iter2CLm has generalized well. Based on WRR-HR, we select users with IDs $\{11,12,13,14\}$ for training the model in iteration 3.

The first three rows of Table 5 reestablish that Iter2CLm has generalized well over the overall test set. A similar trend is observed on the seen test set from the users with ids $\{1,2,3,4\}$ who are common seen users across the three iterations, and the unseen test set from users with ids $\{8,9,15\}$ who are common unseen users across the three iterations. The CRR and WRR improve significantly by around $2-3\%$ and $6-8\%$ compared to the iteration $1$ model for the overall test set and standard unseen test set as shown in rows 1-2, columns 2, 3, 6, and 7 of Table 5. However, as shown in rows 1-2 and columns 4-5 of the table, the performance gains are slightly lesser (around $0\%$ in CRR and $3\%$ in WRR) on the commonly seen test set. The slight performance gains of Iter3CLm over Iter2CLm by $<1\%$ as shown in the last row of Table 5 on all the three types of test sets, show that we have reached close to the saturation performance on the test set of users $1-15$. The first three rows in Fig. 6 for qualitative results on the unseen dataset from users $\{8,9,15\}$ also show that the performance of Iter2CLm and Iter3CLm is similar. One of the rare words corrected in the third iteration and a failure case is shown in the last two rows of Fig. 6. To get a rough idea, one can also compare the performance of Iter3CLm with the CRR and WRR of $93.98$ and $75.41$ reported by Gongidi et al. (Gongidi and Jawahar, 2021), wherein the model has seen the writing style of all the users in the test set. Overall, one may argue that the writing style of common unseen users in our case is similar to the seen users, and that is the reason that the accuracy of the unseen test set is high. However, we reason that the high recognition rates on the unseen test set are due to the user selection process described in the previous sections. The selection process takes care that the users with high validation accuracies are less likely to improve the model’s generic performance. Hence such users do not contribute more to the training data in subsequent iterations and are filtered out as common unseen users.

We proposed a controlled framework for upgrading the handwritten ocr model on the dataset provided by $15$ users. We upgraded the model three times with a fixed budget of $23$ pages, each from four different users in each iteration. Our work lays the foundation for long-term ocr services in crowd-sourcing user scenarios. We believe that our experiments have shown the effectiveness of Curriculum Learning (cl) in regular upgradation of the model and user selection under fixed budgets. We finally provide a way to quantify the robustness of ocr models using the data with writing style from unseen users. In the future, we would like to explore our work in a real crowd-sourcing scenario in multiple Indian languages, along with numerous factors and a model for user selection.