A Lexicon and Depth-wise Separable Convolution Based Handwritten Text Recognition System

Handwritten Text Recognition (HTR) is a complex and widely studied computer vision problem in research community. In HTR, cursive strokes of handwritten text need to be recognized. Available text can be either in online or offline form [1]. In online HTR, the time ordered sequence of pen tip is captured. While in offline HTR, static images of handwritten text is available. In this work, we have focused on offline HTR. Handwriting recognition, specially offline HTR systems pose challenges, such as, variability of strokes not only varying among writers but also in a single writer as well, poor and degraded quality of available document images, slop and slant present in text, variable inter (in between lines) and intra (in between characters) spaces among the letters that need to be recognised and limited availability of labelled dataset needed for training of HTR model. Modern deep learning based techniques are used to solve this complex task.

Initially, Hidden Markov Models (HMM) based techniques are used to solve HTR problems. In this technique, a text image is pre-processed using various computer vision techniques and hand crafted features as aspect ratios of individual characters are manually extracted from an image and fed to HMM based classifiers for recognition. Due to HMMs limited capacity of extracting contextual information and manual feature selection, recognition results are poor. In last few decades, deep learning based methods are primarily used for this task. Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Convolutional Recurrent Neural Networks (CRNN), Gated-CNN, Multi-Dimensional Long Short-Term Memory Recurrent Neural Networks (MDLSTM-RNNs) are some state-of-the-art machine learning techniques used to solve HTR problem. Convolution based techniques used to extract fine features of input image and RNN based techniques provides memory to remember long character sequences. Connectionist Temporal Classification (CTC) is widely used to train and test these Neural Network (NN) based systems in an end-to-end manner. Since, the emergence of artificial intelligence and machine learning based devices in day-to-day life, latest research trends in HTR domain are favoured for a robust, less complex system with less number of trainable parameters and acceptable accuracy. In this work, we have proposed a HTR model that is favoured in this direction as well. We have applied depth-wise separable convolutional operation in place of standard convolution to reduce number of trainable parameters. A combination of convolutions, depth-wise convolutions and Full-Gated convolutions along with BGRU units are used to recognised text lines of benchmarked datasets in an end-to-end manner. Following are the key contributions of our present study,

• •

  A novel end-to-end HTR system is given to recognize text lines.

• •

  Proposed model is able to improve recognition accuracy with less number of trainable parameters (Number of training parameters 820,778).

• •

  Recognition text is lexically confined with the help of Word Beam Search (WBS) [2] decoder.

• •

  Overall pipeline of HTR system including essential steps are presented for better understanding of HTR system.

• •

  We have achieved 3.84% Character Error Rate (CER) and 9.40% Word Error Rate (WER) on IAM dataset and 4.88% CER and 14.56% WER in George Washington (GW) dataset, respectively. These state-of-the-art results achieved using lesser training parameters.

Extensive experiments are performed on benchmarked datasets such as IAM. The rest of the paper is organised as follows, section 2 include the key contributions in text line recognition. Section 3 demonstrates proposed architecture. Extensive experiments are presented in section 4. Results obtained reported in section 5 and section 6 include the conclusion of the present study.

In this section, previous works in the domain of HTR has been discussed. We have focused upon text line HTR recognition task. This section present the genesis of HTR that helps readers to understand evaluation of significant HTR techniques. At start, researchers tackle HTR task by using Dynamic Programming (DP) based approaches on word level images, where optimum path finding based algorithm is used. In DP based approaches although, character accuracy improves but this does not guarantee the improvement in the overall word accuracy [3, 4]. Later, HMM based techniques used in HTR [5]. Combination of N-Gram language models and optical recognition using HMM with Gaussian mixture emission probability (HMM-GMM) is one of the primarily studied HTR technique [6, 7]. Optical models of HMMs are further improved by using Multi Layer perceptron (MLP) as emission probability [8, 9] and discriminative training techniques in HMM-GMMs [10]. The MLP is constrained by fixed length input, thus LSTM-HMM based techniques are used in HTR system [11, 12]. The HMMs have drawbacks such as, probability of each observation depends only on current state, therefore, limited contextual information is less utilised. Apart from that HMMs are generative in nature while discriminative performed better in classification and labelling tasks [13].

As a alternative to HMMs, Recurrent Neural Networks (RNNs) does not suffer from these limitations. A model which is having only RNN, it works on character level only because objective function of NN require separate training signal at each point of input signal [14]. The Connectionist Temporal Classification (CTC) is a RNN output layer used in sequence labelling task [15]. Models trained with the help of CTC do not require pre segmented data and able to provide probability distribution of label sequence. The variation of LSTM such as BLSTM and Multidimensional LSTM (MDLSTM) are used with CTC to give state-of-the art recognition results[13, 16, 17, 18, 19]. The LSTM and BLSTM work in one dimensional sequences and MDLSTM captures long term dependencies across the both directions, but it is highly computationally expensive. Later, it was shown that similar results can be acquired by stacking few more BLSTM layers [20]. One more popular architecture, that is, Convolutional Recurrent Neural Network (CRNN) is first introduced in scene-text recognition task [21]. It consist of stack of convolutional layers followed by one or more layers of BLSTM and softmax output layers, which include occurrence probability of N (numbers of characters in dataset) +1 (CTC blank) tokens [22]. A compact and fast model compare to [20] was proposed using Gated CNN architecture [23]. A combination of both the model is proposed by applying Gated-CNN in [20] model and able to get promising results on benchmarked datasets [24]. In one such study, attention based techniques clubbed with CNN and BLSTM are also used by researchers to recognize cursive text [25]. Similarly, a attention based end-to-end system with internal segmentation for paragraph recognition system is studied [26]. Table 1 summarizes all the related work and their techniques.

In this section, we have presented about the basic building blocks of the system proposed. The Figure 1 shows the

detailed view of our proposed architecture. Here, a text line image $I$ is given as input. The preprocessing is performed to reduce the noise in the image as well as for improving accuracy. This image is processed by series of convolutional, gated convolutional and depthwise separable convolutional layers to extract features from it. The extracted features are propagated by series of by gated recurrent layers. The output of recurrent unit is processed by dense layer and character probability is obtained by applying the softmax as a last layer. The CTC is used to calculate the loss during training of the model. The WBS decoder is used in testing of the model. This model is similar at design level to [24], except for the middle two depthwise convolutional layer, which helps in reducing total number of trainable parameters. Overview of the components of the system has been discussed in followoing subsections,

In this section, the experimental setup of the present work has been presented. The recognition model and WBS decoder have been taken from [24]¹¹1https://github.com/arthurflor23/handwritten-text-recognition and [2]²²2https://github.com/githubharald/CTCWordBeamSearch respectively. A HTR system is a unique combination of many hyperparameters that needs to be looked upon while designing it and these parameters might be unique to that system only such as the minimum number of image size of input line image(1024x128x1 (Height $\times$ Width $\times$ Channel)), the number of convolution layers, kernel and filter size of each layer, number and position of max-pooling layer with suitable kernel, the total number of recurrent layer, choice of type of recurrent layer (LSTM/BLSTM/MDLSTM/BGRU) and number of unit it should contain, choice of activation function to have non linearity, size of batches, total number of epochs, rate and place of dropout (it is used to regularize the network and reduce overfitting [17]), learning rate, choice of data augmentation while training, stopping criteria of training and choice of optimizer used for training.

Here, we have discussed various changes in the layers while studying this model. While doing variations, we did not change the settings of the configuration of gated convolutional layer and recurrent layer. We only varied the positions of the convolutional and depthwise separable convolution layer, thus we study 6 different variants of proposed architecture. The table C represents standard convolution and D represents depth wise separable convolution. Figure 4 represents the training and validation loss curve of each of model variants presented in table 5. It is evident from the figure 4 that variation in the number of depthwise separable convolutional layers does not effect much in the model performance.

For statistical analysis, since each iteration of NN training is independent of each other, we have executed thirty training iterations of each variation of table 5 on the IAM dataset and used one-way Analysis Of Variance (ANOVA) test [43] with 5 % significance. As null hypothesis, we considered $H_{0}:\mu_{1}=\mu_{2}=\mu_{3}=\mu_{4}=\mu_{5}=\mu_{6}$ and alternative hypothesis $H_{1}:$ at least one of the $\mu_{i}$ (where $i=1$ to 6) is different. In this, $\mu_{1},\mu_{2},\mu_{3},\mu_{4},\mu_{5},\mu_{6}$ are the different rows of table 5. We analyse for both CER and WER. The obtained p-value is 0.68614, which is greater than 0.05; thus, we failed to reject the null hypothesis.

We have presented a text line handwritten text recognition model using state-of-the-art approaches in each module of the HTR system. We have used depthwise convolution layers to reduce the number of parameters for training and obtained results similar to state-of-the-art techniques. Various different type of model architectures are discussed by changing the number and position of depth-wise separable convolution layers. The ANOVA statistical test has been performed on these models to show their performance similarity irrespective of the model architecture. We have also implemented the WBS algorithm at the decoding step while testing, which improves the obtained results.