Utilizing Resource-Rich Language Datasets for End-to-End
Scene Text Recognition in Resource-Poor Languages

In the encoder, input image $I$ is converted into continuous vectors ${\mbox{\boldmath$Q$}}=\{{\mbox{\boldmath$q$}}_{1},\cdots,{\mbox{\boldmath$q$}}_{J}\}$ as

where ${\rm CNNFeatureExtractor}()$ is a function that extracts image features by using a CNN-based model, ${\rm Reshape}()$ is a function that translates three-dimensional image features (${\rm width}\times{\rm height}\times{\rm channels}$) into two-dimensional vectors (${\rm width}\times({\rm height}\times{\rm channels})$), and $J$ is the width of the continuous vectors. The encoder proposed by Wang et al. (Wang et al., 2019) outputs continuous vectors $Q$, see Figure 1 (b). On the other hand, in the encoder proposed by Sheng et al. (Sheng et al., 2019) shown in Figure 1 (a), continuous vectors $Q$ are then projected into ${\mbox{\boldmath$R$}}=\{{\mbox{\boldmath$r$}}_{1},\cdots,{\mbox{\boldmath$r$}}_{J}\}$ for input to the Transformer encoder block as

where ${\rm AddPosEnc}()$ is a function that adds a continuous vector in which position information is embedded. The Transformer encoder composes continuous vectors ${\mbox{\boldmath$S$}}^{(K)}$ from $R$ by using $K$ Transformer encoder blocks. The $k$-th Transformer encoder block forms the $k$-th continuous vectors ${\mbox{\boldmath$S$}}^{(k)}$ from the lower layer inputs ${\mbox{\boldmath$S$}}^{(k-1)}$ as

where ${\mbox{\boldmath$S$}}^{(0)}={\mbox{\boldmath$R$}}$, and ${\rm TransformerEnc}()$ is a Transformer encoder block that consists of a scaled dot product multi-head self-attention layer and a position-wise feed-forward network (Vaswani et al., 2017). The encoder proposed by Sheng et al. (Sheng et al., 2019) outputs continuous vectors ${\mbox{\boldmath$S$}}^{(K)}$.

The decoder computes the generation probability of a character string from the preceding character string and continuous vectors output from the encoder. The predicted probabilities of the $t$-th character, $c_{t}$, are calculated as

where ${\rm Softmax}()$ is a softmax layer with linear transformation. The Transformer decoder forms hidden representation ${\mbox{\boldmath$u$}}_{t-1}^{(L)}$ from encoder output $V$ by using $L$ Transformer decoder blocks, where

The $l$-th Transformer decoder block forms the $l$-th hidden representation ${\mbox{\boldmath$u$}}_{t-1}^{(l)}$ from the lower layer inputs ${\mbox{\boldmath$U$}}_{1:t-1}^{(l-1)}=\{{\mbox{\boldmath$u$}}_{1}^{(l-1)},\cdots,{\mbox{\boldmath$u$}}_{t-1}^{(l-1)}\}$ as

where ${\rm TransformerDec}()$ is a Transformer decoder block that consists of a scaled dot product multi-head self-attention layer, a scaled dot product multi-head source-target attention layer, and a position-wise feed-forward network (Vaswani et al., 2017). The hidden representation ${\mbox{\boldmath$u$}}_{t-1}^{(0)}$ is given by

where ${\rm Embedding}()$ is a linear layer that embeds the input character into a continuous vector.

To train the model, the target language’s dataset $\mathcal{D}=\{({\mbox{\boldmath$I$}}_{1},{\mbox{\boldmath$C$}}_{1}),\cdots,\\ ({\mbox{\boldmath$I$}}_{N},{\mbox{\boldmath$C$}}_{N})\}$ is used, where $N$ is the number of data points. The optimization of model parameters is represented as

where $\hat{{\mbox{\boldmath$\theta$}}}_{\rm enc}$ and $\hat{{\mbox{\boldmath$\theta$}}}_{\rm dec}$ are trained parameters.

In this paper, to train an accurate scene text recognition model for the target language when a large dataset in the target language is not available, i.e. the number of data points, $N$, for the target language is small, we propose a training method that utilize not only the dataset in the resource-poor target languages but also a well-prepared dataset of a resource-rich language.

In the pre-training phase, the proposed method uses not only the resource-poor language’s dataset $\mathcal{D}^{\rm rp}=\{({\mbox{\boldmath$I$}}^{\rm rp}_{1},{\mbox{\boldmath$C$}}^{\rm rp}_{1}),\cdots,({\mbox{\boldmath$I$}}^{\rm rp}_{N},{\mbox{\boldmath$C$}}^{\rm rp}_{N})\}$, but also a well-prepared large dataset in a resource-rich language $\mathcal{D}^{\rm rr}=\{({\mbox{\boldmath$I$}}^{\rm rr}_{1},{\mbox{\boldmath$C$}}^{\rm rr}_{1}),\cdots,({\mbox{\boldmath$I$}}^{\rm rr}_{M},{\mbox{\boldmath$C$}}^{\rm rr}_{M})\}$, where $N$ and $M$ are the number of data points in the resource-poor language’s dataset and the resource-rich language’s dataset, respectively. We assume $N<M$. The proposed method pre-trains the encoder and the decoder by using the multilingual dataset and the resource-poor language’s dataset, respectively.

As for encoder pre-training, the multilingual model that consists of multilingual encoder (ME) and multilingual decoder (MD) is trained. Training uses a multilingual dataset made by combining the resource-poor language’s dataset $\mathcal{D}^{\rm rp}$ with the resource-rich language’s dataset $\mathcal{D}^{\rm rr}$. The optimization of model parameters is given by

where $\dot{{\mbox{\boldmath$\theta$}}}_{\rm enc}$ and $\dot{{\mbox{\boldmath$\theta$}}}_{\rm dec}$ are the trained parameters of ME and MD in encoder pre-training.

As for decoder pre-training, the resource-poor language model that consists of resource-poor language encoder (RPLE) and resource-poor language decoder (RPLD) is trained. Training uses the resource-poor language’s dataset $\mathcal{D}^{\rm rp}$. The optimization of model parameters is given by

where $\ddot{{\mbox{\boldmath$\theta$}}}_{\rm enc}$ and $\ddot{{\mbox{\boldmath$\theta$}}}_{\rm dec}$ are the trained parameters RPLE and RPLD in decoder pre-training.

In the fine-tuning phase, the proposed method trains the final recognition model using the pre-trained encoder-decoder parameters. Thus, parameters of ME $\dot{{\mbox{\boldmath$\theta$}}}_{\rm enc}$ pre-trained by encoder pre-training and RPLD $\ddot{{\mbox{\boldmath$\theta$}}}_{\rm dec}$ pre-trained by decoder pre-training are used as the initial values for fine-tuning. Fine-tuning is carried out by using the resource-poor language’s dataset $\mathcal{D}^{\rm rp}$. The optimization of model parameters is given by

where $\hat{\dot{{\mbox{\boldmath$\theta$}}}}_{\rm enc}$ and $\hat{\ddot{{\mbox{\boldmath$\theta$}}}}_{\rm dec}$ are the fine-tuned final parameters.

In the experiments in Section 4, we additionally examined a training procedure that fine-tuned the parameters of ME $\dot{{\mbox{\boldmath$\theta$}}}_{\rm enc}$ with a randomly initialized decoder, and a training procedure that fine-tuned the parameters of ME $\dot{{\mbox{\boldmath$\theta$}}}_{\rm enc}$ and MD $\dot{{\mbox{\boldmath$\theta$}}}_{\rm dec}$.

We used the Japanese dataset created for the ICDAR2019 robust reading challenge on multilingual scene text detection and recognition (Nayef et al., 2019); it is the only publicly available Japanese scene text dataset, for the resource-poor language’s dataset. The ICDAR2019 dataset holds annotated real and synthesized image data created using the synthesizing method of (Gupta et al., 2016) for end-to-end scene text detection and recognition in 10 languages. To construct the Japanese scene text recognition dataset for this experiment, we first selected Japanese real image data and synthesized image data. Then, we cropped the images according to the annotation for scene text detection, and excluded data that contained characters other than standard Japanese characters. This yielded 9,346 real images and 65,452 synthesized images. Mixing and splitting these images at the ratio of $9:1$ yielded training data of 67,368 images and test data of 7,430 images. The training data was used as $\mathcal{D}^{\rm rp}$ for both pre-training and fine-tuning. There were 2,332 character classes.

We used MJSynth (Jaderberg et al., 2014) as the resource-rich language’s (i.e. English) dataset, $\mathcal{D}^{\rm rr}$. The number of training data images was 8,027,346.

We tested the following four training procedures to evaluate the proposed training method. Baseline did not pre-train the recognition model; training used only the Japanese dataset from scratch. In Training w/ ME, we pre-trained ME by using the multilingual dataset made by combining the Japanese dataset and the English dataset, and fine-tuned ME with a randomly initialized decoder by using the Japanese dataset. In Training w/ ME+MD, we pre-trained ME and MD by using the multilingual dataset, and fine-tuned them by using the Japanese dataset. In Proposed training w/ ME+RPLD, we pre-trained ME and RPLD by using the multilingual dataset and the Japanese dataset respectively, and fine-tuned them by using the Japanese dataset.

Two recognition models based on Transformer were evaluated. The first one is the model proposed by Sheng et al. (Sheng et al., 2019); VGG16 (Simonyan and Zisserman, 2015) up to the 10-th convolution layer was applied as the CNN feature extractor. The second one is the model proposed by Wang et al. (Wang et al., 2019); ResNet34 (He et al., 2016) was applied as the CNN feature extractor. The Transformer blocks were composed under the following conditions: the number of Transformer encoder blocks and Transformer decoder blocks were set to 1, the dimensions of the output continuous representations and the inner outputs in the position-wise feed forward networks were set to 512, and the number of heads in the multi-head attentions was set to 4. We also evaluated the models based on RNNs (Shi et al., 2016a; Liu et al., 2016; Wang and Hu, 2017; Borisyuk et al., 2018; Shi et al., 2016b; Lee and Osindero, 2016; Baek et al., 2019). For the models based on RNNs, we evaluated only the baseline training procedure, and the model structures followed the evaluation by (Baek et al., 2019).

For all evaluated models, the input image size was $400\times 64$, and the input images were scaled to satisfy the resolution. For training, the optimizer was based on stochastic gradient descent (SGD) with learning rate of 0.01. The mini-batch size was set to 32 images. Note that a part of the training data was used for early stopping. As an evaluation metric, we used the recognition accuracy as determined by exact matching.

The recognition accuracy values are shown in Table 1. First, the recognition models based on Transformer have the same or higher Japanese character recognition accuracy than the recognition model based on RNNs with limited training data. Next, for models based on Transformer, the accuracy obtained by baseline was low compared to the use of pre-training, and utilizing pre-training improved the accuracy. In detail, when we utilized training with ME or ME+MD, the performance improvement was small. On the other hand, our training proposal utilizing ME+RPLD further improved the recognition accuracy. Examples of characters extracted by the model of Wang et al. (Wang et al., 2019) are shown in Figures 3 and 4. Note that the captions for each image on Figures 3 and 4 are (a) ground truth, recognition results of (b) baseline, (c) training w/ ME, (d) training w/ ME+MD, and (e) proposed training w/ ME+RPLD. The results in Figure 3 show that the proposed training method can prevent the erroneous recognition of words in tilted or blurred images; this is mainly the effect of utilizing the multilingual dataset in pre-training the encoder. The results in Figure 4 show that the proposed training method can prevent the false recognition of words that do not exist given relatively long character strings, which is mainly due to pre-training the decoder by utilizing the resource-poor language’s dataset. These results confirm that the proposed training method is an effective way of improving recognition accuracy when the image-to-text paired dataset in the resource-poor language is limited.

We also evaluated the recognition accuracy while varying the size of the English dataset in the pre-training phase. We prepared English datasets of four sizes: 8M as used in the above evaluation, 4M (4,013,673 data), 2M (2,006,837 data), and 1M (1,076,675 data). The resulting accuracy values are shown in Table 2. They show that increasing the data size improves the recognition accuracy. On the other hand, the proposed method is more effective than baseline even when the size of the English dataset is small.