From Two to One: A New Scene Text Recognizer with
Visual Language Modeling Network

Scene text recognition (STR) has been a long-term research topic in computer vision [42, 47, 7]. With deep learning becoming the most promising machine learning tool [35, 40, 41, 8, 17, 20], significant progress has been made in the past few years for STR research [28, 24]. In this section, we divide these methods into two categories according to whether linguistic rules are used, namely language-free methods and language-aware methods.

Language-free methods [42, 48, 31, 19] view STR as a visual classification task and mainly rely on the visual information for prediction. CRNN [31] extracts sequential visual features through combined CNN and RNN, then a Connectionist Temporal Classication (CTC) [9] decoder is used to maximize the probability of all the paths for final prediction. Patel et al. [27] automatically generate the custom lexicon for an image to greatly boost the performance of text reading systems. Zhang et al. [48] regard text recognition as a visual matching task. They calculate the similarity map between visual features of input image and the pre-defined alphabet to predict the text sequence. Liao et al. [18] regard the text recognition as a pixel-wise classification task. Similarly, Textscanner [36] further proposes an order map to ensure a more accurate transcription from characters to the word. In general, the language-free methods ignore linguistic rules in the recognition process, which usually fail to recognize images with confused visual cues (e.g. blur, occlusion, etc.).

Language-aware methods [16, 4, 47, 43] try to leverage linguistic rules to assist the recognition process. Lee et al. [15] use RNNs to automatically learn the sequential dynamics in word strings without manually defining N-grams. Aster [32] firstly uses a rectification module before recognition, and then adopts RNNs to model the linguistic information by using the character predicted from the last time step. However, such serial and time-dependent operation in RNN limits the computation efficiency and the performance of semantic reasoning [45]. Thus, SRN [45] proposes a global semantic reasoning module based on transformer units [35] for pure language modeling, which takes the prediction of vision model as input and predicts the relationships among characters to refine the recognition results. Fang et al. [7] design a completely CNN-based architecture for both vision and language modeling. Though these methods achieve promising results on scene text recognition task, the additionally introduced language model will significantly increase the computation cost. Furthermore, it is also difficult to comprehensively consider and effectively fuse the independent visual and linguistic information in the two-step architecture for accurate recognition [7, 46]. Different from previous methods considering the visual and linguistic information in two separate structures, we directly endue the vision model with language ability and propose a VisionLAN to view the two information as a union. Thus, it is possible to enhance the confused visual cues by capturing linguistic information in the visual context.

BERT [5] introduces a cloze task to mask the tokens of input sentence, which is used to learn a robust bi-directional representation based on the context. Following [5], some works use a similar concept to handle the vision-and-language task [34, 1, 22]. ViLBERT [22] uses a two stream model to process visual and textual inputs, and pre-trains their model through the two proxy tasks. Su et al. [34] propose a general structure to fit for most visual-linguistic downstream tasks, which takes both visual and linguistic features as input. As the STR datasets are weakly labeled with word-level annotations, it is difficult to directly implement these masking approaches in STR task. Different from these methods that mask in token or image patch level, in this paper, we propose a Weakly-supervised Complementary Learning to automatically mask the input image in the feature level. Thus, VisionLAN learns linguistic information from a new perspective by guiding the model to make word-level prediction on the case of missing character-wise visual cues.

The pipeline of VisionLAN is shown in Fig. 2. In the training stage, given an input image, the 2D features V are firstly extracted from backbone network. Then, MLM takes extracted features $V$ and character index $P$ as inputs, and generates the position-aware character mask map $Mask_{c}$ through the Weakly-Supervised Complementary Learning. $Mask_{c}$ is used to occlude the character-wise visual messages in V to simulate the case of missing character-wise visual semantics. After that, VRM takes occluded feature map $V_{m}$ as input and makes prediction under the complete word-level supervision. In the testing stage, we remove MLM and only use VRM for prediction.

In order to occlude the character-wise visual cues for the guidance of linguistic learning, we propose a Masked Language-aware Module (MLM) to automatically generate the character-wise mask map with only original word-level annotations.

As shown in Fig. 3, MLM takes visual features V and the character index $P$ as inputs. Character index $P$ $\in[1,N_{w}]$ indicates the index of the occluded character, which is randomly obtained for each input word image with length $N_{w}$. Then the transformer unit [35] is used to improve the feature representation ability. Finally, after integrating with character index information, the character mask map $Mask_{c}$ is obtained through a sigmoid layer, which is used to generate the occluded feature map $V_{m}$ in Fig. 2.

To guide the learning process of $Mask_{c}$, two parallel branches are designed based on the Weakly-supervised Complementary Learning (WCL). WCL aims to guide $Mask_{c}$ to cover more area of the occluded character, which complementarily makes $1-Mask_{c}$ contain more region of other characters. In the first branch, we implement the element-wise product between $V$ and $Mask_{c}$ to generate the feature map $V_{mas}$ containing visual semantics of the occluded character (e.g. character “b” in the word “burns” with character index 1 in Fig. 3). In contrast, the element-wise product between $V$ and $1-Mask_{c}$ in the second branch is used to generate the feature map $V_{rem}$ containing visual semantics of other characters (e.g. string “urns” in the word “burns” in Fig. 3). By doing these, the complementary learning process guides the $Mask_{c}$ to only cover the character at corresponding position without overlapping other characters (shown in Fig. 7). We share the weights of transformer unit and prediction layer (Eq. 1) among two parallel branches for the feature representation enhancement and semantic guidance. $V_{in}\in R^{hw\times c}$ is the feature map and $Att\in R^{hw\times N}$ is the attention map, where $c=512$ is the channel number, $N=25$ is the max time step, $h$ and $w$ are the height and width. $O_{c}$ is positional encoding [35] of character orders. $W_{1}$, $W_{2}$, $W_{3}$ are trainable weights and $t$ is the time step.

$${p_{t}=Att_{t}^{T}V_{in}}$$

(1)

$${Att=Softmax(G(V_{in}))}$$

(2)

$${G(V_{in})=W_{1}tanh(W_{2}O_{c}+W_{3}V_{in})}$$

(3)

Compared with BERT [5], though both approaches mask out the information in a certain time step, the proposed MLM masks the visual features in the 2d spatial space instead of covering token-level information. Furthermore, as STR datasets are weakly labeled, it is difficult to obtain the accurate character-wise pixel-level annotations. Thus, it is impractical to directly implement BERT-based methods [34, 1, 22] into STR task. Based on these, MLM helps the model to learn linguistic information from a new perspective, which can not be replaced by exiting masking approaches.

The supervisions of WCL are automatically obtained by using the original word-level annotation and randomly generated character index (detailed in Sec. 4). Thus, MLM automatically generates accurate character mask map without the need of additional annotations, making it possible for the real application.

Different from previous methods capturing the visual and linguistic information in the two-step architecture, we propose the Visual Reasoning Module (VRM) to model the two information simultaneously in a unified structure. As a pure vision-based structure, VRM aims to reason the word-level prediction from occluded features by using the character-wise information in the visual context.

The details of VRM is shown in Fig. 4, it contains two parts: Visual Semantic Reasoning (VSR) layer and Parallel Prediction (PP) layer. VSR layer consists of $N$ transformer units [35], which are proved to be effective for modeling long-range dependencies in recent computer vision tasks [2, 24]. Specially, position encoding is used to perceive the pixel location information. Different from [45] using transformer units for pure language modeling, the transformer units in the proposed VRM are used for sequence modeling, which will not be influenced by length of the word. Then, the PP layer is designed to predict the characters in parallel, which has identical formulation as Eq. 1.

In order to achieve the language modeling process $y_{i}=f(y_{N},...,y_{i+1},y_{i-1},...,y_{1})$, the reasoning process of the $i^{th}$ character $y_{i}$ needs to purely depend on the information of other characters. As MLM accurately occludes the character information in the training stage, VSR layer is guided to predict the dependencies between visual features of characters to infer the semantics of occluded character. Thus, with the word-level supervision, VSR layer learns to initiatively model the linguistic information in visual context to assist recognition. In the testing stage, VSR layer is able to adaptively consider the linguistic information for visual feature enhancement when the current visual semantics are confused (e.g. occlusion, noise, etc.).

We visualize the feature maps generated from VSR layer in testing to better understand how the learned linguistic information improves the recognition performance. As shown in Fig. 5, VSR layer effectively supplements the semantics of occluded character “r” in the word “better”, and correctly highlights the discriminating visual cues of character “t” in the word “trans” with the help of linguistic information in visual context. Without the initiative linguistic learning guided by MLM, VRM wrongly predicts the input images as “bettep” and “rrans”.

The final objective function of the proposed method is formulated in Eq. 4. $L_{rec}$ is loss in VRM, and $L_{mas}$ & $L_{rem}$ are losses for predicting masked character and other characters in MLM respectively. $\lambda_{1}$ and $\lambda_{2}$ are used to balance the losses. Specially, we set $\lambda_{1}=\lambda_{2}=0.5$, and use cross-entropy loss formulated in Eq. 5 for $L_{rec}$, $L_{mas}$ and $L_{rem}$. $p_{t}$ and $g_{t}$ represent the prediction and ground truth. We set N to 25 in our experiments.

$${L=L_{rec}+\lambda_{1}L_{mas}+\lambda_{2}L_{rem}}$$

(4)

$${L_{*}=-\frac{1}{N}\sum_{t=1}^{N}\log(p_{t}|g_{t})}$$

(5)

We conduct experiments following the setup of [45] in the purpose of fair comparison. The training datasets are SynthText (ST) [10] and SynthText90K (90K) [12]. The performance is evaluated on 6 benchmarks containing IIIT 5K-Words (IIIT5K) [25], ICDAR2013 (IC13) [14], ICDAR2015 (IC15) [13], Street View Text (SVT) [37], Street View Text-Perspective (SVTP) [29] and CUTE80 (CT) [30]. Details of above 6 datasets can be found in previous works [45, 28].

In addition, we provide a new Occlusion Scene Text (OST) dataset to reflect the ability for recognizing cases with missing visual cues. This dataset is collected from 6 benchmarks (IC13, IC15, IIIT5K, SVT, SVTP and CT) containing 4832 images. Images in this dataset are manually occluded in weak or heavy degree (shown in Fig. 6). Weak and heavy degrees mean that we occlude the character using one or two lines. For each image, we randomly choose one degree to only cover one character. More examples of OST are shown in the supplementary materials.

We use the ResNet45 [32, 38, 28] as our backbone. Particularly, we set the stride to 2 in stage 2,3,4 and initialize the weights by default. Following the most recent works [45, 28], we set the image size to $256\times 64$ (there is no obvious difference with the size of $128\times 32$ in our experiments). Data augmentation including random rotation, color jittering and perspective distortion. We conduct the experiments on 4 NVIDIA V100 GPUs with batch size 384. The network is trained end-to-end using Adam optimizer with learning rate 1e-4. The recognition covers 37 characters including a-z, 0-9, and an end-of-sequence symbol.

Following [45], we divide the training process into 2 steps: language-free (LF) step and language-aware (LA) step. It is worth mentioning that we control the total number of training sessions to be consistent with existing methods for fair comparison. 1) In LF step, we split the connection between MLM and VRM ($V=V_{m}$ in Fig. 2) to guarantee a more stable learning process of both modules. VRM in this step will not acquire the language capability and only uses visual texture for prediction. 2) In LA step, $Mask_{c}$ generated from MLM is used to occlude the feature map $V$ to guide the learning of linguistic rules in VRM. Specifically, we control the ratio of occluded number in a batch, which aims to balance the cases with rich or weak visual information during the training stage.

As all the training images have word-level annotations, we randomly generate the character index based on the length of word, and use this index and the original word-level annotation to generate the labels for MLM (e.g. when index is 4 and word is “house”, the labels are “s” and “houe” respectively). The label generating process is automatic without manual intervention, making it easy to finetune our model on other datasets.

We illustrate the effectiveness of proposed modules in this section. To be specific, baseline contains VRM with two transformer units in Tab. 1& 2& 3.

We compare our method with previous state-of-the-art methods on 6 benchmarks in Tab. 5. We simply divide the methods into language-free and language-aware methods according to whether linguistic information are used. The language-aware methods perform better than language-free methods in general. Benefiting from adaptively considering the linguistic information for feature enhancement, the proposed VisionLAN achieves state-of-the-art performance across the 6 public datasets compared with both language-free and language-aware methods. Specifically, for regular datasets, the proposed VisionLAN obtains 1%, 0.2% and 0.2% improvement on IIIT5K, IC13 and SVT datasets respectively. For irregular datasets, the increases are 1%, 0.9% and 0.7% on IC15, SVTP and CT respectively.

As VisionLAN adaptively considers the visual and linguistic information in the 2d visual space, our method is less sensitive to the distorted images. Thus, the proposed method can obtain better results than ASTER [32] and ESIR [47] on irregular datasets, which adopt the rectification process before recognition. As shown in Tab. 5, the increases are 7.6%, 7.5% and 9% for [32], and 6.8%, 6.4% and 5.2% for [47] on IC15, SVTP and CT datasets respectively.

We further compare the differences between the existing methods and ours in recognition speed and the extra introduced parameters (EIPs) for capturing linguistic information in Tab. 6. In terms of approaching speed and parameters, we implement one transformer unit in GSRM of [45] (the same goes for Sec. 4.5). As the linguistic information is acquired along with the visual features without the need of extra language model, the proposed VisionLAN significantly improves the speed by at least 39% (11.5ms vs 19ms and 43.2ms) without introducing extra parameters (0M vs 12.6M and 3M). Furthermore, as VisionLAN directly considers the linguistic information in the visual space, its efficiency of capturing linguistic information will not be affected by the word length.

To evaluate the language capability of our VisionLAN in detail, we compare our method with recent most popular language models (RNN [32] and Transformer [45]) on OST dataset to evaluate their performance on the case of missing character-wise visual cues. Specifically, we connect these language models to VRM following the implementation details in their papers. As shown in Tab. 7, though the linguistic information captured by [32] and [45] can assist the prediction of vision model, the proposed VisionLAN significantly outperforms these methods by viewing the visual and linguistic information as a union. Through adaptively aggregating the two information in a unified structure instead of considering them independently, VisionLAN improves the baseline model by 7.3% in average.

We evaluate VisionLAN on non-Latin Long Text (TRW15 [49]) to prove its generalization ability. This dataset contains 2997 cropped images, and we set the max length $N$ to 50. We train the proposed VisionLAN following the setup of [45]. As shown in Tab. 8, compared with language-free (CTC) and language-aware (2D-Attention) methods, VisionLAN outperforms these approaches by at least 14.9%. Benefiting from viewing the visual and linguistic information as a union, the proposed VisionLAN achieves a new state-of-the-art result and significantly outperforms SRN [45] by 3.2%. More experiments on other datasets (e.g. MLT [26], etc.) are available in the supplementaries.