SGBANet: Semantic GAN and Balanced Attention Network for Arbitrarily Oriented Scene Text Recognition

Scene text recognition aims to decode a character sequence from the scene images. The methods can be categorized into language-free methods and language-based methods. The language-free methods usually utilize convolutional features without consideration of the character dependency, such as segmentation-based methods and CTC-based methods. Segmentation-based methods [33, 50, 42] first segment the character regions and then recognize each character region to form the final character sequence. CTC-based methods [12, 17, 16] first extract visual features through CNN and then train with RNN and CTC loss to find the most possible combination. However, these methods lack linguistic information and thus easily miss one or two characters.

The language-based methods mainly employ the attention mechanism [38]. The encoder-decoder architecture makes use of linguistic information and character dependency. To boost the performance, some methods focus on learning a new feature representation. For example, [1] proposed a contrastive learning algorithm that first divides each feature map into a sequence of individual elements and performs the contrastive loss [15]. Then the learned representation features are fed to the recognizer. Yan et al. [51] proposed a primitive representation learning method that aims to exploit intrinsic representations of scene text images. Others may focus on integrating the rectification module [30, 41, 55, 52] to reconstruct normal images for those irregular images. Then the reconstructed images are fed to the encoder-decoder module for further recognition. However, scene text usually has a variety of shapes and sizes. It is difficult for the rectification module to transform all the irregular text instances into regular ones. Since current attention-based methods suffer from the attention drift problem [8, 27], directly decoding upon the convolutional features or linguistic features will degrade the recognition performance. Inspired by [44] which generates the character center masks to help focus attention on the right position. In this work, we introduce the Balanced Attention Module, which learns a balancing parameter based on the convolutional and semantic features and draws back the drifted attention. Through the method, the attention drift problem can be alleviated to some extent.

With the development of GANs [34, 59, 36, 58], image-to-image translation has achieved great success. Several methods integrate GANs to generate clear text images for scene text images [10]. This inspires the researchers to combine the GAN and recognition network. Thus, recent recognition methods focus on integrating the adversarial learning concept into the recognition network. For example, [57] introduced a gated attention similarity (GAS) unit to adaptively focus on aligning the distribution of the source and target sequence data. Luo et al. [31] introduced the generative adversarial network to generate a simple image without the complex background for each scene text image. However, it costs much computation if we first generate the required text image and then feed them to a recognition network. Since the two networks both contain huge convolution operations that can result in huge computation, our goal is to design a novel Semantic GAN to align the semantic feature distribution where the Semantic Generator Module directly generates the high-level semantic features and the Semantic Discriminator Module distinguishes between the support domain and target domain.

As can be seen in Fig. 2, the overall architecture of the proposed method consists of the CNN Encoder, the Semantic GAN and the Balanced Attention Module. The CNN Encoder is used to extract the basic convolutional features. The Semantic GAN consists of the Semantic Generator Module(SGM) and the Semantic Discriminator Module (SDM). The Semantic Generator Module is applied to directly generate the high-level semantic features that can be easier to recognize and the Semantic Discriminator Module aims to distinguish between the support semantic features and target semantic features. After the adversarial learning, the semantic feature distribution between the source and target images can be aligned. The Balanced Attention Module is designed to draw back the drifted attention by learning a balancing parameter based on the convolutional and semantic features. Then the balancing operation can be performed to get the balanced glimpse vector.

There are two inputs for the whole architecture. One is the support image $\theta$, which represents the simple image containing pure text instance. Another is the target image $\phi$, which contains text instances with complex backgrounds. Firstly, the two images are fed into the CNN Encoder and visual feature map $\theta_{v}$ and $\phi_{v}$ are obtained. Secondly, they are fed to a Bi-LSTM layer and the Semantic Generator Module, respectively, and $\theta_{s}$ and $\phi_{s}$ are obtained. Then they are fed to the Semantic Discriminator Module. The Semantic Generator Module and the Semantic Discriminator Module contribute to generating simple semantic features. For the Semantic Generator Module, it generates the semantic feature that the discriminator can’t distinguish from the support domain and target domain. At the same time, the Semantic Discriminator Module aims to distinguish them correctly. After the training of Semantic GAN, the Semantic Generator Module successfully generates the simple semantic feature for scene text images, which shares the same feature distribution with those of clear images. And $\phi_{v}$ and $\phi_{s}$ together with $\theta_{v}$ and $\theta_{s}$ are fed to the Balanced Attention Module for the final recognition.

Similar to [7], we take ResNet-based structure as the backbone network to extract basic visual features. We remove the CBAM module [47] and assemble an extra down-sampling layer. As a result, the size of $\theta_{v}$ and $\phi_{v}$ is restored to $\frac{H}{8}\times\frac{W}{8}\times C$, where $H$, $W$ and $C$ represent the height, width and channel of the input images, respectively. Since the size of text instances varies and there is no extra image-to-image network, FPN [28] is assembled to make the CNN Encoder capable of extracting different levels of features. The input support image and target image can be fed to a shared CNN Encoder for saving memory and computation cost.

The Semantic GAN consists of a Semantic Generator Module and a Semantic Discriminator Module. The Semantic Generator Module directly generates the semantic features for complex scene text images. A Bi-LSTM module is used to extract the semantic features of clear images. Then the Semantic Discriminator Module distinguishes the semantic features between support and target domain. In this way, the semantic feature distribution between the source and target images can be aligned and thus can be easier to recognize.

Since convolutional features contain positions of characters, we can utilize the convolutional features to correct the attention weights on the semantic features. The Balanced Attention Module is designed to draw back the drifted attention and recognize the character sequence. It works iteratively for $T$ steps, producing a target sequence of length $T$, denoted by ($y_{1}$, $y_{2}$, …, $y_{T}$ ). As can be seen in Fig. 4, at time step $t$, the output $y_{t}$ for target images is defined by:

where $W_{out}$ and $b_{out}$ are trainable parameters and $h_{t}$ is the hidden state at the time step $t$. $h_{t}$ is updated as follows:

where $y_{t-1}$ is the output at time step $t-1$ and $g_{t}$ represents the glimpse vector defined by:

where $\lambda$ is learnable parameters. $g_{v}$ and $g_{s}$ are internal glimpse vectors. They are computed as follows:

where $\alpha_{t}^{v}$ and $\alpha_{t}^{s}$ are attention weights, which are defined by:

where $W_{c}$, $W_{h}$, $W_{v}$, $W_{s}$, $b_{v}$ and $b_{s}$ are trainable parameters and AP represents average pooling on the height of $\phi_{v}$. Thus the size of $AP(\phi_{v})$ is restored into $\frac{W}{8}\times C$ (C is set to 256), which is the same as that of $\phi_{s}$. For the recognition of support images, the $\phi_{v}$ and $\phi_{s}$ can be replaced by $\theta_{v}$ and $\theta_{s}$, and the above operations can be performed in the same way. As can be seen in Fig. 5, the balanced glimpse vector learns an accurate attention weight and guide the recognition of characters.

In the inference stage, since the input image can be a clear image or a complex scene text image, we feed the convolutional feature $\phi_{v}$ to the Bi-LSTM and Semantic Generator Module both. Then the two learned features are fed to the Balanced Attention Module for recognition and two character sequences are produced. Finally, we choose the character sequence with a higher confidence score as the final result and the other one will be discarded.

The objective function $L$ of our proposed SGBANet consists of two parts: the recognition loss $L_{R}$ and the GAN loss $L_{G}$, as defined by:

where $\beta$ is the coefficient used to balance the importance of the recognition network and GAN network. We set $\beta$ to 0 during the pre-training stage and 1 during the fine-tuning stage. The GAN loss is defined by:

where $G(\cdot)$ and $D(\cdot)$ denote the Semantic Generator Module and the Semantic Discriminator Module. $\theta_{s}$ and $\phi_{v}$ are semantic feature and visual convolutional feature of support image and target image, respectively.

As it is hard to train the recognition network combined with GAN, joint training of the recognition network and GAN will easily cause the instability of recognition loss. To make the whole network stable and robust enough, we have tried several loss functions for the GAN network, such as the original GAN loss [13], Wasserstein GAN Loss [2], and Hinge loss [48]. Additionally, we have tried several architectures of the Semantic Generator Module and the Semantic Discriminator Module, such as integrating convolution layers, LSTM layers and FC layers. As a result, we find that the Hinge loss, together with the architecture reported in Section 3.3 and Section 3.3.2, gets a robust training.

We train the proposed method SGBANet with the public available synthetic datasets, $i.e.$ SynthText [14] and MJSynth [19] without fine-tuning on individual scene text datasets and lexicons. We evaluate the performance on the six widely used benchmarks, including three regular text datasets (IIIT5K, ICDAR2013, SVT), and three irregular text datasets (ICDAR2015, SVTP, CUTE80). Details of the datasets are as follows.

IIIT5K [35] contains 3000 test images cropped from natural scene images. Most of the text instances are regular with the horizontal layout.

ICDAR2013 (IC13) [23] has 1015 test images. The dataset only contains horizontal text instances.

Street View Text (SVT) [43] consists of 647 word patches cropped from Google Street View for testing. This dataset contains blur, noise and low-resolution text images.

ICDAR2015 (IC15) [22] contains 2077 test images collected by Google Glasses. Most of the text instances are irregular (oriented, perspective or curved). We discard the vertical text images, which results in 2002 test images.

Street View Text Perspective (SVTP) [37] contains 645 cropped images from side view angle snapshots in Google Street View. Most of the images are perspective distorted.

CUTE80 [39] contains 288 images cropped from high-resolution scene text images. Most of the images contain curved text.

The proposed method is implemented by using PyTorch. All the experiments are conducted on an NVIDIA Tesla V100 GPU with 32 GB memory. In our experiments, all the input images are rescaled to the size of $64\times 256$ with the aspect ratio preserved. The character set contains 64 classes, including 10 digits, 52 case-sensitive letters, the $SOS$ token and the $EOS$ token. The maximum sequence length is set to 32. AdaDelta is chosen as the optimizer and the batch size is set to 185.

The training process of the SGBANet is divided into two stages: the pre-training stage and the fine-tuning stage. In the pre-training stage, as we described in Equation. (11), we set the $\beta$ to 0. Thus, we train the network on the SynthText and MJSynth from scratch for 2 epochs. Since images in MJSynth contain only pure text instances and images in SynthText contain complex backgrounds, images in the MJSynth and SynthText can be considered as support images and target images, respectively. In the fine-tuning stage, $\beta$ is set to 1, and we jointly train the whole network for another 5 epochs.

We evaluate our method on the aforementioned six benchmark datasets and compare it with those state-of-the-art methods. For a fair comparison, all the methods are trained on the SynthText and MJSynth without fine-tuning on individual scene text datasets and no lexicons are used during the inference stage. Table 1 presents the details of the comparison results. It has been shown that our proposed SGBANet achieves the best on four datasets including IIIT5K, ICDAR2013, ICDAR2015 and SVTP, and achieves competitive performance on two datasets including SVT and CUTE80. Note that, for training the Semantic GAN, we have to divide the MJSynth and SynthText into support images and target images, respectively. Only the SynthText dataset is trained on the Semantic Generator Module. As a result, the proposed method doesn’t achieve the best on the SVT and CUTE80. If we compare our method with SSDAN[57], which exploits the domain adaptation network, we can find that there is a large performance gap between the two methods on the regular text images. Qualitative results for text recognition of the existing methods and our proposed method are presented in Fig. 6. The existing methods [3, 4] do not recognize the characters correctly, while the proposed method reports correct recognition results. In addition, We have calculated the average FPS of the proposed and existing methods [3, 4] for all the 8 datasets and the results are 6.76, 6.68 and 7.58 for the methods [3], [4] and SGBANet, respectively. This shows that our method is faster than the existing methods. The key reason is that the existing methods perform rectification before prediction while the proposed method does not.

The proposed method mainly consists of three modules: the Semantic Generator Module, the Semantic Discriminator Module and the Balanced Attention Module. To demonstrate the effectiveness of the proposed method, we will first evaluate the performance of individual components and then make a further discussion on the Balance Attention Module.