Optimal Boxes: Boosting End-to-End Scene Text Recognition by Adjusting Annotated Bounding Boxes via Reinforcement Learning

In two-step systems, due to the fact that detected texts are cropped from the image, the detection and recognition are two separate steps. Some of these methods first generate text proposals using a text detection model [45, 24, 40] and then recognize them with a text recognition model [13, 23, 9]. Jaderberg et al. [13] use a combination of Edge Box proposals [46] and a trained aggregate channel features detector [7] to generate candidate text bounding boxes. Liao et al. [23] combine an SSD [26] based text detector and CRNN [37] to spot text in images. In addition, for the detection step, EAST [45] further simplifies the anchor-based detection by adopting the U-shaped design [35] to integrate features from different levels. And for the recognition step, RARE [38] consists of a Spatial Transformer Network (STN) and a Sequence Recognition Network (SRN), which is robust to irregular text. One major disadvantage of two-step methods is that the propagation of error between the recognition and detection models will result in less satisfactory performance.

Many end-to-end trainable networks have recently been proposed [3, 4, 20, 11, 27]. Bartz et al. [3] present a solution that employs a STN [14] to attend to each word in the input image circularly and then recognize them individually. Li et al. [20] substitute the object classification module in Faster-RCNN [34] with an encoder-decoder-based text recognition model and to create their text spotting system [4, 11, 27] develop unified text detection and recognition systems with very similar overall architectures, which consist of a recognition branch and a detection branch. Liu et al. [28] design a novel BezierAlign layer for extracting accurate convolution features of a text instance with arbitrary shapes and adaptively fit arbitrarily-shaped text via a parameterized Bezier curve. Liao et al. [22] propose Mask Text Spotter v3, an end-to-end trainable scene text spotter that adopts a Segmentation Proposal Network (SPN) instead of an RPN [34].

In earlier work, Mnih et al. [30] present the first deep learning model to successfully learn control policies directly from high-dimensional sensory input using reinforcement learning. In recent years, reinforcement learning [18, 15, 5, 29] has evolved considerably in the field of object detection. Some works [33, 43] employ reinforcement learning as a post-processing method for scene text detection to adjust the bounding boxes predicted by the detection model, which can result in a significant time increase. In contrast to earlier research, we employ text recognition as the reward rather than the IOU between the predicted and ground-truth bounding boxes. In addition, our method is not a post-processing of the detection and does not add any extra computational costs to the inference phase.

Based on the current state and reward, the agent chooses which action to take from action space. So it is crucial to capture abundant information from the state. However, one observation can only provide limited information for the agent, and it is necessary to make full use of historical observations for making decisions. Thus, we choose four serial observations as the state and the current state can be defined as $s_{t}$={$o_{t-3}$, $o_{t-2}$, $o_{t-1}$, $o_{t}$}, where $o_{t}$ denotes the current observation at step t. A single observation is composed of background, foreground, and $\operatorname{box-coordinates}$, denoted by $o_{t}$={background, $foreground_{t}$, $\operatorname{\textit{box}-\textit{coordinates}}_{t}$}. The background area is four times the size of the initial bounding box. The $foreground_{t}$ is cropped from the background by a minimum enclosing rectangle of the bounding box at step t. The $\operatorname{\textit{box}-\textit{coordinates}}_{t}$ represents the coordinates of text in background at step t. We have $16$ actions in action space which are combinations of 4 vertexes and $4$ directions. As we can see from Fig.2(b), the first action in action spaces implies that the top-left vertex of the quadrangle moves down by one pixel.

The goal of BoxDQN is to capture appropriate bounding boxes for better recognition. Therefore, a reliable reward is needed to guide the agent to automatically adjust the bounding boxes. We select a few representative text recognition algorithms, including CRNN [37], RARE [38] and others. The average recognition confidence among them is regarded as a reward, so the reward at step t can be formulated as $r_{t}=conf_{t+1}-conf_{t}$, where $conf_{t}$ = $Conf(foreground_{t})$, $conf_{t}$ denotes the recognition confidence at step t.

where $N_{P}$ denotes the number of characters in a prediction word and $N_{G}$ denotes the number of characters in a ground-truth word. The aim of reinforcement learning is to maximize the cumulative rewards:

where $\gamma$ denotes the discount factor and $\gamma$ $\in$ [0,1]. Ignoring the discount factor, the cumulative reward is equal to $conf_{T}-conf_{0}$, where $conf_{T}$ refers to the recognition confidence of foreground in the terminal state, T means the maximum number of steps and $conf_{0}$ refers to the recognition confidence of foreground in the initial state. Because $conf_{0}$ is invariant and only determined by the initial bounding box, maximizing cumulative rewards means maximizing $conf_{T}$ without $\gamma$.

With the defined action space, state, and reward, the details of BoxDQN are illustrated in Fig.2(b). The agent is composed of a feature fusion module (FFM) and a transformer encoder [42]. It accepts a single state with four observations as input and outputs 16 dimensional vectors, each of which specifies the appropriate action to take. With two deep convolutional neural networks [19] and a transformer encoder, the FFM is proposed to integrate background, foreground, and box-coordinates. During a bounding box adjustment, BoxDQN receives four observations and outputs the corresponding action according to the current state. Every observation needs to be fused by the FFM successively. The feature maps of the background and foreground are extracted from two convolution neural networks, respectively. We concatenate two image feature maps and the position encoding as the input of the transformer in the FFM. After all four observations are passed through the FFM, the transformer in the agent selects an action from the action space based on the concatenation of four fused feature maps. The bounding box moves in response to the selected action, changing both the $\operatorname{box-coordinates}$ and the minimum enclosing rectangle of the $\operatorname{box-coordinates}$, and then the next state starts.

In many cases, due to the absence of labeled real data, we train and test models using synthetic data. However, the domain gap between synthetic and real data degrades performance on real data. To address domain shift problems, we propose a domain-adaptive approach based on our BoxDQN. As shown in Fig.3, our method consists of four steps: (1) refer to the labeled synthetic data domain as the source domain and the unlabeled real data domain as the target domain, (2) train a text spotter with the labeled synthetic data, (3) use the trained text spotter to generate pseudo-labels on real data and adjust the pseudo-labels by employing the BoxDQN mentioned above, (4) finetune the text spotter with the adjusted bounding boxes on real data.

We use a value-based reinforcement learning method to adjust the bounding boxes, and the training process is presented in Algo.1. During the inner loop of the algorithm, BoxDQN can only adjust one bounding box in a single iteration. Thus, we crop all backgrounds from the source images by the bounding boxes in advance. M is the number of backgrounds. Firstly, the agent selects and executes an action according to an $\operatorname{\epsilon-greedy}$ policy. The $\epsilon$ gradually decreases with iterations from 1.0 to 0.2. Secondly, we present two methods to determine whether the BoxDQN has reached the terminal state. Thirdly, we store a transition {$s_{t}$, $a_{t}$, $r_{t}$, $s_{t+1}$, $Term_{t+1}$} and sample random mini-batch of transitions in replay memory D. The $a_{t}$ represents the action at step t and $Term_{t+1}$ represents the terminal state at step t+1, respectively. $Term_{t+1}$ has two types: 0 and 1, where 0 and 1 represent termination and continuance, respectively. Finally, we refer to the training method in the paper[31] that uses a separate network termed $\hat{Q}$ for generating the targets $y_{j}$ in the $Q$-learning update. $Q$ represents the BoxDQN agent and has the same network structures as $\hat{Q}$. The $\operatorname{\hat{Q}-network}$ parameters $\theta^{-}$ are only updated with the $Q$-network parameters $\theta$ every C steps and are held fixed between individual updates. The parameters of $Q$ are updated by optimizing the loss function with stochastic gradient descent. The training loss function is defined as follows:

where $y_{j}$ can be formulated as follows:

To verify the effectiveness of our method for the end-to-end text spotting methods and the classic two-step methods, we perform experiments on four different datasets. Furthermore, we conduct domain-shift experiments on these datasets to show the robustness of our method in general scenarios.

Our method mainly focuses on adjusting annotated bounding boxes for better text recognition. To verify its effectiveness, we conduct experiments on the four datasets, i.e.IC13, IC15, MLT19 and ReCTS. We show the adjustment results of our method on the bounding boxes of different datasets. The qualitative results of bounding boxes in English and Chinese are represented in Fig.4. We find that our bounding boxes can achieve higher credibility in recognition. The (a-1) to (a-3) in Fig.4 show that our BoxDQN can adjust the bounding boxes to make them more suitable for recognition models. It can also correct inaccurate recognition of text in images. Furthermore, we can learn from Fig.4(b) that the adjustment steps of BoxDQN are a step-by-step process. The incorrectly labelled ”Europcar” is gradually being correctly recognized. It is worth noting that our recognition confidence is increasing at each step.

To verify the robustness of our method with those baseline methods, we use the annotated bounding boxes refined by BoxDQN to train the baseline methods. During the quantitative evaluation, the same dataset with the original annotations is also used for training the baseline methods as a comparison. Finally, we test the F-Score metrics of our adjusted bounding boxes under recognition. The gain in Tab.1 indicates the gain after including our BoxDQN.

Taking into account the domain gap between the synthetic pretraining datasets and the in-the-wild data, we conduct cross-domain experiments to verify the generalization of our method. In detail, we pretrain BoxDQN on the synthetic datasets (SynthText-80k [10] and Synthetic-MLT [32]). After that, we adopt the pre-trained detection models on the relevant real datasets to obtain pseudo bounding boxes. The BoxDQN adjusts the pseudo bounding boxes, and finally the recognition models work on the adjusted bounding boxes to obtain the recognition results. We simulate in-the-wild data through unlabeled IC15 and MLT19, and verify the domain adaptability of our BoxDQN. The results of domain adaption experiments are shown in Tab.3 and Fig.4(a-1,2,3). From Fig.4(a-4,5,6,7), although our method has a slight visual deviation in the adjustment of pseudo-labels, it can improve the confidence and correct the wrong recognition results. Tab.3 proves that our method can improve at least $4.4\%$ in cross-domain datasets.

To further explore the potential of our approach, we perform experiments on arbitrarily shaped text (TotalText [6] dataset). Our BoxDQN method requires a text representation with a fixed number of boundary points for optimization, so we have to convert polygon contour points that do not have a fixed number of points to a representation that does. We can currently only convert arbitrarily-shaped text to a fixed number of control points ($8$) with the help of Bezier curves. We train our BoxDQN on the SynText150k [28] dataset which contains 150k synthetic arbitrary-shaped text annotated by Bezier curves. Then, we use BoxDQN to adjust the control points of the Bezier curve to obtain the optimal ground truth. The rest of the experimental settings is consistent with the multi-oriented text datasets, except for the differences mentioned above. From Tab.7, we can find there is a considerable improvement($61.5$ vs. $63.8$) in recognition performance when training with the ground-truth Bezier curves optimized by BoxDQN. This experiment illustrates the possibility of extending our approach to arbitrarily-shaped text if there is a more general representation of text boxes.