Inverse-like Antagonistic Scene Text Spotting via Reading-Order Estimation and Dynamic Sampling

Traditional deep learning-based text detection methods [13, 14, 15, 16, 17, 18, 19] mainly focus on multi-oriented texts. Anchor-based methods [13] adopt rotated anchors and RRoI pooling for detecting multi-oriented texts. Anchor-free methods [16] directly regress the offsets from boundaries or vertexes to the current point for detecting texts. Some methods [14, 20] try to design a hidden anchor mechanism to integrate the advantages of the anchor-based method into the anchor-free method. Recently, a series of text detection methods have been proposed for detecting irregular text. The Connected Component (CC) based methods [21, 22] usually detect individual text parts or characters first, followed by a link or group post-processing procedure for generating final texts. Segmentation-based methods [23, 17] use instance segmentation to detect arbitrary shape text and design different schemes to separate adjacent text instances. But, segmentation accuracy significantly determines the quality of detected boundaries. Contour-based methods [24, 25, 26, 27, 28] resort to modeling the text boundary for better representation of arbitrarily-shaped texts.

Scene text recognition involves recognizing texts in a cropped image. Traditional methods [29, 30] rely on character-level annotations for character detection, while methods [31, 32] extract features from line-level text using CNN and RNN, and use a CTC-based decoder for prediction alignment. However, these methods are designed for regular text recognition and struggle with irregular text. To address this problem, Shi et al. [33] propose a rectification network with STN for arbitrary shape text, while Litman et al. [34] use TPS and selective attention decoder for visual and contextual features. CharNet [35] uses a Character-Aware Neural Network to detect characters first and then separately transform them into a horizontal one. AON [36] extracts features with four directions and character position clues, while SAD [37] applies a 2D-attention mechanism to catch irregular text features, both achieving impressive results. To address the attention drift issue, RobustScanner [38] design a position enhancement branch in the recognition model. In addition, some methods use semantic segmentation to assist in text recognition.

Traditional text spotting methods [39, 40] perform text detection and recognition as two separate steps. Generally, a text detector extracts regions of interest (RoI), which are then fed into a recognition model. However, recent end-to-end text spotting approaches [41, 42, 43, 8] have confirmed that detection and recognition tasks are highly relevant and complementary to each other. By sharing features and jointly optimizing the modules [41, 44] in a unified end-to-end trainable network, they achieved improved detection and recognition performances simultaneously.

Recently, several methods [2, 45, 43, 6, 1, 46, 10, 27, 47, 48, 49, 50] have been proposed to address arbitrary shape text spotting. Mask TextSpotter [6] and PAN++ [1] use RoI Masking to focus on the arbitrarily shaped text region. MANGO [4] uses a Mask Attention module to retain global features for multiple instances but still requires centerline segmentation to guide the grouping of the predictions. Boundary TextSpotter [45] and TPSNet [9] localize the boundary points of text instances and rectify the features using Thin-Plate-Spline before feeding them into the recognition branch. CRAFTS [46] uses character region maps supervised by character-level annotations to help the attention-based recognizer attend to precise character center points. PGNet [10] transforms the polygonal text boundaries to the centerline, border offset, and direction offset and performs multi-task learning for these objectives. Inspired by Pix2Seq [51], some methods, such as TESTR [47], TTS [52], and SPTS [48], use a network combining CNN and Transformer that tackles text spotting as a sequence prediction task, similar to language modeling. However, these methods usually require extensive computing and data resources.

Although the text spotting methods (e.g., ABCNet [43, 53], TESTR [47] and SwinTextSpotter [54]) have achieved great improvement for arbitrarily shaped text spotting, they still suffer from inverse-like text because of the absence of key reading-order information. However, the reading of inverse-like scene text is important, even though it has just been noticed in DPText [27]. The other problem is that the reading performance of these methods also suffers from the accuracy of detected boundaries because of fixed feature sample. In this paper, we aim to effectively spot inverse-like scene texts without sacrificing general ones. Hence, we design a reading-order estimation module and a dynamic sampling module, which greatly improves the accuracy of inverse-like text spotting without losing generality.

The framework of our method presented in Fig. 3 mainly consists of six components: feature extraction module, init boundary module (IBM), reading-order estimation module (REM), boundary deformation module, dynamic sampling module (DSM), and text recognition module. The feature extraction module extracts features from input images for the text detection and recognition tasks. To preserve spatial resolution and utilize multi-level information, we use a multi-level feature fusion strategy similar to [55]. As noted in [45], the two tasks have different requirements for feature maps. Specifically, the recognition task needs more detailed information for character sequence prediction, while the detection task focuses on the whole text instance. Therefore, we use lightweight convolutions to separate the detection and recognition features.

The initial boundary module generates rough text boundaries to locate text instances. Similar to [25, 28], our module consists of multi-layer dilated convolutions, including two $3\times 3$ convolution layers with different dilation rates and one $3\times 3$ convolution layer, as illustrated in Fig. 4(a). It uses shared detection features to provide prior information for text location. The prior information can be in the form of field information such as classification, distance, and direction in[25, 28], probability maps in DB [56], or text kernels in PSENet [57] and PAN [58]. Using the prior information, we can obtain the text segmentation and generate a coarse text boundary, as shown in Fig. 4(b). The accuracy of these text boundaries is not deterministic, as they are only used as initial information and will be refined in our method.

The reading-order is crucial for accurate understanding and recognition of text sequences. In existing text detection datasets, the annotation of text detection follows the reading direction of human beings, as it aligns the text features suitably for better recognition. However, the implicit learning of reading-order can degrade the detector’s robustness, resulting in false positives and jagged edges, as shown in Fig. 5 (b). Even with extensive training with rotation augmentations, the detector still struggles to learn correct reading-orders, as shown in Fig. 5 (c).

Learning reading-order explicitly is rare in existing methods, except for Text Perceptron [8] and PGNet [10]. Text Perceptron uses order-aware segmentation to indicate the head and tail of text instance, while PGNet extracts the text reading-order using TDO maps, as shown in Fig. 5 (d) and (e). However, both methods only consider the order of information from the head to tail, neglecting the top and bottom. In some cases, this imperfect reading-order fails to reflect the actual case, such as inverse-like text, which may occur in natural images with more complex layouts like mirror, symmetry, or inversion, etc.. In these cases, simply clipping the detection results with an artificial or imperfect reading-order disturbs recognition, as shown in Fig. 1 (a1-a3) or Fig. 5 (b-e).

To fully explore the reading-order information of humans hidden in annotations [27], we design a reading-order estimation module (REM) to accurately learn the text reading-order information. In detection tasks, text instances are typically found in long strips, with two long edges in the text boundary, as noted in [59, 27]. Hence, we adopt four key corners at both ends of the text’s long side to indicate the reading-order, as depicted in Fig. 5 (f). In this work, we tackle the problem of reading-order estimation by treating it as a classification task. Our approach involves a reading-order estimation module (REM) that identifies whether each point on the initial boundary belongs to one of the four key corners. Specifically, we employ a “CirConv” block, a fusion block, and a prediction head (as depicted in Fig. 6 (a)) to construct REM. The “CirConv” block consists of four circular convolution layers with different dilation rates (e.g., [1, 1, 2, 4]) to enhance REM’s information aggregation capability. We use dense shortcut connections across all layers to improve the interaction between each layer [60]. The fusion block uses a 1$\times$1 convolution layer and max pooling to merge information in the “CirConv” block. Then, the deeply fused features are distributed to each initial boundary point by concatenating their features. Finally, three 1$\times$1 convolution layers with ReLU activation and sigmoid generate the classification information ($[r_{0},r_{1},r_{2},r_{3}]$) for each initial boundary point.

To enable batch processing and avoid missing key corners due to a large sampling interval, we sample $2K$ vertices ($P={p_{{}_{0}},...,p{{}_{i}},...p{{}_{{}_{2K-1}}}}$) for each initial text boundary, forming a closed contour (as shown in Fig. 6 (b)-(1)). Circular convolution will encode the cyclicity of points along the closed contour effectively, building on the success of DeepSnake [61]. However, an excellent network structure is not sufficient to learn the correct reading-order, as the input information also plays a decisive role. To ensure REM obtains more reliable information to accurately estimate the reading-order, the input feature $F_{c}(i)$ consists of geometric attributes ($F_{g}(i)$) of vertex $p_{i}$, visual features ($F_{b}(i)$), and order embedding features ($F_{e}(i)$) of the initial boundary.

The geometric attributes ($F_{g}(i)$) of vertex $p_{i}$ includes cosine ($\cos({\theta}_{i})$) of angle, coordinates $(\Delta x_{i},\Delta y_{i})$ relative to the initial boundary centroid $(x_{c},y_{c})$, and distance ($d_{i}$) relative to the $(x_{c},y_{c})$. Thus, the geometric attributes ($F_{g}(i)$) of vertex $p_{i}$ can be formulated as

The $cos{\theta}_{i}$ can be calculated as

where the angle ($\theta$) of vertex $p_{i}$ is defined as the angle between vector $\overrightarrow{p_{{}_{i}}p_{{}_{i+1}}}$ and $\overrightarrow{p_{{}_{i}}p_{{}_{i-1}}}$, as shown in Fig. 6 (b)-(1). The $(x_{c},y_{c})$ is the centroid of the initial boundary, formulated as

After we get the $(x_{c},y_{c})$, the coordinates $(\Delta x_{i},\Delta y_{i})$ and the distance ($d_{i}$) of vertex $p_{i}$ can be calculated as follows:

where $w$ and $h$ are the width and height of the bounding rectangle of the initial boundary ($P=\{p_{{}_{0}},...,p_{{}_{i}},...p_{{}_{2K-1}}\}$). Here, we have obtained three geometric attributes for vertex $p_{i}$, providing important prior information for vertex classification.

We know that text instances in word-level and line-level typically have two long sides and two short sides in their boundaries, and the key points we need to identify are usually located at the intersection of the long and short sides. These points have distinct geometric attributes $F_{g}(i)=[{\theta}{i},d_{i},\Delta x_{i},\Delta y_{i}]$. As shown in Fig. 6 (b)-(1), the angle ${\theta}{i}$ of these key points usually exhibits sudden changes and is significantly smaller than that of other non-key points. The distances $d_{i}$ from these key points to the centroid $(x_{c},y_{c})$ are typically greater than those of other points, although not always the smallest. To improve the fusion of visual ($F_{b}(i)$) and geometric ($F_{g}(i)$) features during module reasoning, we embed the geometric attributes of each vertex $p_{i}$ into a high-dimensional space as [21]. Specifically, we apply sine and cosine functions with varying wavelengths to $F_{g}$. The geometric attribute embedding is calculated as follows:

where $C$ is the dimension of the embedding vector (emprirically set to 36). The geometric attribute $F_{g}(i)$ of vertex $p_{i}$ is embedded into a vector ${GE}_{(F_{g}(i))}$ of dimension $C$. Because $F_{g}(i)=[{\theta}_{i},d_{i},\Delta x_{i},\Delta y_{i}]$ has four attribute values, the dimension of each attribute scalar is $C/4$.

The $F_{b}(x_{i},y_{i})$ includes a 32-D detection shared features $F_{d}$ obtained by CNN backbone and 4-D prior features $F_{p}$, as

where “$\oplus$” denotes concatenation operation. $F_{b}(i)$ has 36 dimensions. $F_{b}(x_{c},y_{c})$ is the visual features of the centroid $(x_{c},y_{c})$ in $F_{b}$.

As texts vary in scale, the sampled points may be too close to each other, resulting in very little difference between the visual and geometric features. This can cause confusion in the network output. To avoid this, we embed order information ($F_{e}(i)=Embedding(i,C)$) of the point sequence into the input feature $F_{c}(i)$ for vertex $p_{i}$ using an embedding operation in Pytorch. Therefore, the input feature $F_{c}$ of the reading-order estimation module is calculated as follows:

where $i$ denotes the $i$-th sampling point in initial boundary. The classification results ($O=[o_{0},o_{1},o_{2},o_{3}]^{T}$) of these sampling points will be obtained after the input features are encoded and reasoned by REM. $o_{j}=[\hat{p}_{{}_{0}},...,\hat{p}_{{}_{i}},...,\hat{p}_{{}_{2K-1}}]$ is a probability distribution, and each element $\hat{p_{{}_{i}}}$ represents the probability that the vertex $p_{{}_{i}}$ belongs to $j$-th class of the key points.

Joint reading-order estimation loss. To optimize and train our REM, we propose a joint reading-order estimation loss ($\mathcal{L}_{RE}$), consisting of a classification loss, an orthogonality loss, and a distribution loss. We use the balance binary cross entropy loss to supervise classification as

where $y$ represents the predicted labels for the classification of every control point. Thus, $y_{{}_{i}}$ represents the label for $i$-th control point $p_{{}_{i}}$, which is a one-dimensional vector with four elements in total (as $y_{{}_{i}}=[y_{{}_{i}}(0),...,y_{{}_{i}}(j),...,y_{{}_{i}}(3)]$). When the control point $p_{{}_{i}}$ belongs to $j$-th key corner point, $y_{{}_{i}}(j)=1$; otherwise $y_{{}_{i}}(j)=0$. Thus, $\hat{p}_{{}_{i}}(j)$ is probability of control point $p_{{}_{i}}$ belongs to the $j$-th key corner point.

To ensure that the four key points found are independent and different from each other, we further designed an orthogonality loss and a distribution loss to constrain the optimization of REM. The orthogonality loss can be expressed as

where $D_{KL}$ is the KL Divergence loss; $O^{T}O$ is the similarity matrix of prediction of REM ( $O=[o_{0},o_{1},o_{2},o_{3}]^{T}$); $I$ is an identity matrix of dimension $4\times 4$.

As shown in Fig. 7 (a), the four key points not only have independent positions but also follow a specific spatial distribution. For word-level and line-level text detection, these key points are usually located at the intersection of the long and short sides in long strips of text. When these sampling points are flattened into a sequence from the short or long side (as ① ② ③ ④shown in Fig. 7 (a)), the distributions of the four key points are similar regardless of category, as shown in Fig. 7 (b) and (c). Typically, the sequence length of the long side ($l_{2}$) is greater than that of the short side ($l_{1}$). To fully utilize this distribution information and better constrain the REM to learn and extract the hidden reading order information, we introduce a distribution constraint loss via KL-divergence as

where $o_{j}=[\hat{p}_{{}_{0}},...,\hat{p}_{{}_{i}},...,\hat{p}_{{}_{2K-1}}]$ is a probability distribution predicted by REM. $y_{j}=[y_{{}_{0}},...,y_{{}_{i}},...,y_{{}_{2K-1}}]$ is the one hot representation of sample points ($\{p_{{}_{0}},...,p_{{}_{i}},...p_{{}_{2K-1}}\}$) label belonging to $j$-th category. Finally, the proposed reading-order estimation loss ($\mathcal{L}_{RE}$) is a combination of classification loss $\mathcal{L}_{c}$ , orthogonality loss $\mathcal{L}_{o}$, and distribution loss $\mathcal{L}_{d}$, as follows

where $\alpha$ is set to 0.1 because orthogonality loss $\mathcal{L}_{o}$, and distribution loss $\mathcal{L}_{d}$ only serves as auxiliary constraints.

According to the four key points obtained, we divide the text boundary into top and bottom sides, as shown in Fig. 6 (b). At the same time, we re-sample $K/2$ control points according to the principle of equidistance on the top and bottom sides, respectively. As shown in Fig. 6 (b), the re-adjusted initial text boundary has $K$ control points ($T=\{c_{{}_{0}},...,c_{{}_{i}},...,c_{{}_{K-1}}\}$), which are symmetrically distributed on top and bottom sides.

Boundary Refinement Module. Due to the varying directions and shapes of text, it is challenging to directly predict accurate text boundary. Therefore, we first use the initial boundary to roughly locate text and separate the neighboring text. But, these coarse initial boundaries only partially cover the text instances, resulting in inaccurate recognition due to incomplete text regions. Similar to TextBPN++ [28], we employ a lightweight transformer network as our boundary refinement module (BRM). To simplify the model structure and reduce parameters, we adopt the strategy of sharing parameters and self-iterative refinement, as shown in Fig. 3. In each refinement iteration, the BRM takes the previously predicted boundary as input and generates a new boundary, enabling dynamic refinement of text boundaries to adapt to various text shapes and scales.

As illustrated in Fig. 3, the refined text boundaries become closer to the actual text boundaries as the number of iterations increases. Unlike TextBPN++[28], which uses point matching loss, we optimize this module by minimizing the Smooth L1 distance between each refined text boundary ($T={c_{{}_{0}},...,c_{{}_{i}},...,c{{}_{{}_{K}-1}}}$) and its corresponding target ($\tilde{T}={\tilde{c}{{}_{0}},...,\tilde{c}{{}_{i}},...,\tilde{c}{{}_{{}_{K}-1}}}$). Specifically, the loss $\mathcal{L}_{BR}$ of this module is formulated as

where the $N$ is the number of iterations, $c_{{}_{(j,i)}}$ is $j$-th refined prediction of point $c_{{}_{i}}$.

Thin-Plate Spline (TPS) has been widely used as a grid sampling approach in arbitrary shape scene text spotting [8, 45, 9]. However, TPS is an inflexible sampling strategy that only generates fixed sampling grids manually, as illustrated by the blue grid points in Fig. 9. These fixed sampling grids result in the sampled CNN features for recognition sequence being highly dependent on the detected text boundary. As a consequence, it becomes challenging for the recognizer to decode the correct sequences when the detection is inaccurate or the sampling features are inadequate, as shown in Fig. 1 (c1-c3). Therefore, the recognition performance highly relies on the text detection accuracy, leading to potential error propagation to recognition in these methods [8, 45, 9].

To solve this problem, we propose a novel dynamic sampling module with thin-plate spline, named DSM, which can dynamically sample the features in the detected text region for the recognition module. Our DSM can mitigate the incompatibility between text detection and recognition, especially when the detected text boundaries are imperfect for recognition. As shown in Fig. 9, the dynamic sampling module comprises two stacked $3\times 3$ convolutional layers with dilation [3,1] and one fully connected layer. Inspired by DCN[62], which learns offsets to produce deformable convolution kernels based on traditional convolution, we employ a lightweight convolution head to predict a set of position offsets for the basic grid points generated by the TPS algorithm. Specifically, we first use TPS to generate the fiducial grid points $G$ of size $w_{o}\times h_{o}$. Then, the lightweight convolution head produces a set of normalized position offsets for the fiducial grid points $G$, as

where $C_{3\times 3}$ denotes $3\times 3$ convolutional layer, and $C_{1\times 1}$ denotes $3\times 3$ fully connected layer realized by $1\times 1$ convolution. $f_{ij}$ is the input features of points ${g}_{ij}$ extracted from recognition shared features ($F_{r}$). $d_{3}$ and $d_{1}$ indicate that the dilation are set to 3 and 1, respectively. $\Delta\hat{g}_{ij}$ is the normalized offset of the point $g_{ij}$ in gird $G$. To make the learning of offset $\Delta\hat{g}_{ij}$ not affected by the size of the text instance, we normalize it by the sigmoid function. Therefore, the real offset can be calculated as

where $w$ and $h$ are the width and height of the bounding rectangle of the text instance. $\Delta G=\{\Delta g_{ij}||i\in[0,w_{o}),j\in[0,h_{o})\}$ is the set of $\Delta g_{ij}$, and $G^{{}^{\prime}}$ is the updated gird. $\beta$ is the scale coefficient and is set to 0.1, which can ensure a suitable offset space for the sampling points, as the detected text boundaries usually do not have significant errors.

In training, DSM adaptively adjusts the position of fiducial grid points by using the gradient returned from the recognition module without extra supervision. During inference, DSM can dynamically sample appropriate features in the detected text region for the recognition module to decode the text sequence accurately. Our DSM can decode text content more accurately, even in challenging scenes where the detected text boundaries may have flaws or are imperfect for recognition, as shown in Fig. 2 and Fig. 10. In Fig. 10, the detected text boundary does not cover the entire instance, making the recognition module decode the character ‘i’ as ‘l’ incorrectly. However, the sampling points of our DSM can exceed the detection boundary, enabling the recognition module to accurately decode the correct characters, even when the detected boundary does not completely cover the text instance.

By dynamically sampling features by DSM, any recognition model can be applied for the recognition. For a fair comparison, we take the model in [53, 9] as our recognition module, which consists of 6 convolutional layers, one bidirectional LSTM layer, and an attention-based decoder.

For end-to-end training the network, the objective of the function is combined with the losses of modules mentioned above, which is formulated as

where $\mathcal{L}_{IB}$ is the loss of the initial boundary module as in TextBPN++ [28]; $\mathcal{L}_{RE}$ is the loss of reading-order estimation module as Eq. 14; $\mathcal{L}_{BR}$ is the loss of boundary refinement module as Eq. 15; $\mathcal{L}_{REC}$ is the Cross Entropy Loss for the recognition module as in [53, 9]. In pre-training, the ${\lambda}_{re}$ is set to ${1}/{e^{(i-eps)/eps}}$, the ${\lambda}_{br}$ is set to ${0.1}/{e^{(i-eps)/eps}}$. $eps$ denotes the maximum epoch of training, and $i$ denote the $i$-th epoch in training. In this way, our model can prioritise learning to find the interested text region at the beginning of training, ensuring that the training can converge normally. In fine-tuning, ${\lambda}_{re}$ is set to 1.0, ${\lambda}_{br}$ is set to $0.01$. ${\lambda}_{rec}$ is empirically set to 0.2.

We use ResNet-50 with DCN [62] as the backbone and pre-train our model for 100 epochs on a mixture of SynthText 150K, MLT-2017, Total-Text, and ICDAR2015 datasets. During pre-training, we apply the Adam optimizer with an initial learning rate of $0.001$ and weight decay of $0.0001$. The pre-trained model is then fine-tuned on the target dataset for 800 epochs with the initial learning rate set to $0.0001$ and divided by ten at 400 epochs, using the Adam optimizer. We use 16 control points for text boundary and perform five iterations of BRM. The default number of sampling points is $16\times 64$, and data augmentation techniques such as random scaling, cropping, and distortion (e.g., random blur, brightness adjustment, and color change) are applied. To enhance the model’s ability to recognize text in different reading-orders, we add random rotation with a wide-angle ($-180^{\circ},180^{\circ}$)) and denser sampling around [$0^{\circ},\pm 30^{\circ},\pm 60^{\circ},\pm 90^{\circ},\pm 120^{\circ},\pm 150^{\circ},\pm 180^{\circ}$].

During training, we randomly crop text regions without cutting any text and resize them to $640\times 640$. We pre-train our model using two RTX-3090 GPUs with an image batch size of 24 and fine-tune using a single RTX-3090 GPU with an image batch size of 12. During inference, we maintain the aspect ratio of test images and resize and pad them to the same size. Our implementation is based on PyTorch 1.7 and Python 3, and testing is performed on a single RTX-3090 GPU with a single thread. Recall, Precision, and F-measure are represented by R”, P”, and “F”, respectively.

In exploration experiments, we only pre-train the model with ten epochs to reduce training time costs. Other training details are described in Sec. IV-B. Some special details are described in the corresponding section. In default, random rotation data augmentation is used.

Reading-order estimation module (REM). Tab. I shows that using the reading-order estimation module (REM) improves both detection and recognition performance, especially on Rot.Total-Text and Inverse-Text datasets, compared to the baseline model without REM. REM also brings significant improvement in text spotting performance, with 14.6% and 11.4% improvement on ‘None’ and ‘Full’, respectively, on Rot.Total-Text, and 7.4% and 8.2% improvement on ‘None’ and ‘Full’ respectively on Inverse-Text. However, the gain brought by REM is less significant on Total-Text, with only 1.8% and 1.1% improvement on ‘None’ and ‘Full’, respectively. The experimental results suggest that the impact of REM on text spotting performance depends on the ratio of inverse-like text in the testing data, which is about 40% in Inverse-Text and relatively high in Rot.Total-Text due to small angle rotations that can turn some texts into inverse-like. The ratio of inverse-like instances is low in Total-Text, but REM is still important for accurately recognizing texts, as shown in Tab. II. Regardless of the sampling method, REM improves text recognition performance by about 6.5% on ‘None’ and 8% on ‘Full’ on Inverse-Text.

Joint reading-order estimation loss ($\mathcal{L}_{RE}$). In Tab. III, we examine the impact of the proposed joint reading-order estimation loss ($\mathcal{L}{RE}$) on Inverse-Text, which comprises a classification loss $\mathcal{L}{c}$, an orthogonality constraint $\mathcal{L}{o}$, and a distribution constraint $\mathcal{L}{d}$. By conducting incremental experiments, we evaluate the effectiveness of each component in $\mathcal{L}{RE}$. Our results reveal that when the orthogonality constraint $\mathcal{L}{o}$ is applied, there is a performance improvement of 0.8% in F-measure and 1.9% in ‘None’ for both detection and recognition tasks. Similarly, when we add the distribution constraint $\mathcal{L}_{d}$, we observe a 0.5% improvement in F-measure and 0.9% in ‘Full’ for both tasks. Combining $\mathcal{L}{o}$ and $\mathcal{L}{d}$ leads to significant performance improvement of 1.5% in Recall, 1.3% in F-measure, 2.6% in ‘None’, and 2.0% in ‘Full’. We also present intermediate visual comparison results to verify the impact of orthogonality and distribution constraints on the prediction of reading-order, as depicted in Fig. 12 (a-c). Specifically, $\mathcal{L}{o}$ ensures the independence of the four key points of reading-order and maintains their distinctness. In contrast, $\mathcal{L}{d}$ constrains the orderly distribution of these points at the corners of the text boundary. The combination of three losses ($\mathcal{L}{c}$, $\mathcal{L}{o}$, and $\mathcal{L}_{d}$) significantly improves the detection and recognition performance, as shown in Fig. 12 (c).

Sampling methods. In Tab.II, we can see that our DSM sampling method outperforms other methods such as Masking ROI, BezierAlign, and TPS. Specifically, with REM, our DSM improves performance by 4.7% on ‘None’ and 3.1% on ‘Full’ compared to Masking ROI, by 2.1% on ‘None’ and 1.1% on ‘Full’ compared to BezierAlign, and by 2.3% on ‘None’ and 1.7% on ‘Full’ compared to TPS. When the detection results are slightly poor without REM, the advantages of our DSM become even more significant. This demonstrates that our DSM performs well when text detection results are imperfect and can improve text detection. Moreover, our DSM also brings slight performance improvements in detection compared to TPS. Fig. 13 shows visual comparisons between TPS and DSM sampling points.

Initial boundary module (IBM). We conducted experiments on the Total-Text dataset to assess the robustness of our model with different initial boundary generation strategies (such as Text Kernel in PSENet [57], Probability map in DB [56], Distance Field in TextBPN++[28]). To ensure a fair comparison, we trained the detection branch with the same settings (600 epochs with AdamW optimizer and initial learning rate of $0.0001$). Our results, as shown in Tab. IV, indicate that different initial boundary generation strategies do not have a significant impact on detection performance (87.28% F-measure for Text Kernel, 87.69% F-measure for Probability map, 87.93% F-measure for Distance Field). In our method, the initial boundaries are mainly used to locate the text instance roughly, and the subsequent boundary refinement module refines these coarse boundaries to accurate text boundaries. The detected text boundary of any text detection method can be used as our initial boundary, meaning that any text detection method (such as TextField, DB, PSENet, etc.) can serve as our IBM to quickly build an end-to-end text spotting method, even if its detection results are not as satisfactory.

Refining iteration number. Our BRM model allows flexibility in choosing the number of iterations during testing, thanks to its shared parameters and self-iterative design. In Tab. VI, we demonstrate that as the number of iterations increases, the detection performance gradually improves and eventually stabilizes while the inference speed decreases. Notably, even a single iteration leads to a significant improvement in detection performance. With three iterations, the performance stabilizes around 87.9% on F-measure. The approximate time cost for each iteration is about 2ms, much less than the time cost of other parts (about 92 ms). Balancing efficiency and performance, we set the default number of iterations to 3 during testing.

We evaluate our method on four publicly available benchmarks: Total-Text, CTW-1500, ICDAR2015, and Inverse-Text. Quantitative results against other state-of-the-arts are presented in Tab. V, VIII, VII, and IX. For inverse-like scene text detection and spotting, we show qualitative visual results in Fig. 11 and 14, respectively.

Total-Text. In testing, we scale the input image sides into (640, 1024) while maintaining the aspect ratio. As shown in Tab. V, our method outperforms all previous methods in detection, surpassing the best-reported result by 4.5% on Precision and 1.2% on F-measure. For the end-to-end case, our method achieves competitive results (83.5% on ‘Full’) compared to methods based on ”ResNet-50-Transformer” like TESTR [47] (83.9% on ‘Full’). Our method outperforms all previous methods based on ”ResNet-50-FPN” and surpasses the best-reported result [9] by 1.2% on ‘Full’.

CTW-1500. For CTW-1500, we set the number of control points on the text boundary to 32, and the number of sampling points is set to $16\times 128$ due to the line-level curved text dataset. The input image sides are scaled into (640, 1024) while maintaining the aspect ratio. As shown in Tab. VII, our method outperforms all previous methods in both detection and end-to-end results, except for CNN-Transformer-based methods like TESTR [47] and SPTS [48]. Specifically, our method outperforms all previous CNN-based methods like ABINet++[64], TPSNet [9], ABCNet v2 [53], and surpasses the best-reported result [64] by 2.2% on ‘None’ and 2.6% on ‘Full’ in terms of F-measure. In detection, our method only slightly lags behind the best-reported CNN-Transformer-based method [47] by 0.2% on F-measure. Our method ranks second in most metrics, with only a slight gap to the first. Due to CTW-1500 is a line-level dataset, there are some overly long sentences inside. Our method based on CNN is limited by the receptive field, which may result in lower recognition accuracy than these transformer based method. Another problem is that CTW-1500 has some unreasonable or missing annotations, as mentioned in [17, 21], which also can bringing some performance losses.

ICDAR2015. We conducted evaluations on the ICDAR2015 benchmark, which includes many perspective texts, and the results are listed in Tab.VIII. During testing, we scaled the short side of the input image to 960 while maintaining its aspect ratio. Our method achieved comparable performance with text spotting methods like CharNet[35] and TESTR in the detection stage. In the Word Spotting tasks, our method also delivered a competitive performance with text spotting methods like Mask TextSpotter v3 [6] and TESTR, and it achieved remarkable performance (80.0%) on weak lexicon cases. Furthermore, our method significantly outperformed previous TPS-based methods like TUTS and ‘Boundary TextSpotter’, demonstrating the effectiveness of our method.

Inverse-Text. Although reading-order is important for scene text spotting, the ratios of inverse-like scene texts are relatively low in several datasets, such as 2.8% in Total-Text, 5.2% in CTW-1500, and 0.0% in ICDAR2015. Therefore, in these datasets, our reading-order estimation module (REM) does not fully demonstrate our DSM module achieves its advantages and more positive gains. To demonstrate the effectiveness of our REM, we conducted experiments on Inverse-Text, an arbitrary-shape scene text test set with approximately 40% inverse-like instances. As Inverse-Text does not have a training set, we fine-tuned our model on Total-Text with random rotation for 800 epochs and evaluated it on Inverse-Text, scaling the input image to (640, 1280) while maintaining its aspect ratio. Our method outperformed all previous methods and achieved the best performance on multiple evaluation metrics (e.g., Recall, F-measure, None, Full). In the text spotting task, our method surpassed the best result by 5.7% on ‘None’ and by 3.9% on ‘Full’, significantly outperforming CNN-Transformer based methods such as TESTR and SwinTextSpotter. Furthermore, our IAST accurately predicts the reading-order of scene text with very complex layouts and recognizes inverse-like scene text via the predicted reading-order, as shown in Fig. 11 and Fig. 14. Our results demonstrate the effectiveness of our methods in reading inverse-like scene text and highlight the importance of reading-order for scene text spotting, particularly in complex layout scenes.

Visual Comparison. We utilize some previous state-of-the-art spotters to get qualitative results on Inverse-text for giving a more intermediate visual comparison, as shown in Fig. 15. We select ABCNet-v2 [53] representing the CNN-based spotting methods which predict control points. We select SwinTextSpotter [66] representing the transformer-based spotting methods which predict text segmentation. Officially released model weights trained on Total-Text are adopted for producing visual results. We can find that the spotters (e.g., ABCNet-v2 and SwinTextSpotter) failed to correctly recognize inverse-like texts because of the unconventional reading-order, as shown in Fig. 15 (a) and (b). Benefiting from the ability to aware reading-order, our model can accurately recognize these inverse-like texts, as shown in Fig. 15 (c). These examples in Fig. 15 are enough to demonstrate the improvement of our method when reading the complex layout and inverse-like texts.

The proposed method shows strong performance in spotting both normal and inverse-like scene text in most cases, although it may struggle with some special cases, such as mirror text, small blurry text, and occlusion text, as shown in Fig. 16. Some failure cases are also marked with yellow arrows in Fig. 11 and Fig. 14. As shown in Fig. 16 (a) and (d), mirror texts are especially challenging as their control points need to be pressed counterclockwise. Additionally, some small blurry texts are detected, but their recognition accuracy cannot be verified, as shown in Fig. 16 (b), as they were annotated as ‘Don’t Care’. In rare cases, overlapping key points in the reading-order may result in unsatisfactory predictions, as shown in Fig. 16 (c). Fortunately, these errors can be corrected through additional processing. It’s worth noting that these special cases are challenging for all text spotting methods and require further research.