PAN++: Towards Efficient and Accurate End-to-End Spotting of Arbitrarily-Shaped Text

Inspired by Faster R-CNN [27], Li et al. [14] proposed the first framework for horizontal text spotting, which contains a text proposal network for text detection and a CTC-based method for text recognition, as shown in Fig. 3(a). Meanwhile, Busta et al. [28] designed a similar framework to the work in [14]. But its detection head works better for both horizontal and multi-oriented text instances. NguyenVan et al. [29] developed an end-to-end text reading framework by incorporating a pooling-based scene text proposal technique, where false alarms elimination and words recognition are performed simultaneously. Subsequently, Liu et al. [7] proposed a multi-oriented text spotter shown in Fig. 3 (b), termed FOTS, which is equipped with a new RoI extractor (i.e., RoIRotate) to extract the features of quadrilateral text instances. A similar framework is also developed by He et al. [15], whose recognition head is implemented by an attention-based sequence-to-sequence decoder. Although these methods have achieved good performance across straight text benchmarks (e.g., IC15 [25] and MSRA-TD500 [26]), they fail to detect and recognize text lines of curved shapes (see Fig. 1(a)).

As illustrated in Fig. 3(c), Liao et al. [8] proposed Mask TextSpotter (MTS) by adding character-level supervision to Mask R-CNN [18] to detect and recognize text lines and characters simultaneously. This is probably the first arbitrarily-shaped text spotter. Note that character-level annotations are not always available. The improved version [30] of Mask TextSpotter removed the dependence of character-level annotations. CharNet in [31] is the first one-stage arbitrarily-shaped text spotter as shown in Fig. 3(d). Similar to Mask TextSpotter, it requires character-level annotations for training. Feng et al. [32] proposed TextDragon as shown in Fig. 3(e), which contains a novel RoI extractor (i.e., RoISlide) to represent a text line by features of multiple text segments. Qin et al. [16] designed an arbitrarily-shaped text spotter as shown in Fig. 3(f), where an attention-based text recognition head was added on Mask R-CNN to simultaneously detect and recognize irregular text lines. Recently, Liu et al. [22] formulated irregular text instances with parameterized Bezier curves, and proposed a regression-based arbitrarily-shaped text spotter ABCNet (see Fig. 3(g)). Note that pixel-based (mask) representation used in the previous methods [4, 8, 16] may be more flexible and reliable than the Bezier curve representation due to the fact that the latter one can only represent a series of curves under specific constraints.

In [7], Liu et al. proposed a fast text spotter FOTS to detect and recognize multi-oriented texts. However, FOTS cannot process curved text lines. Most methods [8, 30, 31, 32, 16] mentioned in Sec. 2.1.2 have achieved high accuracies on arbitrarily-shaped text spotting, but the inference speed is rarely addressed in those methods.

This inspires us to design an improved pixel-based representation and develop an effective end-to-end text spotting framework based on it in this work.

The detection head of PAN++ consists of only two convolutions, as shown in Fig. 6. The outputs of the detection head are text regions, text kernels, and instance vectors. 1) Text regions keep the complete shape of text lines, but closely lying text lines are often overlapping (see Fig. 4 (f)). 2) In contrast, text kernels can easily distinguish adjacent text instances, but they lose the complete shape of text lines (see Fig. 4 (g)). 3) The instance vector is a high-dimensional representation that contains instance information of each pixel. In other words, pixels belonging to the same text line tend to have similar instance vectors (see Fig. 4 (h)). Therefore, instance vectors can direct the pixels in text regions towards the corresponding text kernel, which successfully combines the advantages of text regions and text kernels.

The three outputs complement each other and make it possible to describe text lines of arbitrary shapes.

Pixel aggregation (PA) is applied to optimize instance vectors and assemble the outputs of the detection head. We borrow the idea of clustering to design PA. As shown in Fig. 7, if we treat different text lines as different clusters, their text kernels are cluster centers, and the pixels inside text regions are the samples the samples to be clustered.

Naturally, to cluster pixels to the corresponding text kernels (see the green arrows in Fig. 7), the distance between the pixel and the text kernel inside the same text lines should be minimized. In the training phase, we use an aggregation loss $\mathcal{L}_{agg}$ formulated as Equ. 1 to implement this rule.

Generally, the aggregation loss $\mathcal{L}_{agg}$ is employed to pull the text pixel to the target text kernel. The symbols in Eqns. (1) and (2) are defined as follows:

• •

  $N$: the number of text lines.

• •

  $T_{i}$: the text region of the $i$th text line.

• •

  $K_{i}$: the text kernel $K_{i}$ of the text line $T_{i}$.

• •

  $\mathcal{D}_{1}(p,K_{i})$: the distance between the text pixel $p$ and the text kernel $K_{i}$.

• •

  $\mathcal{R}(\cdot)$: the ReLU function, ensuring the output is non-negative.

• •

  $\mathcal{F}(p)$: the instance vector of the pixel $p$.

• •

  $\mathcal{G}(K_{i})$: the instance vector of the text kernel $K_{i}$, which can be calculated by: $\mathcal{G}(K_{i})=\sum_{p\in K_{i}}\mathcal{F}(p)/\left|K_{i}\right|$.

• •

  $\delta_{agg}$: a constant, which is set to 0.5 experimentally.

Besides that, the cluster centers need to keep discrimination correspondingly. As a result, the instance vector of the text kernel should keep a certain distance from other text kernels and the background, as shown in the red arrows in Fig. 7. This rule can be summarized as a discrimination loss $\mathcal{L}_{dis}$ as follows:

The goal of the discrimination loss $\mathcal{L}_{dis}$ is to push the text kernels and the background away from each other. The following is the definition of the symbols in Eqns. (3), (4), and (5).

• •

  $B$: the background.

• •

  $\mathcal{D}_{b}(K_{i})$: the distance between the text kernel $K_{i}$ and the background.

• •

  $\mathcal{D}_{2}(K_{i},K_{j})$: the distance between the text kernel $K_{i}$ and the text kernel $K_{j}$.

• •

  $\delta_{dis}$: a constant, which is set to 3 in all experiments.

In the testing phase, we utilize the predicted instance vector to guide the pixels in text regions to the corresponding text kernel. As shown in Fig. 8, the post-processing has three steps as follows: 1) finding the connected components in the segmentation result of text kernels, and each connected component is regarded as a text kernel; 2) for each text kernel $K_{i}$, involving its neighbor pixel $p$ (4-way) in text regions when the Euclidean distance of their instance vectors is less than $d$; 3) repeating the second step until there is no available neighbor pixel in text regions.

Masked RoI is an RoI extractor used to extract the fixed-size feature patches for arbitrarily-shaped text lines. It contains four steps: 1) calculating the minimal upright bounding rectangle containing the target text line; 2) extracting the feature patch within the upright bounding rectangle. 3) filtering noise features by multiplying the feature patch by a binary mask, where the weight outside the target text line is 0. 4) resizing the feature patch to a fixed size. These steps can be summarized as Eqn. (6).

where $m_{i}$ is the binary mask of the $i$th text line. ${\rm Rect(\cdot)}$ refers to the upright bounding rectangle of a binary mask. ${\rm Crop}(F_{f},\cdot)$ is the operation of cropping the feature patch inside an upright RoI region from the feature map $F_{f}$. $\circ$ means element-wise multiplication. ${\rm Resize}_{h\times w}(\cdot)$ represents the operation of resizing the feature map’s size to $h\times w$ via bilinear interpolation. In all experiments, $h$ and $w$ are set to 8 and 32 pixels, respectively.

The proposed Masked RoI has two benefits: 1) The binary mask of the target text line can eliminate the noise features caused by the background or other text lines, so as to accurately extract the features of arbitrarily-shaped text lines. 2) Masked RoI skips the spatial rectification step (e.g., STN in ASTER [6]), reducing the time cost of feature extraction.

Although our Masked RoI is somewhat analogous to RoI Masking proposed in [16], there is a major distinction between them. In RoI Masking [16], the mask used to filter noise features is an instance-wise attention map, whose weight is soft (i.e., a floating number). This approach is not suitable for our PAN++, which cannot predict instance-wise soft masks of text lines. Differently, our Masked RoI removes noise by the binary mask, which can be easily generated by PA. Moreover, the noise weight in the binary mask is 0. Therefore, the proposed Masked RoI can remove noise more thoroughly than RoI Masking.

Our recognition head is a seq2seq model equipped with multi-head attention [21]. As shown in Fig. 9, it is composed of a starter and a decoder.

The starter is proposed to find the start of the string (SOS), which is not necessarily the left-most region of an arbitrarily-shaped text line. The starter only contains a linear transformation (embedding layer) $\mathcal{E}_{1}$ and a multi-head attention layer $\mathcal{A}_{1}$. As described in Eqn. (7), the embedding layer $\mathcal{E}_{1}$ transfer the SOS symbol (one-hot) to a 128-dim vector, and then this vector is fed to the multi-head attention layer $\mathcal{A}_{1}$ along with the flatten feature patch $F_{roi}$, producing SOS’s feature vector ${\rm\textbf{f}_{s}}$ , which is also the input of the decoder at the initial time step.

The decoder is made up of only two LSTM layers and one multi-head attention layer $\mathcal{A}_{2}$. At time step 0 (initial time step), the decoder takes SOS’s feature vector ${\rm\textbf{f}_{s}}$ and zero LSTM initial states as input. At time step 1, we input the hidden state ${\rm\textbf{h}_{0}}$ of time step 0 into LSTM along with the SOS symbol, and predict the output symbol ${\rm y}_{1}$ of time step 1. After that, the output symbol of the previous step is fed into LSTM until predicting the end of the string (EOS) token. We formulate this process as follows:

Here, 0 represents zero states. All input symbols (including SOS and EOS) are represented by one-hot vectors, followed by an embedding layer $\mathcal{E}_{2}$. ${\rm y}_{t}$ is the output symbol of the time step $t$. During training, the input symbols of the decoder are characters in the ground-truth sequence.

Our recognition head is lightweight while maintaining good accuracy. Our recognition head has no encoder, and its decoder only contains two LSTM layers and an attention layer. Moreover, our attention layer is based on multi-head attention, which can effectively fuse temporal (LSTM) features and visual (CNN) features. Besides that, the computational cost of multi-head attention is lower than the common attention module used in [13].

ICDAR 2015 (IC15) [25] is a commonly used dataset for end-to-end text detection and recognition. It contains a total of 1,500 images, 1,000 of which are used for training and the remaining are for testing. In this dataset, text lines are annotated by quadrangles and transcriptions at the word level.

SynthText [62] is a large-scale synthetically generated dataset containing 800K synthetic images. There are a large number of multi-oriented text lines, which are annotated at word-level and character-level rotated bounding boxes, as well as corresponding transcriptions. Following [34, 59, 37], this dataset is used for pre-training or jointly training models of PAN++.

To test the performance of PAN++ on arbitrarily-shaped text, we conduct experiments on Total-Text and CTW1500 and compare it with previous start-of-the-art methods. In these experiments, we use the ResNet [2] pre-trained on ImageNet [65] as the backbone network of our method. In addition, we set the dimension of the instance vector to 4, the negative-positive ratio of OHEM to 3, the shrinking rate $r$ of the text kernel to 0.7, and the distance threshold $d$ of PA to 3.

Following the common practices [3, 4, 8, 22], we ignore the blurred text regions labeled as “DO NOT CARE” during training, and apply random scale, random horizontal flip, random rotation, and random crop on training images. All models are optimized by using ADAM [66] optimizer with a batch size of 16 on 4 GPUs. The initial learning rate is set to $1\times 10^{-3}$. Similar to [67], we use the “poly” learning rate strategy in which the initial rate is multiplied by $(1-\frac{\rm iter}{\rm max\ iter})^{\rm power}$, and the $\rm power$ is set to 0.9 in all experiments. In the testing phase, we resize the input image to different scales and report the performance on text detection and end-to-end text spotting tasks. All results are tested with a batch size of 1 on a V100 GPU and a 2.20GHz CPU in a single thread unless otherwise stated.

In the text detection task, to make fair comparisons, we train PAN++ (without the recognition head) with two widely used strategies as follows: 1) following [68, 3, 17] to train models without external text datasets; 2) following [34, 17, 4] to fine-tune models pre-trained on external text dataset (e.g., SynthText and IC17-MLT). In the first strategy, we train our method for 36K iterations. In the second one, we first pre-train models on the external text dataset for 50K iterations and then fine-tune the pre-trained models for 36K iterations.

In the text spotting task, the training strategies used in previous works [7, 22, 8, 16] can also be divided into two categories: 1) fine-tuning models pre-trained on external text datasets, and 2) jointly training models on multiple text datasets. For fair comparisons, we train our end-to-end text spotting models using the two strategies respectively. Specifically, when using the fine-tuning strategy, we first pre-train models on the training images in the SynthText, COCO-Text, and IC17-MLT datasets for 150K iterations, and then we fine-tune the pre-trained models on Total-Text or CTW1500 for 7K iterations. For the jointly training strategy, we train our models for 150K iterations on the joint dataset that includes all training samples in SynthText, COCO-Text, IC17-MLT, Total-Text and IC15.

As shown in Table I, without pre-training on external text datasets, PAN++ with the shorter side being 320 pixels yields an F-measure of 79.5% on CTW1500, surpassing most of the counterparts, including some methods (e.g., TextSnake [37]) pre-trained on SynthText (79.5% vs. 78.4%). Moreover, our inference speed is 8 times faster than TextSnake and 4 times faster than EAST [3] (the fastest method before). It notable that when fine-tuning the model pre-trained on SynthText, PAN++ (S: 640) achieves the best F-measure of 84.0%, outperforming all counterparts while still keeping the fastest inference speed (36 FPS).

Similar results are also on the Total-Text dataset. When the shorter side is 320 pixels, the inference speed of our method reaches 84.9 FPS, which is at least 6 times faster than previous methods, while the F-measure is still very competitive (80.0%). The best F-measure of our method is 85.3%, surpassing current state-of-the-art methods over 1.7 points. Meanwhile, its speed is close to 40 FPS, which is still 3 times faster than the second-fastest TextSnake [37].

These results prove the superiority of the proposed PAN++ (detection only) in arbitrarily-shaped text detection, in terms of accuracy and speed. We present some qualitative curved text detection results in Fig. 11 (a)(b), which demonstrates that the detection components of PAN++ can elegantly locate text lines with complex shapes.

End-to-end text spotting results on the Total-Text dataset are reported in Table II. Compared with the previous fastest ABCNet [22], our method with shorter side being 640 pixels runs 1.3 times faster, while our end-to-end text spotting F-measure is 2.2 points higher (66.4% vs. 64.2%). When adopting a larger input scale or jointly training strategy, the performance of our method can be further improved. With low-resolution input (the shorter side being 512 pixels), the inference speed of our method reaches 29.2, at least 11 FPS faster than previous methods. At the same time, its end-to-end text spotting F-measure is 64.9%, which is higher than most counterparts. It is notable that PAN++ achieves the best end-to-end text spotting F-measure of 68.6%, which is much better than the second-best method (Qin et al. [16]) and our speed is 4 times faster (21.1 FPS vs. 4.8 FPS).

These results indicate that the proposed method achieves state-of-the-art performance in the text spotting task, significantly outperforming existing methods, especially in the inference speed. Some qualitative results of end-to-end curved text spotting in shown in Fig. 12 (a).

We also evaluate the performance of PAN++ on straight text datasets, including IC15 and MSRA-TD500. IC15 is a representative benchmark for the text detection task and the text spotting task, and MSRA-TD500 is a dataset annotated at the text line level and often used to benchmark long text detection. During training, the shrinking rate $r$ on IC15 and MSRA-TD500 are set to 0.5 and 0.7, respectively. Other model settings, data augmentations, and training settings are the same as those in Sec. 4.2.1. In the testing phase, we rescale the shorter side of the input image to 736 or 896 pixels for fair comparisons with previous methods.

Straight text detection results are presented in Table III. On the IC15 dataset, PAN++ (the shorter side being 736 pixels) yields an F-measure of 81.2% at 28.2 FPS without pre-training on external text dataset. Compared with the previous fastest method EAST [3], our method surpasses EAST by 3.0 points in F-measure, while our method runs 2 times faster. Fine-tuning on the SynthText dataset can further increase the F-measure of our method to 83.1%, which is 0.5 points higher than TextSnake [37], but our method can run 25 times faster. Because IC15 contains many small texts, previous methods [11, 72, 57] usually employ high-resolution input and a heavy backbone to ensure detection accuracy. With these setting, the best F-measure of our method reaches 87.5%, which is higher than all counterparts. Even in this case, our inference speed is still the fastest (12.6 FPS).

On the MSRA-TD500 dataset, our method yields F-measures of 80.9% and 84.7% when using the external text dataset or not. Compared with previous state-of-the-art methods, our method enjoys higher accuracy and faster inference speed. Notably, the best F-measure of our method is 88.4%, which is at least 5.4 points higher than previous methods. This result indicates that our method is also robust for long straight text detection.

In conclusion, the proposed PAN++ (detection only) can also accurately detect straight text lines. Some qualitative results of straight text detection are shown in Fig. 11 (c)(d).

Table IV summarizes end-to-end text spotting results on the IC15 dataset. Note that, the F-measure with a generic lexicon (denoted as “G”) is the most important metric in the IC15 dataset. When the shorter side is 736 pixels, the speed of our method reaches 24.5 FPS, which is 3 to 7 times faster than previous methods. Meanwhile, it still has a competitive “G” F-measure of 66.2, surpassing most counterparts including Mask TextSpotter V2 [30] (66.2% vs. 63.5%). With ResNet50 as the backbone, our method achieves the best “G” F-measure of 69.2%, which is 1.2 points higher than the previous best text spotter (Qin et al. [16]). At the same time, our inference speed is still the fastest (13.8 FPS).

These results show that in addition to curved text, our PAN++ can also achieve an excellent balance between accuracy and speed when detecting and recognizing straight text. We present some qualitative straight text spotting results in Fig. 12 (b).

We choose the RCTW-17 [61] dataset to test the effectiveness of our method in detecting and recognizing Chinese text, which is a commonly used language that is completely different from English. Due to the number of character categories is large (up to 3,000), the detection and recognition of Chinese text is challenging. RCTW-17 has the same detection metric as English text datasets (e.g., IC15 [25]), but it uses the average edit distance (AED) [61] as the recognition metric. For a fair comparison, we randomly selected 1,000 images from the RCTW-17 training set as the validation set, and trained all models for 300 epochs in the remaining training set without using external text datasets. Other model settings, data augmentations, and training settings are the same as those in Sec. 4.2.1. In the testing phase, we rescale the shorter side of the input image to 736 pixels.

The text detection and recognition performance on the RCTW-17 dataset are reported in Table V. With ResNet18 [19] as the backbone, our PAN++ obtains 65.1 text detection F-measure and 23.4 AED, which is 11.2 points (65.1% vs. 53.9%) and 1.7 points (23.4% vs. 25.1%) better than those of the baseline (SegLink [34] + CRNN [5]). At the same time, our method runs 3 times faster than the baseline (18.5 FPS vs. 5.9 FPS). When using a heavier backbone (i.e., ResNet50 [19]) and a larger input scale, the text detection F-measure and AED of our method further improve to 68.0% and 22.2%, 14.1 points (68.0% vs. 53.9%) and 2.9 points (22.2% vs. 25.1%) better than the baseline, while keeping a faster inference speed (10.7 FPS vs. 5.9 FPS). Some qualitative Chinese text detection and recognition results are presented in Fig. 12 (c). These results prove that our PAN++ can also works well in Chinese text detection and recognition.