Deformable Kernel Expansion Model for Efficient Arbitrary-shaped Scene Text Detection

The entire architecture of our model is summarized as Fig.2. Following the conventions [28, 26, 30, 11], the ResNet [5] is employed to extract multi-scale features, and then these features are fused as the contextual feature $f$ by a feature-pyramid network [40]. This step can be mathematically denoted as $f=F(I)$, where $F(\cdot)$ is the hybrid mapping of the ResNet and the feature-pyramid network, and $I$ is an input image.

In the TKG step, we follow the pipeline of segmentation-based scene text detection methods, and leverage a lightweight segmentation network $S(\cdot)$ as an initial detection head to predict the probability map $\hat{Y}=S(f)$, which encodes text kernel $\mathcal{K}$ at the pixel level. We uniformly sample $N$ vertices on the kernel boundary as the contour representation, which is denoted as $P^{\mathcal{K}}=\{p_{i}:=(x_{i},y_{j})\}^{N}_{i=1}$, and used as the initial detection boundary. A contour composed of $N$ (e.g., $N=128$) vertices is sufficient to describe most of the instances well [22]. Moreover, these vertices are ordered in certain rules, which will be introduced in Sec.3.2. By default, $P^{\mathcal{K}}$ mentioned in our literature is the re-ranked vertex collection.

In the DCE step, we obtain the convolution features for each vertex in $P^{\mathcal{K}}$ by retrieving them in the feature $f$ based on its coordinates. The previous method [14] indicates that learning the coordinate mapping between Cartesian space and the pixel space is challenging. Therefore, we follow some recent works [32, 16], which append the relative coordinates of vertices at the end of convolution features as extra information cues, to alleviate this issue. In our method, the upper left corner of the minimum bounding box is considered the origin of the coordinate system for calculating the relative coordinates. The aforementioned hybrid feature processing is formulated as follows,

where $U:=\{u_{i}\}_{i=1}^{N}$ are the obtained vertex features and $\omega(\cdot,\cdot)$ is the mapping function of the aforementioned process.

The goal of the DCE step is to learn the mapping between these vertex representations and the offsets of vertices on the kernel contour to the ones on the boundary of the final detection box with a contour deformation network[22] $D(\cdot)$,

where $\triangle G^{\mathcal{K}}$ are the learned coordinate offsets of $P^{\mathcal{K}}$ to the ground truth $G^{\mathcal{K}}$. Then the predicted coordinates of the corresponding vertices on the final bounding box can be obtained as follows,

We adopt the small text kernel, because it is easier to separate two adjacent text regions and less likely disturbed by the background. Therefore, the contour deformation process is actually a kernel expansion process.

In the training phase, we need to generate the text kernel and the contour ground truth for each training example based on its annotated text boundary. We follow the previous works [28, 11], and shrink the annotated text boundary into the text kernel via the Vatti clipping algorithm [27]. This algorithm is able to calculate the shrinking margin $m$ between the shrunken kernel contour and the original text boundary based on a given shrink ratio. In our experiments, we empirically set the shrink ratio to 0.4 for generating the small text kernel.

After that, we uniformly sample $N$ vertices on the generated text kernel boundary for depicting the kernel contour. These sampled vertices are sorted in a clockwise manner, and the vertex that is the closest point to the upper left corner of the minimum surrounding rectangle of the text kernel is deemed as the first vertex.

Let $\Phi(\cdot)$ be such a vertex sample and sorting operation. The kernel boundary can be denoted as a sorted vertex collection as mentioned, $P^{\mathcal{K}}=\{p_{i}:=(x_{i},y_{j})\}^{N}_{i=1}=\Phi(\tilde{Y})$, where $\tilde{Y}$ can be the text kernel of a testing example generated by the segmentation network $\hat{Y}$ or the pre-processed text kernel of a training example (the ground truth) $Y$.

The Deformable Kernel Expansion (DKE) model accomplishes accurate text detection from two aspects. On the one hand, DKE generates text kernels for an input image via a segmentation network. The kernels are used to capture the shape characteristics of text regions. On the other hand, a regression network is applied to learn the contour deformation of text kernels to the final detection boundaries only with several sampled vertices. In such a manner, the DKE model should simultaneously minimize the text kernel generation loss and the deformable contour expansion loss to achieve the above goals.

In the inference phase, we can use the obtained networks $\hat{F}$,$\hat{S}$,and $\hat{D}$ for extracting the features, generating the text kernels and predicting the contour deformation offsets for an image $I$ as follows,

Then, the final detection boundary depicted by $N$ vertices is produced as follows,

SynthText [3] is a synthetic dataset consisting of more than 800$k$ synthetic images. It is only used to pre-train our model.

MLT-2017[21] is a multi-lingual dataset that contains 7200 training images, 1800 validation images, and 9000 test images. Those images are annotated with quadrilateral boxes at the word level. The training and validation images are used to pre-train our model.

ICDAR2015 [7] dataset consists of 1000 training images and 500 testing images, which are captured by Google Glass. Most of the including text instances are severely distorted or blurred. The text instances are labeled at the word level with quadrilateral boxes.

MSRA-TD500 [35] dataset is a multi-language dataset that includes English and Chinese text instances. There are 300 training images and 200 testing images. The text instances are all labeled at the text-line level.

Total-Text [1] is a curved text dataset including 1255 training and 300 testing images. It contains horizontal, multi-oriented, and curve text instances labeled at the word level.

CTW1500 [36] is another curved text dataset, including 1000 training images and 500 testing images. It contains both English and Chinese texts annotated at the text-line level with polygons.

FPN [40] with ResNet [5] is used as our backbone. Following previous works [28, 30, 37, 29], we first pre-train our models on external text datasets (e.g., SynthText and MLT-2017). Following a poly learning rate policy[40], we fine-tune the pre-trained models on several related real-world datasets with the initial learning rate $2\times 10^{-4}$ and the power 0.9. All models are optimized by the Adam optimizer with the batch size of 16 on two RTX 3090 GPUs.

To make fair comparisons in inference speed between different methods, we set up a platform to test some top-performance detectors with the same strategy. Our system consists of one RTX-3090 GPU and one Ryzen-5700X CPU. Considering that the performance of the model will affect the time consumption of post-processing, we use official weights provided instead of training them by ourselves. All tests are processed in a single thread with a batch size of 1. More details can be referred to Appendix.

Discussion of Contour Deformation Strategy: Recent segmentation-based text detection methods, like [30, 23, 11, 12], share a similar pipeline in locating the text kernel. DB [11] leverages the Vatti clipping algorithm [27] as a simple post-processing method to generate the final detection contour based on segmentation results. The main merit of this method is efficiency, but it greatly relies on the accuracy of segmentation results, neglecting adaptively adjusting detected contours. We conduct several experiments under this setting to compare with our proposed contour deformation module DCE, which employs a network for learning the deformation from the kernel boundary to the final detection boundary. According to the results tabulated in Tab.1, DCE obtains 2.9% and 1.5% performance gains over baseline in F-measure on Total-Text and CTW1500, respectively, only with a marginal drop in FPS. These results imply that it is worthwhile to incorporate the contour deformation part together with the kernel segmentation for optimization. And learnable kernel expansion module enables outputting more accurate and robust detection boundaries than expanding kernel in a fixed manner. Moreover, our proposed optimal bipartite graph matching loss contributes additional 0.9% and 1.0% performance gains in F-measure on Total-Text and CTW1500, respectively, while maintaining the same detection efficiency. This phenomenon validates the importance of contour deformation loss in optimization. The contour deformation losses will be discussed in detail in the next section.

Discussion of Contour Deformation Loss: We have employed three contour deformation losses, namely Direct Matching Loss (DML), Nearest Neighbor Matching Loss (NNML), and Optimal Bipartite Graph Matching Loss (OBGML), to our Deformable Kernel Expansion (DKE) model. DML does not need to conduct any further vertex pairing. The latter two losses apply different vertex pairing strategies to calculate loss. NNML employs the nearest neighbor search strategy to find the corresponding vertex in ground truth for each predicted vertex. While our proposed OBGML pairs vertex sequences by considering the vertex pairing issue as a bipartite graph matching problem for solution, which guarantees the global optimum. Fig.6 demonstrates the performances in F-measure under the same settings. The results show that OBGML outperforms the other two losses on several settings and datasets. We attribute this performance to the superior bipartite graph matching-based vertex pairing strategy. Surprisingly, NNML performs much worse than the most simple and common loss DML when applied for kernel expansion. We attribute its fail to the fact that there is a big difference between our practical scenario and the one designed for NNML. In the process of expansion, predicted vertices are optimized towards the longer side of the text region, which caused the lossing of key ground-truth vertices and complete shape information as shown in Fig.4.

Iterations for Contour Deformation: Traditional contour-based scene text detection methods[41, 38, 2] often need to iteratively adjust the coarse predicted boundaries several times. The initial inputs predicted by them often target at text region boundaries. Different from them, DCE module applies the kernel expansion to model the contour deformation. The local features on text boundaries are less reliable than the ones on the kernel boundary, which are located at the more central area of the text region. The reliable features speed up the optimization process, and only a few iterations of contour adjustment can guarantee good convergence. The observations in Tab.2 well validate this. Tab.2 reports the performances of our model in the first two iterations on Total-text and CTW1500 datasets. The results reveal that our model can obtain an outstanding performance in the first iteration, but more iterations degenerate the performances.

Curved text detection. Tab.3 reports the scene text detection performances on two curved text datasets named Total-Text and CTW1500. Our approach achieves the best and the second-best performances in F-measure on CTW500 and Totat-Text, respectively. For example, compared with DBNet++, which is known as one of the most influential and efficient segmentation-based approaches, our method almost enjoys the same efficiency. However, our method outperforms it by 2.6% and 1.9% in F-measure on Total-Text when the backbone is ResNet18 and ResNet50, respectively. This gain on CTW500 is 0.6%. CT is also a segmentation-based approach and performs very well on all two datasets when the backbone is ResNet18. However, it is worthwhile to point out that our method enables us to achieve similar performance with only 70% of its inference time. TextBPN is the best-performed contour-based approach among all baselines. Our method achieves the similar or even better performance in F-measure but enjoys 2.4 times faster inference speed. Clearly, our method enjoys a good tradeoff between detection performance and inference speed. Fig.5 visualizes some detection results of several top-performance detectors on the Total-Text dataset. These visualizations also confirm that our method can provide more accurate detection results and better separate two adjacent text regions.

Quadrangular Text detection. According to the results in Tab.3 and Tab.4, similar phenomena can be observed on two quadrangular text datasets named MSRA-TD500 and ICDAR2015. Our method still achieves competitive performances on these two benchmarks. For example, our method performs best in F-measure on MSRA-TD500 when the backbone is ResNet50. FewNet[25] is the recent SOTA detector. The performance gain of our method over it is 0.8% on MSRA-TD500.