Arbitrary-shape Scene Text Detection via Visual-Relational Rectification and Contour Approximation

Typical bottom-up methods divide text into segments and then reason the link between them. Following this design structure, CTPN [22] was a pioneering work that brought text detection into the deep-learning area, but it simply merges text segments according to a certain threshold and could only deal with horizontal texts. SegLink [25] was designed to detect multi-oriented texts, aiming to connect the centers of two segments with an eight-neighbourhood link prediction. Subsequently, as the research focus changed to curved text, text detection transitioned from the detection of bounding boxes to the detection of contours. TextSnake [23] was the first work to attempt this transition. It used several circles with predictable angles and radius to fit curved texts. At this point, both geometric and visual information have largely been considered in curved text detection.

To further establish the relationship between text segments, Baek et al. [7] used affinity boxes to describe the neighbouring relation between text segments for further spatial reasoning, but it was insufficient for reasoning the complex relationship between curved texts and reflecting the “characterness” of texts. Later, researchers used GCNs to integrate geometric features, visual features, and relational features for link prediction [12, 8, 9]. These three methods shared a similar framework but differed in the size and number of text segments. PuzzleNet [9] used the largest text segments to merge adjacent text segments with angle differences less than $\pi/36$, which was inflexible in the following segment grouping stage. RelaText [8] generated smaller text segments than PuzzleNet, but they were still too large and too sparse to depict the “characterness” feature of the text. The size of the text segments in DRRG [12] was the smallest, and although it contained more character-level features and was more flexible when grouping, its route-finding process was of significantly higher complexity. Thus, there is often a trade-off between the size and number of text segments and the difficulty of the grouping operation, which needs to be further explored.

In our approach, we use “characterness” and “streamline” to depict the character-level and adjacency-level features of text content and a shape approximation (SAp) strategy to avoid the computational overload of the route-finding process while greatly increasing flexibility.

Suppressing false detection requires distinguishing text and non-text areas accurately. A first prior approach is to strengthen the network’s capability for depicting text features by adding additional text content information. Some methods try to use character-level annotation to increase the “characterness” expression. For example, CRAFT [7] adopted a weakly-supervised training strategy to train the character region score to split words into characters; TextFuseNet [10] fused word-level, character-level and global-level features of text to conduct text instance segmentation using Mask-RCNN [32]. Similarly, Zhang et al. [33] also used this framework to obtain character-level annotation with a weakly-supervised training strategy. However, training character-level annotation as an instance segmentation task means doing detection and recognition together, which brings additional computational burden. Other methods such as [9, 33, 34, 8, 35, 20, 36, 37] attempted to insert blocks like Non-local [38] or Deformable Convolution [39] to increase the network’s capability for extracting text features, but detection results showed that these methods often resulted in limited improvement. We need to focus more on the characteristics of the text itself.

A second prior approach is to jointly consider multiple text maps to get a final text area. ContourNet [11] tried to suppress non-text areas by considering text maps in horizontal and vertical directions, which effectively suppressed false positives. However, their method still suffered from false negatives when both horizontal and vertical text maps had defects. TextSnake [23] and DRRG [12] predicted an additional Text Center Line (TCL) map and multiplied it with the original Text Region (TR) map for FPNS. As shown in Fig. 1, the FPN produces both TCL and TR maps. It is very likely that both TCL and TR maps are flawed, which may result in error accumulation. Moreover, simply setting thresholds on different text maps [11] or directly multiplying different text maps increases the risk of suppressing the true positive text areas as well. What is required is a more effective and accurate approach to consider the feature of text instances and indirectly use text maps.

Text segments are basic components of many bottom-up methods and have therefore been used in existing works. In this paper, our re-designed dense overlapping text segments play an important role in our method, since they are involved in the generation of graph structure that is used in GCNs and the proposed shape approximation strategy. In the following sub-sections, we first give the details of the text segments, then describe how the ground-truth type of each text segment is obtained in a weakly supervised manner.

We now give details about building the relational representation between text segments, as well as link prediction and node classification, after the dense overlapping text segments have been obtained.

To predict the links between text segments and classify their types, we first need to integrate the feature representation of each text segment and construct their graph structures.

We consider two types of features for text segments, i.e., the visual feature $F_{vis}$ \addis obtained by applying convolutional operation on $P_{4}$ in FPN for generating the TCL map and the geometric feature $F_{geo}$. We note $F_{seg}$ as the feature representation matrix of text segments. Following the methods in [9, 12], the RRoI Align [41] is adopted to pool the visual features from FPN. $F_{geo}$ is formed by concatenating the feature maps $TCL$, $H$, $W$ and $\Theta$, and is embedded into the same space as $F_{vis}$ as:

where the $[;]$ operator indicates concatenation.

To build the graph structure of text segments, we treat each text segment as the pivot and connect its eight closest text segments (measured by the Euclidean distance of their center points) as its 1-hop nodes. Then, for each 1-hop node, we connect its four closest text segments (excluding the pivot).

The pivot, its 1-hop nodes and 2-hop nodes compose the basic graph structure for the text segment, which can be represented by an adjacency matrix $\mathcal{A}$. We consider the top three closest text segments of a pivot to have a link with the pivot. Similar to those in [42, 28, 12], the graph convolution is represented by:

where the $F_{seg}^{(i+1)}$ is the feature representation for the $i+1$ layer after the graph convolution of $F_{seg}^{(i)}$. $\tilde{\mathcal{D}}^{\frac{1}{2}}\tilde{\mathcal{A}}\tilde{\mathcal{D}}^{\frac{1}{2}}$ is the re-normalization trick from the original GCN paper [28] and can be calculated by $\mathcal{A}$. $\sigma$ is a non-linear activation function and $W^{(i)}$ is the weight of the graph convolutional layer.

Thus, the relational link prediction can be written as:

where $m^{(i+1)}$ denotes a multi-layer perceptron with a PReLU activation. The feature matrix $F_{seg}$ can also be written as $F_{seg}=[f_{1},f_{2},...f_{v},...]^{T}$, where $f_{v}$ is the feature representation vector for the single text segment $v$.

The GCN segment classification results for the text segment $v$ can be represented by:

where $\mathcal{N}(v)$ means the neighbor segments in the graph structure $\mathcal{V}$ of text segment $v$, and $\mathcal{W}$ and $b$ are the weights of the graph convolution. $\varphi$ is the non-linear activation function. Eq. (6) is the node-level graph feature aggregation of Eq. (4).

After feature aggregation, the relational reasoning results can be used to combine text segments that have a strong relationship. The weakly supervised node classification results are used to refine the detection results inherited from the previous FPN layer. Thus, the text segment type reasoning prevents detection errors from being further accumulated. Both link prediction and node classification make inferences based on the high-dimensional relational features on the same local graph, and can benefit each other when sharing the same weights. The node classification steps further classify Text Segments and Non-Text Segments, reducing the number of false text segments being incorrectly linked in link prediction.

The relational features from GCN layers provide a ready-made high-dimensional representation of long-range dependency depicting the relational connection of text segments. We fuse them with the visual features to complement the long-range dependency between text segments for generating graph-guided text regions. However, the dimensionality of the relational features is different from that of the visual features, which means they cannot be fused directly.

To address this issue, we propose Graph features’ Location-Aware Transfer (LAT) to reconstruct the graph convolutional feature of $i$-th GCN layer $F_{seg}^{i}$ in Eq. (4) with consideration of the location information of each node in the graph. The relational features after GCN contain the structural information of the graph in its first two dimensions. For example, the relational feature $F_{seg}^{i}\in\mathbb{R}^{N\times M\times D}$, where $N$ is the total number of nodes (i.e., text segments) in the input image, $M$ is the number of 1-hop nodes, and $D$ is the dimension of GCN features which is similar to the channel space in the convolutional layer. Our goal is to reshape the $F_{seg}^{i}$ of dimension ${N\times M\times D}$ to ${C\times H\times W}$, where $C$, $W$ and $H$ are the channel, width and height of the input image. The core idea is to fill in the graph features in the corresponding region of the transferred map based on the location information of each text segment. Namely, building connections of the nodes on the graph structure with their location on the image, as the location of each node is also translation invariant compared to the visual feature. The transferred feature $T_{i}\in\mathbb{R}^{C\times H\times W}$ for $F_{seg}^{i}$ after LAT can be calculated with Algorithm 1.

First, we calculate the geometric feature $(x_{k},y_{k},h_{k},w_{k},\theta_{k})$ of text segment $S^{k}$ from the geometric feature map $F_{geo}$. Here, we only consider transferring the graph features of Char/Interval Segments. Then, we can find the bounding box (bbox) of each text segment on the feature map of $T_{i}$ according to $(x_{k},y_{k},h_{k},w_{k},\theta_{k})$. For each point ($x,y$) in the bbox, we fill the flattened graph feature into $T_{i}$.

Since FPN is unable to reason the relationship between different text segments, we introduce a multi-modal Fusion Decoding (FD) module to capture the long-range dependency between individual text regions according to their relational features as well as FPN features. The final goal is to generate a Graph Guided Text Region (GGTR) map for further FPNS.

As shown in Fig. 6, the proposed FD module produces the final fused feature $F_{i+1}$ as: \add

where $CBR$ denotes $Conv_{3\times 3}$, Batch Normalization and ReLU, \add$CR$ donates $Conv_{1\times 1}$ and ReLU, and \add$UP$ denotes two times of up-sampling. Finally, we apply the $Conv_{1\times 1}$ on $F_{3}$ to obtain a GGTR map, which will be used as a visual-relational guide in the inference stage.

The overall objective function consists of six parts and is formulated as:

where $L_{GGTR}$ and $L_{TCL}$ are the cross-entropy losses from the GGTR and TCL maps, respectively, and $L_{H}$ and $L_{\theta}$ are Smoothed $L1$ losses for Height and Angle. We adopt OHEM [43] for training $L_{TCL}$, and the ratio of positive sample and negative sample areas is set to $1:3$. The linkage loss $L_{Link}$ is the cross-entropy between the ground truth linkage relationship of text segments and $\hat{r}$ in Eq. (5).

For the text segment classification loss $L_{Node}$, since our annotation is obtained with weakly supervised training, we only consider the text segments that are labeled as Char Segments, Interval Segments and Non-Text Segments (denoted as $f^{v}_{c}$, $f^{v}_{i}$, and $f^{v}_{n}$ respectively) and ignore those that have not been labeled. Thus,

where $\mathcal{N}$, $\mathcal{M}$, and $\mathcal{O}$ are the collections of all Char Segments, Interval Segments and Non-Text Segments, respectively.

The Graph Guided Text Region map and the text segment classification results from the GCN are then used to rectify the TCL map to remove false detection. The relationship prediction and the dense overlapping text segments together ensure the completeness and accuracy of using the contour of the grouped text segments to approximate the contour of the text. The connection of the text areas is ensured by the “characterness” and “streamline” of our dense overlapping text segments. The overlapping of text segments also reflects their connectivity, ensuring that the shape of the grouped segments can be used to approximate the contour of the original text instances accurately. In our work, instead of jointly rectifying multiple text maps as in [11, 12, 23], we decompose the FPNS problem based on the grouped text segments reasoned by visual-relational features. Therefore, our approach does not use element-wise multiplication or search the entire map, which avoids accidentally removing true positive text areas.

Algorithm 2 illustrates our proposed shape-approximation algorithm. The inputs require the text maps of GGTR, TCL, H, W and $\Theta$, as well as node classification $GCN_{node}$ and link prediction components $GCN_{link}$ from the GCN. First, we perform a union operation on the TCL map and shrink the GGTR map to produce a rectified TCL map $TCL^{\prime}$. The text area of the GGTR map is shrunk from the top and bottom towards the middle until single-pixel height is reached. With this step, the GGTR map is introduced to provide the long-range dependency of text segments and also provide more candidates for the graph reasoning stage. The function $cv2.findContours$ is used to find the potential text center lines of the text instances. Following that, for each text instance, the $Getboxes$ function is used to generate dense text segments for pixels in the rectified TCL map. The classifications from $GCN_{node}$ are used here to suppress false detection of text segments, and the remaining text segments are stored in the list $\mathcal{S}$. Then, the link prediction components from $GCN_{link}$ are used to restore the order of text segments that are grouped into $ConnectedSegments$ based on their linkage relationships.

For shape approximation (SAp), we first draw the text segments on a blank image and then use the contour of these overlapping text segments to approximate the original text contour. To connect text segments with each other more closely, an opening operation with a kernel size of $3\times 3$ is performed. Similar to [12, 23], the GGTR map is used to further suppress the text instances with IoUs less than 0.5.

It is worth noting that our inference method discards the route-finding process of existing bottom-up methods for determining the visiting order of text contour points. Instead, we generate dense text segments to approximate the text contour more accurately with only one NMS processing step.

Our algorithm is implemented using PyTorch 1.7. The VGG16 is pre-trained with ImageNet, with FPN adopted for multi-scale feature extraction. We conducted our experiments on an RTX3090 GPU with 24GB memory. All images used for training and testing are of a single scale. For training, the images are resized to $640\times 640$ for images from CTW1500 and Total-Text, and $1280\times 1280$ for images from ICDAR2015, MSRA-TD500 and MLT2017. Data augmentation techniques including rotation, random cropping, color variations, adding random noise, blur, and changing lightness were adopted. The batch size was set to 10.

We first used SynthText to pre-train our model for 10 epochs using Adam optimizer and a learning rate of $0.001$. Then, we fine-tuned our model on real text benchmark datasets for 800 epochs with SGD optimizer and a learning rate of $0.001$. The momentum of SGD was set to $0.9$. For testing, the short side of images was kept at $640$ pixels for CTW1500 and Total-Text, and 1,280 pixels for ICDAR2015, MSRA-TD500 and MLT2017, while retaining their aspect ratios.

Examples of the text detection results obtained with our proposed method are shown in Fig. 7. Table I shows quantitative comparisons of the detection results with the state-of-the-art approaches on all mainstream benchmark datasets, i.e., CTW1500, Total-Text, ICDAR2015, MSRA-TD500 and MLT2017, respectively. The “-” in the table indicates that the comparative method did not report results on the dataset.

Curved text detection: Our results on CTW1500 and Total-Text, the two most representative benchmark datasets of arbitrary shape text, demonstrate the effectiveness of our algorithm in dealing with the highly curved and spatially separated texts. As shown in Table I, our method has achieved F-measure scores of $86.4\%$ and $87.6\%$ on these two datasets, respectively, which outperforms all existing state-of-the-art (SOTA) methods. Specifically, our method remedies the shortcomings of the bottom-up methods [9, 12, 8] and can achieve SOTA results with only a VGG16 backbone without introducing deformable convolutional blocks to increase the backbone network’s feature extraction ability as in [8, 35, 20, 36, 37]. Compared with the SOTA top-down methods [34, 20], our method shows improvements of 0.9% on CTW1500 and 0.7% on Total-Text respectively, thanks to our proposed false-detection suppression and shape-approximation strategies, as well as the dense overlapping design of text segments. Different from [10] requiring a weakly supervised instance segmentation to predict the mask and class of each character, our method focuses on classifying the types of text segments without extra segmentation. \addFig. 8 qualitatively compares the visual detection results of our proposed method with results obtained by the GCN-based bottom-up approach DRRG [12] and Mask RCNN based top-down method TextFuseNet [10] on some challenging text instances in CTW1500 and Total-Text. As shown in this comparison, our proposed false-detection suppression and shape-approximation strategies are effective in handling false detection and the route-finding failure.

The SOTA results obtained on these two datasets support our claim that the relative underachievements of existing bottom-up methods are not caused by the limited feature capturing ability of the text proposal backbones or GCN. Also, bottom-up methods can be superior to top-down methods when adopting our proposed false-positive/negative suppression and shape-approximation strategies in the inference stage.

Multi-oriented text detection: On the dataset ICDAR2015, as shown in Table I, our method has also achieved a comparable SOTA result of an F-measure of $89.5\%$, outperforming some high-performing top-down methods such as FCENet [20] and ContourNet [11] by more than 2.5 percentage points.

Our result is slightly lower than that of the TextFuseNet method [10]. We argue that TextFuseNet adopted an instance segmentation strategy, which introduced classification (recognition) to enhance detection results. Also, the approaches of [9, 8, 10, 11] have adopted a multi-scale training/testing strategy, which is a widely-known tip used on ICDAR2015 that can boost the detection accuracy significantly on this dataset, whereas we only use single-scale training and testing rather than multi-scale training or testing for a fair comparison.

On the dataset MSRA-TD500, as shown in Table I, our method has equalled the SOTA result with an F-measure of $87.0\%$. Moreover, thanks to our proposed FPNS strategy, our method has achieved the highest recall rate of $83.8\%$. This shows the effectiveness of our idea of depicting the streamline of the texts, which helps to retrieve some misdetected texts. As our approach focuses on detecting arbitrary-shape text, it is a common situation that arbitrary-shape text detectors cannot significantly improve the SOTA results in both multi-oriented detection benchmarks, especially ICDAR2015, which exhibits large scale variations of texts. For example, the arbitrary-shape text detectors [36, 9, 12, 8, 11, 29, 20, 37, 21] can only achieve SOTA results on at most one multi-oriented detection benchmark. The arbitrary-shape text detectors focus more on the space variation of texts, whereas multi-oriented text detectors focus more on scale variation.

Multi-lingual text detection: As shown in Table I, on the multi-lingual scene text dataset MLT2017, our network has also surpassed the SOTA method SDM [34] with a Precision of $82.9\%$, a Recall of $74.0\%$ and an F-measure of $78.2\%$. This is because the dense design of the text segments can effectively depict the ‘characterness’ property that non-Latin texts also exhibit. Moreover, our proposed FPNS and SAp strategies enable the network to accurately identify connectivity within multi-lingual texts. The highest F-measure demonstrates that our method has good stability in detecting multi-lingual texts.