Turning a CLIP Model into a Scene Text Spotter

Segmentation-based Methods. Segmentation-based techniques typically perform operations at the pixel, segment, or contour level, subsequently grouping these into text instances. Notable methods include the Segment Linking (SegLink) by Shi et al.[10], using a fully-convolutional neural network for detecting segments and links; the TextSnake by Long et al.[11], an adaptable approach for detecting text of varying shapes; and the Progressive Scale Expansion Network (PSENet) by Li et al. [12], generating diverse kernel scales for each text instance. Efficient and accurate systems like the Pixel Aggregation Network (PAN) developed by Wang et al.[13] have emerged, combining a low computational-cost segmentation head with a learnable post-processing system. Additionally, unique methods such as the Differentiable Binarization (DB) module introduced by Liao et al.[14] incorporate the binarization step directly into the segmentation network. Meanwhile, the transformer-based architecture proposed by Tang et al. [15] performs detection based on select representative features to decrease computational cost and reduce background interference. These methods underscore the wide range and adaptability of segmentation-based techniques in text detection. Further, Long et al. [16] introduced an end-to-end model capable of performing unified scene text detection and visual layout analysis simultaneously.

Regression-based Methods. Regression-based methods view text as a single object and directly regress the bounding boxes of the text instances. Zhang et al.[17] propose a multi-oriented text detection method utilizing Fully Convolutional Networks, which uses both local and global cues to locate text lines. Liu et al.[18] develop the Deep Matching Prior Network (DMPNet), using quadrilateral sliding windows and a sequential protocol for regression to predict text with a compact quadrangle. He et al.[19, 20] introduced models for text detection that utilize a regional attention mechanism and deep direct regression to predict the text bounding box. Liao et al.[21] create a unified deep neural network for natural image text detection, and Zhou et al. [22] designed the EAST model that predicts words or text lines of any orientation and quadrilateral shape in full images. Innovative methods like LOMO by Zhang et al.[23], and the adaptive text region representation by Wang et al. [24] have also been developed. Zhu et al.’s FCENet [25], Liu et al.’s MOST [26], and Zhang et al.’s adaptive boundary proposal network [27] further contribute to the field by introducing novel concepts and methodologies. Dai et al.[28] use a progressive contour regression strategy, and Ye et al.’s DPText-DETR[29] employs explicit point coordinates and an enhanced self-attention module. Zhang et al. [30] present a unified coarse-to-fine framework for text detection using an iterative boundary transformer.

Regular End-to-end Scene Text Spotting. Li et al. [31] propose one of the earliest end-to-end trainable scene text spotting methods. Their approach effectively merged detection and recognition features using RoI Pooling [32] in a two-stage framework. Originally designed for horizontal and focused text, their method showed significant performance improvements in an enhanced version [33]. Busta et al. [34] made contributions to the field with their end-to-end deep text spotter. In further advancements, He et al. [35] and Liu et al. [36] incorporated anchor-free mechanisms to enhance training and inference speed. They employed different sampling strategies, such as Text-Align-Sampling and RoI-Rotate, respectively, to extract features from quadrilateral detection results.

Arbitrarily-shaped End-to-end Scene Text Spotting. Liao et al.[37] introduced Mask TextSpotter which uses Mask R-CNN with character-level supervision to detect and recognize arbitrarily-shaped text. Mask TextSpotterv2[38] reduces reliance on character-level annotations. Qin et al.[39] employ RoI Masking for attention on arbitrarily-shaped text regions. Feng et al.[40] utilize RoISlide for handling long text, whereas Wang et al.[41] focus on boundary points detection, text rectification, and recognition. CharNet by Xing et al.[42] also caters to arbitrarily-shaped text spotting. Liao et al.’s Segmentation Proposal Network (SPN)[43] and Liu et al.’s ABCNet [44] are other noteworthy contributions. ABINet++ by Fang et al. [45] innovatively uses a vision model and a language model with an iterative correction mechanism. Huang et al.’s SwinTextSpotter [46] uses a transformer encoder for detection and recognition. Approaches based on DETR [47] and variants [48] for RoI-free scene text spotting have also shown promising results. TESTR [49] uses an encoder and dual decoders, while TTS [50] uses a transformer-based approach. SPTS [51] employs a single point for each instance and uses a Transformer to predict sequences. DeepSolo [52] allows a single decoder to perform text detection and recognition.

Detection-only Results. As demonstrated in Tab. II, we noted consistent enhancements in F-measure on text spotting benchmarks when TCM-CR50 was combined with five distinct text spotting methods. Particularly, TCM-CR50 outperformed the baseline methods such as MTSv3, ABINet++, ABCNet, DeepSolo, and TESTR with an R50 backbone, with performance boosts ranging from +0.2% to +1.8% in terms of F-measure on the TT dataset. Consistent improvements were also witnessed on IC15 and CTW datasets, underscoring TCM-CR50’s suitability for text-spotting methods. Furthermore, when FastTCM-CR50 was integrated with MTSv3, ABINet++, ABCNet, DeepSolo, and TESTR, we observed an average performance enhancement of 0.2% compared to TCM-CR50 based methods, accompanied by similar speed improvements, indicating FastTCM-CR50’s superior efficacy. Additionally, the inclusion of an extra large-scale dataset, TextOCR, resulted in further performance gains, such as a 0.9% improvement on the TT dataset using TESTR.

End-to-end Spotting Results. In Tab. II, we present the end-to-end spotting performance of our method combined with existing scene text spotters. TCM-CR50 demonstrates favorable performance when integrated with various cooperative methods. Specifically, under the end-to-end setting with the strong lexicon on dataset IC15, TCM-CR50 outperforms the original MTSv3, ABINet++, ABCNet, DeepSolo, and TESTR by +0.8%, +0.3%, +2.3%, +0.1%, and +0.4%, respectively, in terms of the ‘S’ metric. Similar consistent improvements are also observed for datasets TT and CTW, indicating that TCM-CR50 effectively enhances the performance of both existing scene text detectors and spotters. Furthermore, when replacing TCM-CR50 with FastTCM-CF50, we observe a further improvement in performance, with an average gain of 1.5% compared to baseline methods and an average gain of 0.56% compared to TCM-CR50. Additionally, the inference speed of FastTCM-CR50 is increased by approximately 46.4%. These results highlight the superiority of FastTCM-CR50 and its potential for efficient and accurate text spotting tasks. Besides, when using additional large-scale TextOCR as training data, our model can achieve further improvement, suggesting the compatibility of our method with large-scale datasets.

Results for Text Detection Task. To verify the few-show training ability of our method on text detection tasks, we directly train our model on real datasets using various training data ratios without pretraining, and evaluate it on the corresponding 4 benchmarks. As shown in Fig. 7, DB-FastTCM-CR50 shows robustness on limited data and outperforms the baseline methods DB in an average of 26.5% in terms of 10% training data ratio settings. Besides, DB-CR50 has limited improvements compared to our specific design FastTCM. The results show that the FastTCM can capture the inherent characteristic of text via leveraging the pretrained vision and language knowledge of the zero-shot trained CLIP model.

Few-shot Experiments for Text Spotting. In addition, we performed few-shot experiments on text spotting tasks using ABCNet and TESTR on Total-Text, as illustrated in Tab. III. Considering that the recognizer module in text spotting methods often struggles to learn effectively with very limited data, we followed the text spotting pretraining step to obtain a suitable initialization for the corresponding text spotting methods. Subsequently, we applied different training ratios of the Total-Text dataset to evaluate the few-shot learning ability. The results demonstrate that both TCM-CR50 and FastTCM-CR50 exhibit advantages in few-shot learning for text spotting tasks compared to DB-R50 and DB-CR50. Moreover, using our method outperforms baseline methods by an average of 5.5%. This demonstrates the effectiveness and superiority of FastTCM-CR50 over simply replacing other counterparts.

CLIP Backbone Generalization. We conducted an experiment to investigate the generalization performance of DBNet by directly replacing the backbone of DBNet with CLIP backbone (CR50), as shown in Tab. IV. It shows that the CLIP-R50 can indeed bring benefits for generalization. However, by integrating with FastTCM-CR50 backbone, the performance can be significantly improved. It suggests that directly using the pretrained CLIP-R50 is not strong enough to improve the generalization performance of the existing text detector, which further indicates that synergistic interaction between the detector and the CLIP is important. Meanwhile, FastTCM-CR50 also consistently outperforms TCM-CR50.

Synth-to-real and real-to-real Adaptation. We conduct two types of experiments including synthtext-to-real adaptation and real-to-real adaptation on text detection tasks, as shown in Tab. IV and Tab. V, respectively. Real-to-real adaptation contains monolingual and multi-lingual scenarios. From the tables, we can see that by integrating the FastTCM-CR50 into DBNet, we significantly improve the performance by an average of 12.4% in terms of F-measure for four different settings including synthtext-to-real and real-to-real, which further demonstrates the effectiveness of our method for domain adaptation. Notably, FastTCM-CR50 also consistently demonstrates improvements by an average of 0.4% compared to TCM-CR50 as well, further emphasizing the remarkable generalization ablity of our methods.

Real-to-Real Adaptation on Scene Text Spotting. Besides, we also conducted real-to-real adaptation experiments with existing spotting methods, as shown in Tab. VI. The results show that the FastTCM-CR50 has the capacity of improving the existing scene text spotting methods by an average of 14.8%, further demonstrating the effective generalization ability.

Ablation Study for the Predefined Prompt. When using the predefined prompt, as illustrated in the second row of Tab. VIII, the performances are slightly improved on all four datasets (IC15, TD, TT, and CTW), with 0.05%, 0.2%, 0.04%, and 0.1% higher than the baseline method, respectively.

Ablation Study for the Learnable Prompt. Then, results combing the learnable prompt with the predefined prompt on four datasets are provided in the third row of Tab. VIII. We notice that a consistent improvement can be achieved by adding the learnable prompt. We also show the influence of using different numbers of the learnable prompt in row 4 to row 6 of Tab. VIII. We observe that as the value of the number of the learnable prompt increases, the performance increases gradually on all datasets. Compared to the value 4, the value 32 obtains obvious improvements on CTW, TD, and TT. We conjecture that this is because the larger number of the learnable prompt can better steer the pretrained text encoder knowledge which is useful for text detection. In the following experiments, the default number of the learnable prompt is set to 4 for simplicity.

Ablation Study for the Language Prompt Generator. Besides, we evaluate the performance of the proposed language prompt generator shown in 7_th row of Tab. VIII. With the help of the language prompt generator, we find that TCM achieves further improvements on all four datasets, especially on ICDAR2015, indicating that the conditional cue generated by the language prompt generator for each image can ensure better generalization over different types of datasets.

Ablation Study for the Visual Prompt Generator. Furthermore, combining the proposed visual prompt generator with the above other components, the improvement of F-measure is better than the baseline on all four datasets, with larger margins of 1.7% and 2.0% on IC15 and TD, respectively. The reason for this obvious complementary phenomenon is that the visual prompt generator can propagate fine-grained visual semantic information from textual features to visual features. Besides, the prompted locality image embedding generated by the visual prompt generator can guide the model to obtain more accurate text instance-level visual representations, which boosts the ability of instance-language matching and generates a precise segmentation score map that is useful for downstream detection head.

Ablation Study for the Bimodal Similarity Matching. We further conducted a comparison of the results with and without bimodal similarity matching, as outlined in the last group of Tab. VIII. The results clearly demonstrate that the utilization of bimodal similarity matching leads to higher performance. This finding indicates that bimodal similarity matching plays a crucial role in training the model by dynamically enriching text embeddings with visual information, resulting in improved overall performance.

Ablation Study for the Auxiliary Loss. We compare the results of with and without auxiliary loss, as shown in Tab. VIII. We observe that using auxiliary loss achieves higher performance. The results indicate auxiliary loss is beneficial to train the model via imposing constraints on instance-language matching score map. In addition, the improvement of the performance suggests that it might help the image encoder of pretrained CLIP to perceive locality text regions effectively.

Ablation Study for the Key Component on Generalization Performance. As presented in Tab. IX, removing the meta query and BSM elements from FastTCM dramatically deteriorates the generalization performance, highlighting the importance and effectiveness of these components. Similarly, removing the VG and LG elements from FastTCM also results in a substantial drop in generalization performance, further validating their effectiveness. Finally, when we remove all of these components, the performance experiences an additional significant drop, indicating that each of these components contributes to the overall effectiveness and performance of FastTCM-CR50.

Ablation Study for the Parameters Comparison. For a fair comparison, we have increased the parameters of DBNet by replacing the backbone with a larger ResNet and then conducting text detection experiments on TD dataset and a domain adaptation experiment on IC13 $\rightarrow$ IC15. Trainable parameters and FLOPs are calculated with an input size of 1280$\times$800. Results are shown in Tab. X. The results show that DBNet with FastTCM-CR50 has better performance than DBNet with less model size and computation overhead compared to DBNet with R152 backbone, demonstrating its effectiveness.

Ablation Study for the number of transformer decoder layers of VG. The last group of Tab. X demonstrates the impact of varying the number of transformer decoder layers of VG on the performance. The results show that the performance remains robust across different numbers of decoder layers. This indicates that in practical applications, we have the flexibility to decrease the number of transformer decoder layers to achieve a better trade-off between model parameters and performance. By reducing the number of layers, we can potentially save computational resources and memory while maintaining satisfactory performance, making the model more efficient and practical for real-world applications.

Ablation Study for the Different Predefined Language Prompt. We conducted ablation study on the predefined language prompt with different strings using DBNet with FastTCM-CR50 in Tab. XI. Results show that without predefined language prompt, the performance is harmed. In addition, it can be seen that there is little performance variation with different predefined language prompt. When the predefined language prompt becomes long and complex, the model performance drops a little. We deem that the CLIP is not good at handling complex instructions because it is pretrained on a dataset of 400 million image-text pairs that contain noise. As a result, this noise can affect the CLIP’s ability to deal with long instructions.

Ablation Study for Different Amount of Data. To further explore whether the FastTCM can learn the additional knowledge which is hard to be obtained from increasing data, we have trained the model on large-scale public joint data including IC13, IC15, TD, CTW, TT, and MLT17, with a total of 13,784 images, and testing it on a NightTime-ArT data (326 images) carefully collected from ArT. The nighttime examples of ArT are provided in appendix. Results are shown in Tab. XIII. The results show that even with the addition of large amounts of training data, existing methods still show limitations to the nighttime data that is obviously out-of-distribution from the training set. However, integrating FastTCM-CR50 can still perform robustly in such cases, indicating its robust generalization ability.