VGTS: Visually Guided Text Spotting for Novel Categories in Historical Manuscripts

For the input support and query images, $(I_{s},I_{q})$, are passed through two feature extraction CNN branches with shared weights. Since most of the antique manuscripts are insufficient to train the network from scratch, the feature extractor can only gain experience from the base category. It cannot get targeted training from the novel category. For each input support image $I_{s}^{i}$, the feature map $f_{\hat{s}}^{i}\in R^{h_{\hat{s}}^{i}\times w_{\hat{s}}^{i}\times D}$ obtained after backbone, is uniformly resized to the same size $f_{\hat{s}}\rightarrow f_{s}\in R^{h_{s}\times w_{s}\times D}$. Finally, the feature extraction networks extract the support feature $F(I_{s})=f_{s}\in R^{h_{s}\times w_{s}\times D}$ and query feature $F(I_{q})=f_{q}\in R^{h_{q}\times w_{q}\times D}$, respectively, where $F$ represents the backbone, $h\times w$ is the spatial resolution and $D$ is the channels.

To ensure the model can have sufficient generalizability to unseen categories. Intuitively, the model must fully avail of the reference information provided by the support image. A common strategy for computing the correlation map is to match a pair of individual feature maps using normalized cosine similarity [Rocco et al., 2017, Liu et al., 2020b]. Given a pair of individual features $f_{s}\in R^{h_{s}\times w_{s}\times D}$ and $f_{q}\in R^{h_{q}\times w_{q}\times D}$ of support and query images, the correlation map is computed as:

where $\phi(\cdot)$ means cosine similarity, and $C_{abkl}$ denotes the matching score between the $(a,b)$-th position in query feature map and $(k,l)$-th position in support feature map. The result is thus a 4D tensor that captures the similarities between all pairs of spatial locations.

When humans are guided by support images, during a glance, human will remember the most critical salient regions that contribute to visual spotting. Besides, when we need to find a character in a page of a book, blank areas or illustrations are ignored in the process of visual spotting. Thus, the spotting area is narrowed to concentrate on potential regions of interest within the query image, which means human will focus potential discriminative area.

Step 1. Refinement for the support image. Given the correlation map $C_{r}\in R^{h_{q}\times w_{q}\times d}$ by the correlation matching module, where $d=(h_{s}\times w_{s})$. The correlation map not only contains all information from the support and query images, but also captures the similarity between the two images. For correlation map $C_{r}$, each channel $d_{j}$ of $C_{r}$ represents the correlation of the $j$-th pixel on the feature map $\{f_{s}\}_{j}$ with $f_{q}$, we produce a 1d spatial-wise attention map to focus “where” is meaningful localization in the support image, as shown in Fig. 3. We first feed the obtained $C_{r}$ into the dual spatial attention block to find regions with key discriminative properties in the support image. Aggregating the spatial information of the correlation map $C_{r}$ by using average-pooling and max-pooling operations, generating two different spatial context descriptors $f^{avg}_{c}$ and $f^{max}_{c}$. After that, features pooled by each pooling layer are then passed through the support refinement network, which consists of two convolutions layer with kernel size 1. The support refinement network analyzes the correlation map $C_{r}$ to identify the most important regions in the support image by learning the relationships between different channels. Then the element-wise summation is used to merge the results which have:

where $W_{0}\in R^{d\times(d/\tau)}$, $W_{1}\in R^{(d/\tau)\times d}$ are parameters of convolutions, $\sigma_{s}$ represents the sigmoid activation function and $\tau$ is the reduction ratio. Then multiply $M_{s}\in R^{1\times 1\times d}$ with the correlation map $C_{r}$, expressed as:

where $C_{r}^{s}$ captures the key points in spatial dimension of the support image and $\otimes$ denotes element-wise multiplication.

Step 2. Refinement for the query image. Note that, at a particular position $(a,b)$ of $C_{r}^{s}$ contains the similarities between $f_{q}(a,b)$ and all the features of $f_{s}$. As illustrated in Fig. 3, we also generate a 2d spatial attention map to focus on the positional informative key point for the query image. By using average-pooling and max-pooling operations along the channel dimension and concatenating the pooled features. Then a convolution layer is applied to generate the spatial attention map:

where $\sigma_{q}$ represents the sigmoid activation function, and $Conv$ represents a convolution operation with the filter size of $3\times 3$. To obtain the final refined features, we multiply $M_{q}\in R^{h_{q}\times w_{q}\times 1}$ with $C_{r}^{s}$, which can be expressed briefly as:

where $C_{r}^{sq}$ captures the key points in the spatial dimension of the query image and retains the key points focus of the support image.

For the obtained attention correlation map $C_{r}^{sq}$, the geometric matching block is applied to find the spatial correspondence between $I_{s}$ and $I_{q}$. To achieve this, we optimize a transformation function $W_{\theta}:\mathbb{R}^{2}\rightarrow\mathbb{R}^{2}$, where $\theta$ denotes the alignment parameter. The spatial coordinates between $I_{q}$ and $I_{s}$ can be found by $(k’,l’)=W_{\theta}(k,l)$, where $(k’,l’)$ are the corresponding spatial coordinates of $(k,l)$ in $f_{q}$. Similar to [Rocco et al., 2018], the geometric matching block consists of two consecutive convolutional layers without padding and stride equal to 1, with batch normalization and ReLU connected between the two convolutional layers, and finally a fully-connected layer to regress the alignment parameter $\theta$. The goal of the GM block is to find the best parameters $\hat{\theta}$:

where $\underset{(k,l)}{\sum}\phi(f^{q}_{W_{\theta}(k,l)},f^{s}_{k,l})$ means the summation of the feature similarities between the support feature and the regions corresponding to those on the query image obtained through the 2D geometric transformation. After that, the obtained transformations are input to the grid sampler to generate a grid of points, which is aligned with the support image at each location of the query image.

Note that, the process of location is already done implicitly in the alignment process, by extracting the position frames based on the maximum and minimum values of the obtained grid tensor, by simply outputting a square tight bounding box based on the transformed grid points. The spotting score can be obtained by computing $\underset{(k,l)}{\sum}\phi(f^{q}_{W_{\theta}(k,l)},f^{s}_{k,l})$, i.e., the similarity score of the local region on the query image feature $f^{q}_{W_{\theta}(k,l)}$ to the input support image feature $f^{s}_{k,l}$.

Initially, it is hypothesized that certain one-shot object detection methods could also address the task of one-shot text spotting. To test this hypothesis, we retrained some popular few-shot object detection methods [Fan et al., 2020] using the four datasets previously summarized. Unfortunately, the experimental performance fell short, struggling to surpass 8$\%$ mAP. This is mainly due to the fact that historical manuscripts usually comprise a large number of characters, frequently exceeding 100 different categories. These characters possess similar glyphs or significantly vary in scale, thereby presenting significant challenges for one-shot text spotting. Although we attempted to use the popular object detector method Faster RCNN [Ren et al., 2017], the paucity of training data made it difficult to achieve meaningful results. Therefore, we opted to use OS2D [Osokin et al., 2020], a dense correlation matching feature-based method for one-shot object detection, as our baseline. This approach trains the transform network from a model pre-trained with weak supervision. As there has been no prior research on one-shot text spotting, we adapted OS2D [Osokin et al., 2020] to suit our historical manuscripts by fine-tuning the method’s parameters and using the same data augmentation scheme as our proposed method. Although CoAE’s intended purpose is to detect objects in natural scenes, it has been included as one of the comparison methods for its efficacy in one-shot object detection using the RPN approach in recent years. To make a fair comparison, we have also added multi-scale pyramids to the training and testing of CoAE. Furthermore, we have experimented with DSW [Wicht et al., 2016], a classic unsupervised approach for text spotting based on the “sliding window” technique, and reproduced its main idea by directly mapping the feature map of the support image as a convolution on the query image after extracting the features using a Siamese network with specifications identical to our method.

The statistical results for the Novel class across various datasets are presented in Table 1. The Novel class denotes an unseen category absent during training. In contrast, the statistical outcomes for the Base class across different datasets are displayed in Table 2, with the Base class signifying a seen category present during training.

Experimental results reveal that our method attains the highest recall score across all datasets, suggesting a low propensity for missed text spotting occurrences. Additionally, our proposed method surpasses the other methods in mAP and F1 scores. It is pertinent to acknowledge that the sliding window method may be deemed an unsupervised technique, and the accurate aspect ratio of support image feature maps is provided for this approach. Nevertheless, the performance of the unsupervised sliding window method is inferior to supervised learning methods, such as OS2D and VGTS. Moreover, the unsupervised DSW method occasionally outperforms the supervised CoAE method, likely due to inadequate data for comprehensive CoAE learning. Regarding data distribution, the DBH, EGH and NC datasets exhibit a more dispersed inter-class character distribution in manuscript images, whereas the VML and TKH datasets display a more compact inter-class distribution and smaller character size, which CoAE struggles to accommodate. With the EGH, VML and TKH datasets containing numerous similar characters, our VGTS method outperforms the baseline method. We also observed that performance on novel categories occasionally exceeds that of base categories, a phenomenon evident across multiple methods. This observation may be attributed to the relatively smaller data volume for novel categories, which can significantly impact their precision and recall rates during evaluation. Specifically, when a method accurately spots limited samples of novel categories, precision may increase, resulting in higher Average Precision (AP) for those categories.

During training, VGTS learns to match potential query image regions with the support character instance rather than accumulating class-specific knowledge. This learning strategy facilitates effective performance with limited training samples, crucial in the context of historical manuscript data featuring small, sparse datasets. By circumventing the need for an extensive training sample collection, the proposed method distinguishes itself from other one-shot methods that require larger sample volumes to achieve similar results.

Similar Characters Challenge. Misidentification can occur due to the structural similarity between characters, particularly when characters in ‘Novel’ categories bear a high resemblance to those in ‘Base’ categories. To assess this, we specifically selected ‘Base’ and ‘Novel’ categories, which consist of morphologically similar characters from the TKH dataset. As indicated in Table 3(d), VGTS maintains high-performance metrics.

Long-tailed Distribution Challenge. A challenge in text spotting in historical manuscript manuscripts is their long-tail distribution, with some characters appearing only once. Data from the TKH training set indicates that over 58% of characters appear less than five times. This poses significant difficulties in training models, especially in capturing features of low-frequency characters. Fig. 5 compares the effectiveness of various methods in addressing this long-tail distribution issue. OS2D outperforms CoAE in high-frequency categories, while VGTS, our category-agnostic method, demonstrates higher mAP across all frequency categories, particularly in low-frequency ones such as ‘[1,5)’ and ‘[5,10)’. This capability of VGTS is crucial for handling long-tail distributions, indicating its effectiveness in processing rare categories.

Rotational Disturbance. Our proposed method demonstrates exceptional robustness against minor rotational variations in test images, eliminating the need for rotation augmentations during training. We subjected test images to rotational disturbances ranging from 0^∘ to 15^∘ at 2.5^∘ intervals. As depicted in Fig. 6, our method surpasses the comparative methods in managing rotated test images, experiencing only minimal performance degradation as the rotation angle increases. Conversely, the RPN-based model, CoAE, exhibits a rapid decline in performance when confronted with rotational disturbances. The OS2D model’s performance reveals a gradual degradation as the query image’s rotation angle increases. The unsupervised sliding window-based method appears to be minimally impacted by rotational disturbances.

Spotting Performance Under Varied Image Conditions. The support image quality substantially influences the localization and spotting performance. To demonstrate our method’s robustness under varying image quality conditions, we present visual results with low-quality support images in Fig. 7. Remarkably, our method consistently delivers outstanding performance, even when the support image is blurred, as illustrated in the first row of Fig. 7. Both OS2D and VGTS exhibit accurate character spotting when the support image is affected by illumination, as shown in the second row of Fig. 7.

We introduce Gaussian noise to the source images to simulate varying degrees of image degradation. The noise is added at variances corresponding to 15%, 25%, and 35% of the full intensity scale, respectively. A qualitative analysis is illustrated in Fig. 8, with error detections highlighted by red dotted boxes. It is observed from Fig. 8 that the model maintains robust performance under low to moderate levels of Gaussian noise. However, when the noise level reaches 25%, the model begins to falter, resulting in instances of both missed and false spotting. We provide a demo video³³3https://youtu.be/8GRDOxCDMjw ⁴⁴4https://www.bilibili.com/video/BV12x4y1K7MX/ that effectively demonstrates the effectiveness of our proposed method, comprising several brief clips.

In practical applications, character images with distinct glyph styles may be considered as support images for the spotting procedure. Quantitative analysis of this scenario is challenging; therefore, a qualitative assessment is provided in Fig. 9. The red box delineates the support image, while all predicted boxes are depicted in green. Within Fig. 9, it has been noted that using a support image with a different character style from the query image can produce favorable outcomes, thereby illustrating the model’s generalization capacity and transferability.

Spotting Novel Combination Characters. The proposed approach affords the flexibility to manage diverse and evolving application scenarios by enabling the spotting of novel classes. Specifically, we define a “word” as a combination of characters belonging to a novel class when a single character sample serves as the support image during training. The extensive annotation required to determine which characters can constitute a word renders quantitative experimental comparisons challenging. Consequently, we present qualitative experimental results that illustrate the proposed approach’s effectiveness. Impressively, our method delivers precise spotting performance for combination characters. Fig. 10 showcase the performance of the proposed method on combination characters, emphasizing its capacity to accurately spot novel classes. It is essential to acknowledge that users must manually add unseen characters to the support image gallery for novel character classes, as our approach cannot automatically detect unknown categories in query images.

Spotting All Characters. Figs. 11 and 12 offer a glimpse of VGTS’s visualization results on the DBH and EGY datasets and show its adeptness in identifying symbols from ‘Novel’ categories. Categories not part of the training are marked with blue boxes, while those that have been trained are highlighted with green boxes. VGTS possesses the capability to swiftly assist historians in cataloging character categories, this greatly benefits the decryption and study of historical manuscripts.

When confronted with a limited amount of historical manuscript data, training the network from scratch proves to be a daunting task. As evidenced by the experimental results presented in Table 4, it becomes clear that the ResNet-50 backbone pre-trained on ImageNet (R50-ImgNet) delivers the best performance. Historical manuscripts in various languages may display unique patterns of texture features. We also tested the ResNet-50 pre-trained on the SynthText dataset [Gupta et al., 2016] (R50-Synth) as the backbone, which similarly achieved high performance. Conversely, utilizing the ResNet-101 architecture did not yield superior results. Based on experimental observations, we opted for R50-ImgNet as the backbone for our model. Additionally, if we freeze the backbone to prevent feature learning, the model still performs quite well, achieving a 90.81$\%$ mAP on DBH Novel, as seen in Table 5.

To empirically validate the effectiveness of our proposed framework, we conduct a comprehensive ablation study on the dual spatial attention block. Specifically, we perform one of the following operations at a time: (1) interchanging the order of the two spatial attention operations, and (2) removing one of the spatial attention operations. We evaluate all the models on the DBH dataset, and the results are presented in Table 6. The human cognitive process typically involves examining the support image first and then seeking relevant similar regions in the query image. By altering the order of spatial attention in VGTS, with query spatial attention preceding support spatial attention, we observe a decrease in mAP scores by approximately 0.7$\%$ and 3$\%$ for the novel and base categories, respectively. The experimental results attest to the effectiveness of the proposed module sequence. Eliminating the spatial attention for support images resulted in a decrease of about 1.7$\%$ in mAP scores for the base category, and removing the spatial attention for query images led to approximately a 3.4$\%$ decrease in mAP scores for the novel category, providing evidence that the spatial attention for both support and query images plays a crucial role in achieving stable and significant improvements.

In contrast to widely-used object detection datasets such as COCO [Lin et al., 2014] and VOC [Everingham et al., 2010], historical manuscript images present unique challenges, including smaller sample sizes, a larger number of categories, and considerable similarities between categories. To assess the efficacy of our approach in low-resource scenarios, we conduct experiments by varying the number of base categories in the training set while maintaining the number of input images constant. Fig. 13 presents a performance comparison between our proposed method and OS2D on the TKH dataset, showcasing results for varying numbers of training set categories, ranging from 600 to 1392. As illustrated in Fig. 13, the number of categories in the training set significantly influences the performance outcomes. Specifically, when the number of participating categories is reduced by 792, there is a noticeable impact on the F1 scores. For OS2D, this reduction leads to a decrease of 4.83% on the TKH Set 1 and 4.32% on the TKH Set 2. In contrast, for our VGTS method, the F1 scores decline by 2.87% on the TKH Set 1 and 3.06% on the TKH Set 2.

To investigate the influence of the number of input images on the model’s performance, we further evaluate the model on training sets comprising 40, 80, 120, and 171 images, corresponding to a total of 1104, 1232, 1320, and 1942 categories, respectively. The findings depicted in Fig. 13 indicate a positive correlation between the number of input images and model performance, as evidenced by higher F1 scores with larger image datasets. Notably, our method demonstrates robustness even in low-resource conditions, achieving F1 scores of over 92.2% and 89.78% on the TKH Set 1 and TKH Set 2 datasets, respectively, when trained with just 40 images.

The loss function comprises both localization and spotting components. For localization, we employ the smooth $L_{1}$ loss, while for the spotting loss, we assess the effectiveness of our proposed new loss by testing several different configurations of the loss function within the same architecture. Our results in Table 7 demonstrate that the triplet loss function does not yield substantial performance improvements, whereas the contrastive loss function surpasses the triplet loss function. The ranked list loss, which can adaptively balance positive and negative samples, achieves commendable performance. However, the proposed torus loss function outperforms all other loss functions. This can be attributed to its ability to focus on challenging samples in the margin gap and balance positive and negative samples, which facilitates learning distance metrics and enhances text spotting in low-resource scenarios. The experimental results on all four datasets using the torus loss function corroborate its superiority compared to other loss functions, thus validating the effectiveness of our proposed approach.

We have conducted a comprehensive evaluation of the cross-domain performance of our proposed model by training it on various datasets and selecting the DBH Novel dataset for testing purposes. The experimental results are presented in Table 8, demonstrating the effectiveness of our proposed model. For example, when the model is trained on the VML dataset, the DBH Novel dataset can be considered as a novel class. However, the model’s optimal performance is obtained when it is trained and tested on the same dataset, as the data distributions are similar. Nevertheless, cross-domain training may result in a decline in performance, particularly when the data types differ significantly, and a limited number of samples are used in the training process.