Localize, Group, and Select: Boosting Text-VQA by Scene Text Modeling

Considering Text-VQA as a typical multi-modal task with inputs from several different modalities, our model utilizes the hybrid fusion technique, where the first step is to generate separate unimodal representations of different modalities. We use BERT [6] and Faster R-CNN [26] to encode the text and image respectively. Following previous works [12, 33], we also include the pyramidal histogram of characters (PHOC) representations [1] for the OCR tokens.

To let the model aware of ROI given a certain question, we leverage question-visual pretraining before Text-VQA, and question-OCR modeling during Text-VQA training, with object tokens to help align two modalities, with details shown in Section 3.2.

With the aforementioned inputs, we add grouping information for OCR tokens based on their locations. The details of the clustering algorithm are introduced in Section 3.3.

As shown in Figure 2(b), after uni-modal features are obtained, the representations of the same OCR and object tokens from different modalities are concatenated respectively. We feed the textual representation of the question and the fused representation of detected objects and OCR tokens into a multi-modal transformer. We utilize an answer decoder with a pointer network for output generation and design a selection framework for multiple OCR sources, which we discuss in Section 3.4.

To successfully predict the answer given the question information, a robust model should be capable of first pointing out the specific region that is most related to the question, then relying on the OCR text of that specific region to generate the output.

It is the most desirable case if we have large amounts of alignment data between question and OCR regions to learn the question-OCR grounding. Since there is no publicly available large-scale question-OCR dataset, we view the question-OCR grounding problem as a two-stage learning problem. Specifically, we first learn the general grounding between text and image regions usin existing region-description datasets, for example, Visual Genome [20]. Second, following the intuition from Oscar [21], we use the object tokens as textual “guidance” to help the model learn to ground between question and OCR regions.

Context understanding has always been important in language processing and question answering. This problem becomes even more vital in Text-VQA, where the evidence of clear separation of text groups lies in the visual modality rather than text. The detection scope of current OCR systems also mainly remains at individual token level or “line” level (grouping tokens aligned closely forming a line).

Figure 3(a) shows an example of raw detection results by an OCR system. For most text groups such as names and authors of the books, its words are split into multiple lines and thus detected individually. Simply aligning text pieces by their generated sequence, and not providing extra visual evidence will often lead to misordered text and incomplete/incorrect context modeling.

A straightforward idea would be to extract potential text groups based on visual modality evidence, which may serve as additional spatial information along with raw OCR bounding box coordinates, and provide weak evidence for token realignment or token context to form complete sentences in the text domain.

In LOGOS, we perform unsupervised clustering of the OCR text bounding boxes during data preparation. Given an image containing $m$ lines $\{L_{1},\dots,L_{m}\}$ detected by OCR, with $L_{i}=t_{i1}t_{i2}\dots t_{il_{i}}$ containing $l_{i}$ tokens, we define the distance between two bounding boxes as the minimum distance between any two points on the two bounding boxes, and perform clustering using DBSCAN [10] on the bounding boxes of all lines $\{B_{i}\}$. Through clustering, each token $t_{ij}$ is assigned extra spatial hierarchical information in its cluster $C_{i}$, line $i$ and token $j$. We generate a $d$ dimensional sinusoidal embedding as a positional representation for each attribute to form the overall descriptor $v^{sp}_{ij}$, and concatenate it as part of the final OCR representation.

Where $\text{Pos}(i,d)$ is the $d$-dim sinusoidal positional embedding at position $i$. Figure 3(b) shows a sample result of this clustering approach.

As aforementioned, current OCR systems are not robust enough to perfectly extract high-quality scene text from images. Different systems also result in different kinds of errors: while a careless OCR system will not be capable of finding all text in the images correctly and make more detection error, a meticulous OCR system may detect too much text from details inside the image, which will increase the difficulty of localizing the question to the correct region. This opens up the possibility of minimizing OCR error and maximizing reasoning effectiveness by combining different OCR sources.

We modify the training and predicting process order to best utilize different OCR systems. Given OCR scene text from $k$ systems, during the decoding stage, $k$ independent answers are generated separately. LOGOS then calculates the confidence score for the $i$-th answer $S^{i}=t_{1}^{i}t_{2}^{i}...t_{l}^{i}$ as

where $i\in[k]$ and $x$ is the input of other features including visual features and questions. The answer with the highest score is selected as the final answer. During training, all $k$ OCR sources are trained equally, which also serves as extra training data for the learning process.

In this paper, we use TextVQA [28] and STVQA [3], two commonly used scene-text VQA datasets as our main testbeds. Visual Genome dataset [20] is used for the referral expression auxiliary training.

Our model is implemented based on the framework of M4C [12]. For the visual modality, we follow the settings of M4C and use Faster R-CNN [26] fc7 features of 100 top-scoring objects in the image detected by a Faster R-CNN detector pretrained on the Visual Genome dataset. The fc7 weights are fine-tuned during training. For the text modality, we use two OCR systems, Rosetta OCR [4] and Azure OCR  ¹¹1https://azure.microsoft.com/en-us/services/cognitive-services/computer-vision, to recognize scene text. We use a trainable 3-layer BERT-base encoder [6] for text representation. We follow M4C and include pyramidal histogram of characters (PHOC) representations [1] of OCR text. We perform scene text clustering on normalized bounding boxes using DBSCAN with $\epsilon=0.02$.

The candidate vocabulary for decoding consists of the top 5,000 frequent words from the answers in the training set, as well as the detected OCR tokens in the current image. The answer is generated with a pointer network [30].

LOGOS contains 116M trainable parameters. During pretraining steps, we first pretrain LOGOS using the question-visual grounding task, then continue to train on the Text-VQA task. We set the batch size to 48 and train for 24,000 iterations for both pretraining and fine-tuning stages. The initial learning rate is set to 1e-4 with a warm-up period of 1000 iterations, and the learning rate decays to 0.1x after 14,000 and 19,000 iterations respectively. We use the checkpoints that achieved the best performance on the validation set for evaluation.

Table 1 lists the performance of LOGOS on the TextVQA dataset comparing to other baselines. We find that LOGOS outperforms all current baselines which do not use extra OCR data.

We have several interesting observations from this table: (1) We see a huge gap between models using another OCR system (M4C+Azure OCR, SA-M4C, TAP, LOGOS) and models using the Rosetta system. The gap illustrates that previous performances on the Text-VQA task are severely limited by the quality of the scene texts detected. This proves the importance of better modeling and refining for the textual modality. (2) LOGOS shares the same idea of introducing auxiliary tasks for pretraining or joint training with TAP, and the results improve similarly although the training tasks are very different. However, LOGOS still outperforms TAP by 0.93% on the TextVQA dataset because of better modeling of text features. (3) It’s noteworthy that the current highest score from TAP is obtained by pretraining on other large-scale OCR datasets (OCR-CC) which are designed to better utilize scene text features in multimodal tasks. This idea is similar to ours and our model is fully compatible with pretraining on this dataset. We expect a further improvement of our model when it can get access to more OCR data.

Looking at STVQA results, we can see that training using only STVQA data does not outperform the TAP model [33]. This may due to the fact that STVQA contains more short-length answers, compared with TextVQA where the questions are relatively longer and harder to answer. As the dataset size of STVQA is small and the spatial relationship of this dataset is relatively simple, we observe that LOGOS suffers from overfitting in this dataset. As introducing the joint training with TextVQA dataset, we do see a huge improvement on the STVQA dataset: our model outperforms SNE, previously best STVQA model jointly trained on TextVQA dataset by 2.9 points in ANLS. This reveals that LOGOS can also achieve great performance in STVQA dataset with the additional training data from other Text-VQA datasets.

Besides the full model, we also ran several experiments on variants of LOGOS to examine the effectiveness of each component. Results are shown in Table 2. We analyze model variant performance from three perspectives.

In this section, we analyze predictions of different model variants of LOGOS on validation set questions to check module effectiveness. We show two examples in Figure 4.

The case on the left demonstrates the effect of text clustering. The two images show OCR text pieces without and with text clustering respectively. Without the clustering information and with only the bounding box coordinates, the model still struggles to find the connection between individual words to form the phrase “coloring with stain”. Adding clustering-based word position information leads to much easier inference for the model to predict the correct answer.

The case on the right further emphasizes the importance of utilizing detection results from multiple OCR sources. When one OCR source (Azure) incorrectly misses the key evidence text (the “3” on the player’s jersey), the model will most certainly generate an incorrect answer. After incorporating the source selection module, the model is able to dynamically choose the most reasonable source, based on its understanding of the question and relationship between text and objects. In this case, the model successfully grounds the question to the correct object (the jersey on the left) and answers the correct jersey number 3 instead of 17.

In this section, we present analysis of error types to see how our model achieved better accuracy and what limits further improvement. We randomly sampled 30 negative examples from M4C and another 30 from LOGOS and counted the number of errors caused by bad OCR quality, failure to group scene text, or failure to locate questions in the related area. The proportion of such errors reduced from 80.0% to 53.3%, which proves LOGOS’s ability to better locate questions and utilize scene text clusters from multiple OCR systems.

When looking into negative examples from LOGOS, we also notice two typical types of error which we present in Figure 5. In these two cases, the model correctly recognizes and locates the question to the scene text, but still failed to generate the correct answer.

In the first case, the question asks about the size of the TV. Answering such questions requires a QA model to be equipped with proper external knowledge. Here, the model needs to understand measurements and different kinds of notations such as 8K for resolution and 98” for size. Other types of external knowledge in similar TextVQA error cases include: recognizing time, identifying dates, and counting and calculation.

As for the second case, although all the words are detected correctly by the OCR system and grouped together, the model struggles to completely understand the long text. Answering this type of question further requires the model to fully understand the content, which is almost impossible to be achieved without a more specific design, for example, a pipeline with a reading comprehension module.