Hierarchical Visual Feature Aggregation for OCR-Free Document Understanding

Contrary to early multimodal learning approaches that focused on joint embedding learning for diverse downstream tasks [1, 2, 3, 4, 5, 20, 21, 22, 23, 24], recent frameworks [9, 12, 13, 25] have shifted toward generating open-ended language responses by leveraging LLMs [26, 27]. This evolution has contributed to various architectural innovations; GiT [9], BLIP [12], and Qwen-VL [28] pioneer encoder-decoder architectures for unified vision-language understanding and generation. Building upon these foundations, BLIP-2 [13] and mPLUG-Owl [25] introduce resamplers (Q-Formers) to bridge the vision-language modality gap through query-relevant feature extraction and dimensionality reduction. Further advances have come through InstructBLIP [29] and LLaVA [30], which enhance multimodal capabilities by instruction tuning on vision-language conversation data.

Recent studies explore MLLMs’ text-reading capabilities and their relationship with image resolution. Wang et al. [9] shows MLLMs gradually learn to read text in images through large-scale pretraining. However, Liu et al. [31] reveals that most MLLMs are constrained by their visual encoder architecture designed only for 224$\times$224 resolution inputs, significantly limiting their ability to capture fine-grained details in text-centric tasks. While Monkey [32] demonstrates enhanced performance by supporting high-resolution inputs, their analysis was limited to single-scale processing. These investigations collectively indicate that, although MLLMs can read text in images, their performance is significantly influenced by input resolutions, affecting their ability to handle diverse font sizes and capture fine-grained textual details.

Early studies adopted external OCR engines to explicitly model and understand the texts within images [33, 34, 35, 36, 37, 38, 39]. LayoutLM [33] directly extends BERT [40] to jointly model interactions between text, visual and layout information for the first time. Based on LayoutLM, there have been efforts to introduce various pretraining objectives to learn layout information effectively by incorporating image-text alignment [34], cell-level information from OCR outputs [36], and multi-modal masking [35, 37]. Despite their strong performance on document understanding benchmarks, these models rely on off-the-shelf OCR engines and struggle with challenging cases such as diverse fonts, handwritten texts, and degraded historical documents. Moreover, they require additional modules for processing OCR results, which increases computational complexity.

To address the drawbacks of OCR-based models, OCR-free models powered by end-to-end multi-modal pretraining have been proposed. Dessurt [16] and Donut [17] combine a learnable visual encoder with an autoregressive text decoder for text recognition. Pix2Struct [18] focuses on layout understanding by learning from HTML data and masked webpage screenshots. However, these methods suffer from high computational costs because they require pretraining models from scratch before task-specific fine-tuning. UReader [19] takes a different approach by leveraging pretrained MLLMs, reducing training costs while handling diverse document types through document instruction tuning and shape-adaptive cropping. Despite these advances, current OCR-free models still struggle to capture diverse visual scales and font sizes, often missing local details in documents.

MLLMs generate instruction-following responses given multi-modal inputs, an image $I$ and text (instruction) $T$. The output token sequence with a length of $n_{s}$ is denoted by $S=\{s_{1},...,s_{n_{s}}\}$, where the probability of generating $s_{i}$ is given by $p(s_{i}|I,T,S_{<i})$, $1\leq i\leq n_{s}$. Given the correct answer $Y=\{\mathbf{y}_{1},...,\mathbf{y}_{n_{y}}|\mathbf{y}_{i}\in\mathbb{R}^{|V|}\}$, where $n_{y}$ is the length of the answer and $V$ is the vocabulary set, the learning objective of the MLLMs, parameterized by $\theta$, is given by

	$\displaystyle\theta^{*}$	$\displaystyle=\operatorname*{arg\,min}_{\theta}~{}\mathcal{L}_{\text{MLLM}}(I,T;\theta)$
		$\displaystyle=\operatorname*{arg\,min}_{\theta}~{}\frac{1}{|\mathcal{D}|}\sum_{(I,T)\in\mathcal{D}}\left[-\sum_{i=1}^{n_{s}}\sum_{v=1}^{|V|}\mathbf{y}_{i,v}\log p(s_{i}=v|I,T,S_{<i};\theta)\right],$		(1)

where $\mathcal{D}$ is the image-text dataset.

MLLMs generally comprise three modules: 1) a vision encoder, 2) a projector, and 3) a large language model (LLM). First, we generate visual inputs at multiple scales from a given document image. The vision encoder extracts visual features for detailed image understanding. The resampler-based projector converts these visual features into a compact set of query tokens for the language model. Unlike traditional MLLMs, which feed these query tokens directly into the LLM, we introduce the Hierarchical Visual Feature Aggregation module to reduce the number of visual query tokens while preserving local information. Finally, the LLM processes both the visual and instruction tokens to generate a response autoregressively. Figure 1 illustrates the overall framework.

We focus on developing a document understanding framework that incorporates multi-scale visual inputs to address various image scales for MLLMs. However, using pretrained MLLMs constrains the input resolution to a small, square-shaped configuration defined by the visual encoder. This limitation hampers capturing detailed information in high-resolution images and leads to spatial distortions when handling document images of diverse resolutions and aspect ratios.

We adopt shape-adaptive cropping (SAC) [19] to generate multiple sub-images, allowing our model to handle varying aspect ratios and resolutions. We split each image using a predefined grid $\{g=(n_{h}\times n_{w})|n_{h},n_{w}\in\mathbb{N}\}$, where $n_{h}$ and $n_{w}$ respectively denote the number of rows and columns for the grid $g$. Then, each sub-image is resized to fit the visual encoder. Note that the optimal grid $g^{*}=(n_{h}^{*}\times n_{w}^{*})$ for each image is selected by two metrics; resolution coherence, and shape similarity. Refer to Appendix C for more details.

Although SAC handles diverse aspect ratios and high-resolution images, it may miss local details within each sub-image due to its receptive fields of fixed-size. To address this issue, we consider an additional level of detail by applying $2\times$ upscaling to each sub-image to capture detailed visual features and small text fonts through enlarged local resolution. To be specific, this augmentation process involves subdividing each sub-image into half along each spatial dimension, resulting in $2n_{h}\times 2n_{w}$ sub-images. We again resize the subdivided images to ensure compatibility with the visual encoder. By incorporating these high-resolution representations, the model now observes the input image in three different levels: global, $n_{h}\times n_{w}$, and $2n_{h}\times 2n_{w}$ scales.

Unlike vision encoders and resampler-based projectors, which process sub-images in batches and thereby exhibit linear computational complexity, LLMs demonstrate quadratic complexity with respect to the number of sub-images. Thus, when the number of visual inputs increases from $n$ to $5n$, the computational burden of LLMs escalates 25 times, while that of vision encoders and resamplers grows 5 times. Because LLMs are represented by networks with significantly more channels than other models, this quadratic increase of computational cost is problematic in many practice scenarios.

The HVFA module leverages a feature pyramid to fuse multi-scale features for LLMs and reduces computational overhead. We first construct a feature pyramid using the visual features from two adjacent scales, $i^{\text{th}}$ and $(i+1)^{\text{st}}$ scales. Cross-attentive pooling is then applied to features at scale $(i+1)$ to compress and preserve detailed information, and the pooled features are aggregated with low-resolution features at scale $i$. Figure 2 depicts the mechanism of the HVFA module.

Let $\mathbf{F}_{i}\in\mathbb{R}^{n_{h}^{i}\times n_{w}^{i}\times q\times c}$ be the visual features for the $i^{\text{th}}$ scale extracted from resampler, where $(n_{h}^{i}\times n_{w}^{i})$ is the number of sub-images at $i^{\text{th}}$ scale, $q$ denotes the number of queries from resampler, and $c$ is the number of channels. We perform $2\times 2$ max-pooling on $\mathbf{F}_{i+1}\in\mathbb{R}^{n_{h}^{i+1}\times n_{w}^{i+1}\times q\times c}$ to obtain a pooled feature $\mathbf{F}^{\prime}_{i+1}\in\mathbb{R}^{n_{h}^{i}\times n_{w}^{i}\times q\times c}$. To reduce the information loss by the max-pooling operation, we conduct the following operations including a lightweight cross-attention:

	$\displaystyle\hat{\mathbf{F}}_{i+1}=\mathbf{F}^{\prime}_{i+1}+\text{Softmax}\left(\frac{(\mathbf{F}^{\prime}_{i+1}\mathbf{W}_{\text{query}})(\mathbf{F}_{i+1}\mathbf{W}_{\text{key}})^{T}}{\sqrt{d}}\right)(\mathbf{F}_{i+1}\mathbf{W}_{\text{value}})\mathbf{W}_{\text{proj}}$		(2)
	$\displaystyle\bar{\mathbf{F}}_{i}=\mathbf{F}_{i}+\hat{\mathbf{F}}_{i+1},\hskip 113.81102pt$		(3)

where $\mathbf{W}_{\text{query}},\mathbf{W}_{\text{key}},\mathbf{W}_{\text{value}}\in\mathbb{R}^{c\times d}$, and $\mathbf{W}_{\text{proj}}\in\mathbb{R}^{d\times c}$are learnable parameters, and $d$ is the embedding dimension. In Equation (3), a pooled feature $\mathbf{F}^{\prime}_{i+1}$ corresponds to a query and the original feature, $\mathbf{F}_{i+1}$, works as a key and a value. The layer normalization is omitted for simplicity of the expression. We feed $\bar{\mathbf{F}}_{i}$ to LLMs to generate the language responses. Note that our method maintains the original complexity of LLMs, despite considering multiple scales in practice, because the feature pyramid structure aggregates them into the initial local scale of visual features generated by SAC.

We assume that well-compressed features retain essential information necessary for the reconstruction of the original features. After obtaining the pooled features, based on this assumption, we introduce a small decoder network, used only for training, to reconstruct the original features from the pooled features. The decoder is trained with the mean squared error (MSE) loss between the reconstructed and original features, which is given by

$\displaystyle\mathcal{L}_{\text{MSE}}(\hat{\mathbf{F}}_{i+1},\mathbf{F}_{i+1})=\mathbb{E}[(r(\hat{\mathbf{F}}_{i+1})-\text{stopgrad}(\mathbf{F}_{i+1}))^{2}],$

(4)

where $r(\cdot)$ is an MLP-based reconstruction network and $\text{stopgrad}(\cdot)$ indicates a stop-gradient operation preventing the collapse in training. By the MSE loss, we guide the model to retain important visual details while reducing feature dimensionality. The final objective function is given by

$\displaystyle\mathcal{L}_{\text{Final}}=\mathcal{L}_{\text{MLLM}}+\lambda\mathcal{L}_{\text{MSE}},$

(5)

where $\lambda$ is the weight for the balance between two terms.

Our approach is similar to pooling-based methods such as MViT [41] and MViT-v2 [42], which construct attention mechanisms utilizing pooled features as keys and values to build efficient ViT variants [43]. However, unlike these methods, we use pooled features as queries to reduce query size rather than optimizing attention mechanisms themselves.

Our cross-attentive pooling can be seen as a form of resampler [13, 44] that reduces spatial dimensionality. In this interpretation, the pooled features, $\mathbf{F^{\prime}_{i+1}}$, act as the query tokens for learning. Unlike traditional query-based resamplers that require a new set of trainable query vectors, our method uses pooled features, which are more effective for resampling than randomly initialized query vectors, as discussed further in Section 4.4.

Another key challenge in document understanding is learning to read text using layout information. Although [19] trains models to read entire document text, this approach entails significant computational challenges and information losses due to text truncation caused by the input capacity of LLMs. To address these limitations, we introduce two novel tasks: Reading Partial Text (RPT) and Predicting Text Position (PTP). RPT focuses on reading specific text segments at given positions, wwhile PTP aims to predict the positions of given text segments. These tasks facilitate enhanced layout understanding with reduced computational cost.

Suppose that we can access full document text, represented by a sequence of tokens and we adopt a standard reading order of text (top-to-bottom, left-to-right). For the RPT task, we randomly select one of three types: 1) "first": reading up to $p_{end}\%$ of the text, 2) "middle": reading from $p_{start}\%$ to $p_{end}\%$ of the text, and 3) "last": reading from $p_{start}\%$ to the end. We also sample an instruction template for the selected type, where the instruction template examples are presented in Table 1 and the full list is available in Appendix I. We then compute the maximum text coverage, $c_{\text{max}}$, to mitigate text truncation, which is given by

$\displaystyle c_{\text{max}}=\min(1.0,l_{\text{max}}/l),$

(6)

where $l$ is the length of the tokenized text and $l_{\text{max}}$ is the maximum sequence length acceptable by LLMs.

We set a threshold for the minimum coverage as a hyperparameter, $c_{\text{min}}$, to ensure that sufficient text is read. The range for reading, denoted by $t_{\text{range}}$, is given by

$$t_{\text{range}}=\begin{cases}c_{\text{max}}&\text{if $c_{\text{max}}\leq c_{\text{min}}$}\\ \text{random}(c_{\text{min}},c_{\text{max}})&\text{otherwise},\end{cases}$$

(7)

where $\text{random}(a,b)$ denotes a uniform sampling from an interval $(a,b)$. Then, the positions to predict are expressed as follows:

$$(p_{\text{start}},p_{\text{end}})=\begin{cases}(0,\hskip 2.84544ptt_{\text{range}})&\text{if $\text{type}=\text{`first'}$}\\ (p^{\prime}_{\text{start}},\hskip 2.84544ptp^{\prime}_{\text{start}}+t_{\text{range}})&\text{else if $\text{type}=\text{`middle'}$}\\ (100-t_{\text{range}},100)&\text{else if $\text{type}=\text{`last'}$}.\end{cases}$$

(8)

where $p^{\prime}_{start}=\text{random}(0,100-t_{\text{range}})$.

For the PTP task, given a segment of text, the model aims to infer its relative position within the whole text. We determine text segments and their relative positions using the same process as in the RPT task. Through diverse training examples, the model learns to understand spatial relationships between text segments and other components in a document.

We utilize document instruction dataset corpora for training, following [19]. The training dataset includes various types of document images, including tables, charts, natural images, and web page screenshots. The datasets used are DocVQA [45], InfographicsVQA [46], DeepForm [47], KleisterCharity [48], WikiTableQuestions [49], TabFact [50], ChartQA [51], VisualMRC [52], TextVQA [53], and TextCaps [54]. The combined dataset comprises approximately 650K image-instruction pairs. For more details about the datasets, please refer to Appendix E.

We evaluate our model on the test splits of each dataset. For DocVQA and InfoVQA, we employ the ANLS score [55], while we adopt the F1 score for DeepForm and KLC. The performance on WTQ, TabFact, and TextVQA is measured using accuracy, whereas ChartQA utilizes relaxed accuracy [56]. The metric for TextCaps and VisualMRC is the CIDEr score [57]. Evaluations for TextVQA and TextCaps are conducted on their respective official challenge websites.

We conduct experiments on two pretrained resampler-based MLLMs: BLIP-2-OPT-2.7B [13] and mPLUG-Owl-7B [25]. We utilize LoRA [58] to fine-tune the language model efficiently while keeping the vision encoder frozen. For the BLIP-2-based model, we freeze the resampler and apply LoRA, whereas for the mPLUG-Owl-based model, we train the resampler to ensure a fair comparison with [19]. Regarding the SAC, we set the maximum number of sub-images to 9 for the BLIP-2-based model and 20 for the mPLUG-Owl-based model. Refer to Appendix D for more details.

We compare our framework to other OCR-free baselines, including document-specific pretraining approaches [16, 17, 18] for completeness. Given that previous methods utilize varying numbers of model parameters, trainable parameters, and training data, we provide the detailed configurations in Table 2. It is important to note that our comparisons are made with OCR-free methods based on comparable MLLMs in terms of model size and dataset scale. Table 3 presents the overall results of the proposed algorithm across 10 document understanding benchmarks. When integrated with both BLIP-2 and mPLUG-Owl, our approach consistently outperforms UReader [19] by a significant margin across all benchmarks. Among MLLM-based methods, our framework outperforms Qwen-VL-based methods [28, 32] in most benchmarks, despite using a smaller set of pretraining data and smaller model. The results show that our model successfully transfer the knowledge of pretrained MLLMs to understand document samples.

We perform several ablation studies with the BLIP-2-based model to validate the effectiveness of each component of our approach.

To analyze the trade-off between performance and computational cost, we compare variants of our model that process different scales of visual inputs: the first scale ($S_{1}$), the second scale ($S_{2}$), multiple scales ($S_{1}+S_{2}$), and multiple scales with HVFA ($S_{1}+S_{2}$ w/ HVFA, ours). Figure 3(Left) presents the average benchmark scores and throughput for each model. The throughput is measured on a single NVIDIA Quadro RTX GPU using the largest possible batch size for each model. where the input size of the visual encoder is $224\times 224$. The results indicate that incorporating a finer scale significantly improves performance, albeit with increased computational cost ($S_{1}$ vs. $S_{2}$). Additionally, combining features from multiple scales boosts performance over single-scale features, though at the expense of higher computational load ($S_{1}$, $S_{2}$ vs. $S_{1}+S_{2}$). Finally, our method efficiently integrates finer-scale features into a coarser scale, achieving a balance between performance gains and computational demands ($S_{1}+S_{2}$ vs. $S_{1}+S_{2}$ w/ HVFA).

To demonstrate the effectiveness and robustness of RPT against truncation, we conducted experiments varying the maximum sequence length of LLMs during training. Figure 3(Right) presents the average benchmark scores and the proportion of truncated text data across the entire dataset. RPT consistently outperforms RFT in all settings, indicating that RPT is robust to the capacity constraints of LLMs. The design of RPT is helpful for mitigating truncation issues while RFT suffers from truncation of text reading data. Notably, with a maximum sequence length of 256, approximately 33.5% of the dataset consists of truncated texts, which is a considerable amount. Consequently, model performance declines even further compared to training without any text-reading task, as represented by the red dotted line in Figure 3(Right). Our approach ensures reliable data quality while incorporating stochasticity and positional information, contributing to its effectiveness. These results imply that our partial text reading task can inspire further research on knowledge distillation and the development of compact models for OCR-related tasks.

Figure 4 illustrates question-answer pairs from the DocVQA [45] and InfographicVQA [46] datasets for comparisons between our framework and UReader [19]. While both models can read large text, our model is more powerful in identifying small text than UReader. This highlights our model’s strength in effectively handling text at multiple scales. Furthermore, our method delivers more accurate answers for questions requiring precise reading comprehension. For instance, to answer the question "Which meeting in December has the location not finalized yet?", the model needs to accurately read the text "To be announced" as demonstrated in the rightmost example in Figure 4.