LORE++: Logical Location Regression Network for Table Structure Recognition with Pre-training

Early works [17, 18] introduce segmentation or detection frameworks to locate and extract splitting lines of table rows and columns. Subsequently, they reconstruct the table structure by empirically grouping the cell boxes with pre-defined rules. These models would suffer from tables with spanning cells or distortions. The latest baselines [2, 13, 19] tackle this problem with well-designed detectors or attention-based merging modules to obtain more accurate cell boundaries and merging results. However, they either are tailored for a certain type of datasets or require customized processing to recover table structures, and thus can hardly be generalized. So there arise models focusing on directly predicting the table structures with neural networks.

[7] proposes to model table cells as text segmentation regions and exploit the relationships between cell pairs. Precisely, it applies graph neural networks [20] to classify pairs of detected cells into horizontal, vertical, and unrelated relations. Following this work, there are models devoted to improving the relationship classification by using elaborated neural networks and adding multi-modal features [21, 8, 1, 22, 9]. This framework bypasses the extraction of precise boundaries of cells, but the nearest neighbor graphs in these models encode biased prior to the model. Moreover, there is still a gap between the set of relation triplets and the global table structure. Complex graph optimization algorithms or pre-defined post-processings are needed to recover the tables.

[23, 10, 24] make the pioneering attempts to solve the TSR problem in an end-to-end way. They employ sequence decoders to generate tags of markup language that represent table structures. However, the models are supposed to learn the markup grammar with noisy labels, resulting in the methods being difficult to train and requiring tens of times more training samples than methods of other paradigms. Besides, these models are time-consuming owing to the sequential decoding process.

[12] propose to perform ordinal classification of logical indices on each detected cell for TSR, which is close to our approach. The model utilizes graph neural networks to classify detected cells into the corresponding logical locations, while it ignores the dependencies and constraints among logical locations of cells. Besides, the model is only evaluated on a few datasets and not against the strong TSR baselines.

The pre-trained model shows remarkable performance on numerous CV tasks, e.g. classification[15], segmentation[25, 26], detection[27, 28], and information extraction[29, 30], which proved the pre-trained representations generalize well to various downstream data. Unfortunately, there is currently no pre-training work in TSR with images as input only.

MAE[15] masks random patches of the input image and reconstructs the missing pixels. Successfully reconstructing the object in the image indicates that the model understands what the object is and what it looks like. So, we utilize MAE as one of our pre-training tasks to absorb various layout and structure information from massive data.

Besides, ESP[30] constructs key-value linking task in pre-training to model entity linking task in fine-tuning. Due to the consistency between pre-training and fine-tuning, ESP improves the SOTA accuracy of linking tasks from 81.25% to 92.31% on XFUND[31] dataset. Inspired by ESP, we use text segments instead of table cells to model logical relationship, for which the text segments can be easily obtained by the OCR engine.

In this paper, we consider the TSR problem as the spatial and logical location regression task. Specifically, for an input image of the table, similar to a detector, a set of table cells $\{O_{1},O_{2},...,O_{N}\}$ are predicted as their logical locations $\{l_{1},l_{2},...,l_{N}\}$, along with the spatial locations $\{B_{1},B_{2},...,B_{N}\}$, where $l_{i}=(r_{s}^{(i)},r_{e}^{(i)},c_{s}^{(i)},c_{e}^{(i)})$ standing for the starting-row, ending-row, starting-column and ending-column, $B_{i}=\{(x_{k}^{(i)},y_{k}^{(i)})\}_{k=1,2,3,4}$ standing for the four corner points of the $i$-th cell and $N$ is the number of cells in the image.

With the predicted table cells represented by their spatial and logical locations, the table in the image can be converted into machine-understandable formats, such as relational databases. Besides, the adjacency matrices and the markup sequences of tables can be directly derived from their logical coordinates with well-defined transformations rather than heuristic rules (See supplementary section 1).

The transformation from the table representation in the logical location of cells to cell adjacency and markup sequence representation is well-defined for general settings, without any approximation algorithm or heuristic rules.

In order to streamline the joint prediction of spatial and logical locations, we employ a key point segmentation network [33, 2] as the feature extractor and model each table cell in the image as its center point. Besides, it is compatible with both wired and wireless tables and is easier to implement on inconsistent annotations of different datasets, i.e., aligned boxes in WTW and TG24, or text region boxes in SciTSR and PubTabNet.

For an input image of width $W$ and height $H$, the network produces a feature map $f\in\mathbb{R}^{\frac{W}{R}\times\frac{H}{R}\times d}$ and a cell center heatmap $\widehat{Y}\in[0,1]^{\frac{W}{R}\times\frac{H}{R}}$, where $R$, $d$ are the output stride and hidden size; $\widehat{Y}_{x,y}=1$ corresponds to a detected cell center, while $\widehat{Y}_{x,y}=0$ refers to the background.

In the subsequent modules, the CNN features $\{f^{(1)},f^{(2)},...,f^{(N)}\}$ at detected cell centers $\{\hat{p}^{(1)},\hat{p}^{(2)},...,\hat{p}^{(N)}\}$ are considered as the representations of table cells.

We choose to predict the four corner points rather than the rectangle bounding box to better deal with the inclines and distortions of tables in the wild. For spatial locations, the features of the backbone $f$ are passed through a $3\times 3$ convolution, ReLU and another $1\times 1$ convolution to get the prediction $\{\hat{B}^{(1)},\hat{B}^{(2)},...,\hat{B}^{(N)}\}$ on centers $\{\hat{p}^{(1)},\hat{p}^{(2)},...,\hat{p}^{(N)}\}$, where $\hat{B}^{(i)}=\{(\hat{x}_{k}^{(i)},\hat{y}_{k}^{(i)})\}_{k=1,2,3,4}$.

As dense dependencies and constraints exist between the logical locations of table cells, it is rather challenging to learn the logical coordinates from the visual features of cell centers alone. The cascading regressors with inter-cell and intra-cell supervisions are leveraged to explicitly model the logical relations between cells.

The losses of cell center segmentation $L_{center}$ and spatial location regression $L_{spa}$ are computed following typical key point-based detection methods [33, 2].

The loss of logical locations is computed for both the base regressor and the stacking regressor:

The total loss of joint training is then computed by adding the losses of cell center segmentation, spatial and logical location regression along with the I2C supervisions:

As illustrated in Figure 4, We follow the framework of LORE with corresponding modifications to facilitate the joint pretraining of spatial and logical tasks. The CNN-based backbone of LORE is replaced with the ViT encoder and the MAE ViT-based decoder is added, catering to the MAE-like pre-training. The logical decoder shares a similar structure with the base regressor and the stacking regressor, i.e. layers of self-attention mechanism, but with the linear map on paired features for the logical distance prediction task.

The model takes the patchified images and the corresponding masks as input. Aiming at preventing information leakage from the masked patches to the unmasked ones, we leverage a unidirectional self-attention, where the unmasked tokens can only attend to each other but not the masked ones, while the masked patches can attend to all patches. The masked patches are replaced with the mask token before being inputted into the spatial decoder, while the entire encoded feature maps are forwarded to the logical decoder for the Logical Distance Prediction task.

Inspired by ViT-MAE, we utilize the MAE task to guide the model to learn general visual clues of tables such as text region, ruling lines, etc. We mask out 50% patch tokens of the image randomly, which is different from the fashion as was suggested in ViT-MAE. Because table images are highly semantic and information-dense, excessively masking out patches could impede the reconstruction task. To introduce order information for the MAE task, we employ 1-D fixed sinusoidal position embeddings. The image encoder and decoder are trained using a normalized mean squared error (MSE) pixel reconstruction loss, which quantifies the disparity between the normalized target image patches and the reconstructed patches. This loss is specifically computed for the masked patches.

For this task, the model is trained to predict the row and column logical distances between each pair of cells, as illustrated in Figure 5. In this way, the model learns the ability to understand the basic grids of tables, which serves as the foundation for recognizing complicated table structures. We first pre-process the table images with an off-the-shelf Optical Character Recognition (OCR) system to obtain the 2D position of texts in table cells. Then the horizontal and vertical positions of cells are clustered to conform to the grid rows and columns. In this way, we can obtain the training target of each pair of cells as illustrated in Figure 5. During the training stage, the features of word region boxes are extracted according to their center points in a similar fashion as is in Section IV-C2. Then the cell features are fed into the logical decoder as introduced in model architecture, where the encoded features are paired for the prediction of logical distance. We employ an L1-loss for the logical distance prediction task.

LORE is trained and evaluated on table images with the max side scaled to a fixed size of $1024$ ($512$ for SciTSR and PubTabNet) and the short side resized equally. The model is trained for 100 epochs, and the initial learning rate is chosen as $1\times 10^{-4}$, decaying to $1\times 10^{-5}$ and $1\times 10^{-6}$ at the 70th and 90th epochs for all benchmarks. All the experiments are performed on the platform with 4 NVIDIA Tesla V100 GPUs. We use the DLA-34 [40] backbone, the output stride $R=4$, and the number of channels $d=256$. When implementing on the WTW dataset, a corner point estimation is equipped following [2]. The number of attention layers is set to 3 for both the base and the stacking regressors. We run the model 5 times and take the average performance.

LORE++ adopts a 12-layer vision transformer encoder with 12-head self-attention, hidden size of 384, and 1536 intermediate size of feed-forward networks to better fit the MAE pre-training and controls a comparable amount of parameters with prevalent vision backbones in TSR framework such as ResNET-50. The number of attention layers in the logical head is set to 3 as in the vanilla LORE. The images are resized to 224. We pre-train the model using Adam optimizer with a batch size of 196 for steps. We use a weight decay of 0.05, $(\beta_{1},\beta_{2})=(0.0,0.95)$, a learning rate of 1.5e-4 and we linearly warm up the learning rate over the first 5% steps. For downstream fine-tuning, the vision backbone is initialized with the parameters of the pre-trained encoder, while both the base regressor and stacking regressor are initialized with the parameters of the pre-trained logical decoder. Other fine-tuning configurations are identical to that of the vanilla LORE.

The TSR models of different paradigms are evaluated using different metrics, including 1) accuracy of logical locations [37], 2) BLEU and TEDS [41, 10], and 3) F-1 score of adjacency relationships between cells [32, 38].

For cell logical location evaluation, a table cell coordinate is represented as $(\text{\emph{start-row}},\text{\emph{end-row}},\text{\emph{start-col}},\text{\emph{end-col}})$ as in the ICDAR competition [38]. The accuracy, that is the proportion of the cells with four coordinate values correctly predicted, is calculated as the metric of cell location evaluation score. BLEU[41] is widely adopted in the natural language processing community. TEDS [10] models the markup sequence as a tree (graph) and computes the edit distance between the output structure and label. As for evaluating the adjacency relations, we first convert the table to a list of triplets contains a pair of nodes and their adjacency relation (adjacent/ no adjacent), and make a comparison on relations extracted from output structure and ground truth by precision, recall and F1 score. When evaluating markup generation-based methods, BLEU and TEDS are employed. The accuracy of logical locations, BLEU, and TEDS directly reflect the correctness of the predicted structure, while the adjacency evaluation only measures the quality of intermediate results of the structure.

In our experiments, LORE is evaluated under all three types of metrics, since the logical coordinates are complete for representing table structures and can be converted into adjacency matrices and markup sequences by simple and clarified transformations as introduced in Section. When evaluating TEDS, we use the non-styling text extracted from PDF files following [4]. We also report the performance of cell spatial location prediction, using the F-1 score under the IoU threshold of 0.5, following recent works [8, 12]. In our experiments, We consider a detected cell to be true positive if its IOU with a ground truth cell bounding box is more than 0.5, following [8, 12, 1].

In this section, we further compare models of different TSR paradigms introduced before. Previous methods that predict logical locations lack a comprehensive comparison and analysis between these paradigms. We demonstrate how LORE overcomes the limitations of the adjacency-based and the markup-based methods by controlled experiments.

The adjacency of cells alone is not sufficient to represent table structures. Previous methods employ heuristic rules based on spatial locations [9] or graph optimizations [21] to reconstruct the tables. However, it takes tedious modification to make the pre-defined parts compatible with datasets of different types of tables and annotations. Furthermore, the adjacency-based metrics sometimes fail to reflect the correctness of table structures, as depicted in Figure 6. Experiments are conducted to verify this argument quantitatively. We turn the linear layer of the stacking regressor of LORE into an adjacency classification layer of paired cell features and employ post-processings as in NCGM [9] to reconstruct the table. The results are in Table VI. Although this modified model (Adj. paradigm) achieves competitive results with state-of-the-art baselines evaluated on adjacency-based metrics, the accuracy of logical locations obtained from heuristic rules decreases obviously compared to LORE (Log. paradigm), especially on WTW, which contains more spanning cells and distortions.

The markup-sequence-based models leverage image encoders and sequence decoders to predict the label sequences. Since the markup language has plenty of control sequences formatting styles, they can be viewed as noise in labels and impede model training [12]. It requires much more training samples and computational costs. As shown in Table VII, the number of training samples of the EDD model on the PubTabNet dataset is more than ten times larger than that of both LORE and LORE++. Besides, the inference process is rather time-consuming (See Table VII) due to the sequential decoding pattern, while models of other paradigms compute for each cell in parallel. The average inference time is computed from the validation set of PubTabNet with the images resized to $1280\times 1280$ for both models.

We conduct experiments to investigate the effect of the cascade framework on the prediction of logical coordinates. In Figure 7, we visualize the attention maps of the last encoder layer of the cascade/single regressors of two cells, i.e., the models 1d and 2b in Table IV. In the cascade framework, the base regressor in Figure 7 (a) focuses on the heading cells (upper or left) to compute logical locations. While the stacking regressor in Figure 7 (b) pays more attention to the surrounding cells to discover finer dependencies among logical locations and make sure the prediction is subject to natural constraints, which is in line with human intuition when designing a table. However, the non-cascade regressor in Figure 7 (c) can only play a role similar to the base regressor, which leaves out important information for the prediction of logical locations.

We summarize the model size and the inference operations of LORE and LORE++ in Table VIII, with the input images at $1024\times 1024$ and the number of cells as 32. It is observed that the complexity of LORE is at an equal level to a key point-based detector [33] with the same backbone, showing the efficiency of LORE. The LORE++ is relatively larger since the ViT backbone is employed, but the model size maintains a similar level to the original LORE. Besides, the ablation in Table V has validated that the improvements are not owing to the different backbone networks.

To validate the effectiveness of pre-training LORE in improving data efficiency, we compare the pre-trained LORE++ model with the baseline model LORE at different training settings: training them using 20%, 60%, and 100% training sets of WTW and SciTSR for equivalent 100 epochs regarding the full training set, e.g., we employ 5 times epochs when using 20% data for training compared with using 100% data. The results are shown in Figure 9. As can be seen, LORE++ consistently outperforms the LORE baseline by a large margin in terms of data efficiency. LORE++ using 60% training data achieves comparable performance with LORE using all data on the SciTSR dataset. With the proposed proxy tasks, the pre-trained LORE++ has a grasp of the notion of basic vision and logical clues, which makes learning the TSR task more efficient with less training data.

In this section, we conduct experiments to validate whether the pre-training enhances the generalization ability of LORE. We train the model on a hybrid dataset which contains the WTW training set, 20,000 samples of the PubTabNet training set, and the TableGraph24K training set, and evaluate this model on the ICDAR2013 and SciTSR-comp datasets. The results are displayed in Table IX, which depicts the generalization ability is obviously boosted after pretraining.

In order to reveal the effectiveness of considering the interaction among logical locations, we visualize the structure recognition results of LORE models without/with a stacking logical regressor and the I2C losses. As depicted in Figure 8, models without the stacking regressor and I2C losses encounters problem when predicting the logical location of complicated structures and blank cells, such as the logical errors marked as red lines in Figure 8. While the model with a stacking logical regressor and the I2C losses fixes these errors owning to the stacking regressor refining a rough results of logical locations and knowledge learned from the constrains among logical location of cells.