SEMv2: Table Separation Line Detection Based on Instance Segmentation

Early datasets for addressing TSR include UW-3 [14], UNLV [15], ICDAR-2013 [4], ICDAR-2019 [16] and TabStructDB [17]. However, the magnitude of these datasets is limited. To meet the requirement of data-driven approaches for TSR, large-scale datasets such as Table2Latex [18], TableBank [19] and PubTabNet [6] are proposed, but incomplete annotations still impede their development. For instance, TableBank collects 145,463 training tables from the Word and Latex documents. Each table in TableBank solely presents its corresponding HTML tag sequence, devoid of any physical coordinate information. Recently, FinTabNet [7], SciTSR [5] and PubTables-1M [20] add the cell coordinates and row-column information to become relatively comprehensive datasets for TSR. Of particular significance is PubTables-1M, which collects nearly a million fully annotated tables sourced from scientific articles. These encompass comprehensive information for table detection, recognition and functional analysis (such as column headers, projected rows and table cells). Due to the inconsistency in annotations among these datasets, the efficacy of the model is compromised. [21] also aligns these benchmark datasets through removing both errors and inconsistency between them, which improves model performance. Although dataset scale has been significantly increased, these datasets solely focus on digital documents (e.g., PDF files). Recently, the WTW [11] dataset is introduced, which contains tables in multiple real scenes. However, it mainly focuses on wired tables, ignoring the more challenging wireless ones. To comprehensively evaluate the TSR performance, we present a new large-scale dataset iFLYTAB. Different from WTW, tables in iFLYTAB encompass both wired and wireless tables in various scenarios.

Due to the rapid development of deep learning in documents, many deep learning-based TSR approaches [6, 22, 10, 23] have been presented. These methods can be roughly divided into three categories: bottom-up methods, image-to-markup based methods and split-and-merge based methods.

One group of bottom-up methods [24, 22, 25, 26, 27] treat words or cell contents as nodes in a graph and use graph neural networks to predict whether each sampled node pair is in the same cell, row, or column. These methods rely on the assumption that the bounding boxes of words or cell contents are available as additional inputs, which are not easy to obtain from table images directly. To eliminate this assumption, another group of methods [23, 9, 28, 29, 30, 31] proposed to detect the bounding boxes of table cells directly. After cell detection, they designed some rules to cluster cells into rows and columns. However, these methods regard the cells as bounding box, which is difficult to handle the cells in distorted tables. Other methods [32, 33, 34] detect cells through detecting the corner points of cells. they can more suitable for distorted cells, but they suffer from tables containing a lot of empty cells and wireless tables.

The image-to-markup based methods [6, 18, 35, 36, 37] treat table structure recognition as a task similar to image-to-markup generation and directly generate the markup tags that define the structure of the table through an attention-based structure decoder. These methods rely on a large amount of training data and are inefficient as the number of table cells increases.

The split-and-merge based methods [8, 10] first split a table into the basic table grid pattern, and then merge grid elements to recover table cells. Previous methods [8, 10] utilize semantic segmentation [13] for identifying rows, columns within tables in the “split” stage. However, segmenting table row/column separation lines in a pixel-wise manner is inaccurate due to the limited receptive field, and heuristic mask-to-line modules designed with strong assumptions in split stage make these methods work only on tables in digital documents. To more accurately split table grids even in distorted tables, RobustTabNet [12] uses a spatial CNN-based separation line predictor to propagate contextual information across the entire table image in both horizontal and vertical directions. TSRFormer with SepRETR [38] formulates the table separation line prediction as a line regression problem and regresses separation line by DETR, but it can’t regress too long separation line well. TSRFormer with DQ-DETR [39] progressively regresses separation lines, which further enhances localization accuracy for distorted tables. GrabTab [40] flexibly fuses multiple components to robustly predicate cell edges. TRACE [41] first segments the cell corners, explicit lines and implicit lines, then obtains the grid lines from the segmentation result by post-processing, but it needs detailed labels to supervise the train progress which are not available in public datasets. In our work, we formulate the table separation line detection as the instance segmentation task. The table separation line can be accurately obtained by processing the table separation line mask in a row-wise/column-wise manner.

Instance segmentation is a challenging task, as it necessitates instance-level and pixel-level predictions simultaneously. The dominant framework for instance segmentation is Mask R-CNN [42], which first detects the bounding boxes of objects and then segments the object in the box. Many works [43, 44, 45] with top performance are built on Mask R-CNN. Due to the slender shape of table separation lines, this widely utilized box-anchor based instance segmentation methods cannot be employed directly. Another approach to instance segmentation is based on dynamic filter network [46]. For example, SOLOv2 [47] and CondInst [48] learn instance-dependent convolutional kernels, which are applied to generate instance masks. Inspired by CondInst, we aim to resolve the table row/column instance-level discrimination problem, and propose the conditional table separation line detection strategy.

Given an input table image $\bm{I}\in\mathbb{R}^{H\times W\times 3}$, as illustrated in Figure 5, the objective of the splitter is to predict the table grid structure with a set of grid bounding boxes $\bm{B}\in\mathbb{R}^{M\times N\times 4}$, where $M$, $N$ are the number of rows and columns occupied by the table grid structure respectively. Previous split-and-merge based methods apply the semantic segmentation to predict all table row/column separation lines in one mask and subsequently represent the intersection of detected row/column separation lines as grid bounding boxes $\bm{B}$. In contrast to prior methods, we formulate table separation line detection as an instance segmentation task and endeavor to predict an individual mask for each table row/column separation line.

The overall architecture of our splitter is depicted in Figure 5. The ResNet-34 [50] with FPN [51] is utilized to generate a feature pyramid with four feature maps $\{\bm{P}_{2},\bm{P}_{3},\bm{P}_{4},\bm{P}_{5}\}$, whose scales are 1/4, 1/8, 1/16, 1/32 respectively. To amalgamate the information from all levels of the FPN pyramid into a single output $\bm{F}$, we also propose a straightforward fuse operation as follows:

where $\text{Up}_{{n}\times}$ denotes the $n$ times bilinear upsample operation. $\bm{F}\in\mathbb{R}^{\frac{H}{4}\times\frac{W}{4}\times C}$, where $C$ denotes the number of feature channels.

Inspired by CondInst [48], we decouple the table separation line mask generation into a feature branch and a kernel branch. The feature branch contains two $1\times 1$ convolution layers for generating $\bm{F}^{\text{col}}$/$\bm{F}^{\text{row}}\in\mathbb{R}^{\frac{H}{4}\times\frac{W}{4}\times C}$, which will be convoluted with convolution kernels from kernel branches to predict separation line masks. Since the table separation lines are usually slender and traverse the entire table image, it is necessary to design a kernel branch that has a broader receptive field. To address this issue, we propose the Gather module to capture the horizontal/vertical visual clues as shown in Figure 6.

Taking the Column Gather as an example, we first conduct three repeated down-sampling operations on $\bm{F}$, and each operation is composed of a sequence of a $2\times 1$ max-pooling layer, a $3\times 3$ convolutional layer and a ReLU activation function. The down-sampled feature map $\bm{\tilde{F}}^{\text{col}}\in\mathbb{R}^{\frac{H}{32}\times\frac{W}{4}\times C}$ will be taken as the input of two following spatial CNN modules [52]. The first spatial CNN module divides the feature map into ${H}/{32}$ slices, which are denoted as $\bm{S}^{\text{td}}=\left\{\left.\bm{s}_{i}^{\text{td}}\in\mathbb{R}^{1\times\frac{W}{4}\times C}\right|i=1,2,...,\frac{H}{32}\right\}$. Specifically, the topmost slice $\bm{s}^{\text{td}}_{1}$ is convolved by a $1\times 5$ convolution layer, and its output feature map is merged with the next slice $\bm{s}^{\text{td}}_{2}$ by element-wise addition. This procedure is done iteratively so that the information can be propagated from the topmost to the bottommost effectively. The second spatial CNN module transmits information in a reversed direction. In this way, each pixel in the output feature map can leverage the structural information from both sides to enhance its feature representation ability. $\bm{G}^{\text{col}}\in\mathbb{R}^{1\times\frac{W}{4}\times C}$ is obtained by taking the row mean of the enhanced feature map. We add a linear transformation following the $\bm{G}^{\text{col}}$ to predict $C$-dimensional output $\boldsymbol{\theta}^{\text{col}}\in\mathbb{R}^{1\times\frac{W}{4}\times C}$. $\boldsymbol{\theta}^{\text{col}}$ will be used as the weights of a $1\times 1$ convolution layer to predict table column separation line masks. We also detect table column separation line instance by predicting $\bm{\hat{p}}^{\text{col}}\in\mathbb{R}^{1\times\frac{W}{4}\times 1}$ through a linear transformation. The loss function on $\bm{\hat{p}}^{\text{col}}$ is formulated as follows:

where $L_{\text{bce}}$ is the binary cross-entropy loss, $\tilde{\bm{p}}^{\text{col}}$ denotes the ground-truth distribution of starting points of table column separation lines on the x-axis. $\tilde{p}^{\text{col}}_{i}$ is 1 if the start point of a table column separation line is located in the $i$-th column, otherwise 0. To eliminate the duplicated predictions of the starting point of a table column separation line in $\tilde{\bm{p}}^{\text{col}}$, as shown in Figure 7(a), we perform non-maximum suppression as follows: 1) binarize the $\hat{\bm{p}}^{\text{col}}$ into $\bm{p}^{\text{col}}$, 2) for the continuous pixels whose value equals $1$ in $\bm{p}^{\text{col}}$, the pixel with maximum score in $\bm{\hat{p}}^{\text{col}}$ will be selected to represent a table column separation line instance.

According to the detected table separation line instance, we select convolution kernels from $\boldsymbol{\theta}^{\text{col}}$ to conduct dynamic convolution with $\bm{F}^{\text{col}}$ to predict table column separation line masks $\bm{\hat{F}}^{\text{col}}\in\mathbb{R}^{\frac{H}{4}\times\frac{W}{4}\times N^{\text{col}}}$, where $N^{\text{col}}$ represents the number of detected table column separation line instances. The loss function on $\bm{\hat{F}}^{\text{col}}$ is defined as follow:

in which

where $\bm{\tilde{F}}^{\text{col}}$ denotes the ground-truth of table column separation line masks. ${\tilde{{F}}}_{i,j,k}^{\text{col}}$ is $1$ if the pixel in $i$-th row, $j$-th column and $k$-th channel belongs to $k$-th table column line, otherwise 0. The function $L_{\text{f}}$ is actually the sigmoid focal loss [53] and the $\sigma$ is the sigmoid function.

Considering the table column separation lines are typically spread through vertical direction, we process $\bm{\hat{F}}^{\text{col}}$ in a row-wise manner to obtain the final table column separation line. Specifically, as shown in Figure 7(b), given a table column separation line mask, we first find the maximum score of each row, which is presented as a set of red dots as shown in Figure 7(b). The table column separation line can be obtained by connecting these red dots together. The grid bounding boxes $\bm{B}$ can be derived from the intersection of table row/column separation lines.

The embedder aims to extract the grid-level feature representations $\bm{E}\in\mathbb{R}^{M\times N\times D}$, where $D$ is the number of feature channels. As shown in Figure 8, we take the image-level feature map $\bm{F}$ and the well-divided table grids $\bm{B}$ obtained from the splitter as input, and apply the RoIAlign [42] to extract a fixed size $R\times R$ feature map $\bm{\hat{e}}_{i,j}\in\mathbb{R}^{R\times R\times C}$ for each grid.

Then two linear transformations with a ReLU function are conducted on $\bm{\hat{e}}_{i,j}$ to obtain the $D$-dimension output:

where $\bm{W}_{1}$ and $\bm{W}_{2}$ are learned projection matrices, $\bm{b}_{1}$ and $\bm{b}_{2}$ are learned biases. So far, the features of each basic table grid are still independent of each other. Therefore, we introduce the transformer [54] to capture long-range dependencies on table grid elements and utilize its output as the final grid-level features $\bm{E}$.

The merger takes the grid-level features $\bm{E}$ as input and yields a set of merged maps, which can be formulated as: $\bm{M}=\left\{\bm{m}_{1,1},\cdots,\bm{m}_{M,N}\right\},\bm{m}_{i,j}\in\left\{0,1\right\}^{M\times N},i\in\left\{1,\cdots,M\right\},j\in\left\{1,\cdots,N\right\}$. Following the approach of the splitter, as illustrated in Figure 9, we use a feature branch and a kernel branch to predict $\bm{M}$ jointly. Each branch contains only one $1\times 1$ convolution layer to generate feature maps $\bm{E}^{f}\in\mathbb{R}^{M\times N\times D}$ and kernel parameters $\bm{E}^{k}\in\mathbb{R}^{M\times N\times D}$. We first convolute the feature map $\bm{E}^{f}$ with kernel parameter $\bm{e}^{k}_{i,j}$, which is utilized as the weights of a $1\times 1$ convolution layer, to predict the merged map $\bm{\hat{m}}_{i,j}\in\left[0,1\right]^{M\times N}$. The loss function is formulated as follows:

where $\lVert\cdot\rVert_{1}$ is the L1 norm, the function $L_{\text{f}}$ has been defined in Eq. (6), $\tilde{\bm{m}}_{i,j}\in\left\{0,1\right\}^{M\times N}$ denotes the merged map ground-truth of the $i$-th row, $j$-th column table grid. If the value of $\tilde{\bm{m}}_{i,j}$ is equal to $1$, then it indicates that the corresponding grid is associated with the identical table cell; otherwise, 0. The final merged map $\bm{m}_{i,j}$ can be obtained by binarizing $\bm{\hat{m}}_{i,j}$.

SciTSR comprises of 12,000 training samples and 3,000 testing samples of axis-aligned tables extracted from the scientific literature. Furthermore, to reflect the model’s ability of recognizing complex tables, it also selects all the 716 complex tables from the test set as a more challenging test subset, called SciTSR-COMP. It is worth noting that the test set of SciTSR encompasses the presence of annotation errors. To ensure a more accurate evaluation of the model’s performance, we follow the RobusTabNet [12] and rectify the annotation errors present within the test set. Thus, it is pertinent to acknowledge that a comparison with other methods on the SciTSR dataset may entail a degree of unfairness, while the comparison on PubTabNet will emerge as a more equitable benchmark.

PubTabNet is a large-scale table recognition dataset, which contains 500,777 training samples and 9,115 validating samples. PubTabNet annotates each table image with information about both the structure of table and the text content with position of each non-empty table cell. All tables are also axis-aligned and collected from scientific articles.

cTDaR TrackB1-Historical [16] contains 600 training samples and 150 testing samples. It is worth noting that the table images utilized in this dataset are historical documents with handwritten. This dataset provides the physical coordinates and structure information for each table cell.

WTW contains 10,970 training images and 3,611 testing images collected from wild complex scenes. This dataset focuses on wired tabular objects only and provides the annotated information of tabular cell coordinates, and row/column information.

iFLYTAB is the proposed dataset in this paper, which contains 12,104 training samples and 5,187 testing samples. We provide comprehensive annotation for each table image including physical coordinates and structure information. However, it is worth noting that iFLYTAB does not provide annotations for the textual content within table images. In addition to the axis-aligned digital documents, the collected table images also include images taken by cameras, which are more challenging due to the complicated background and non-rigid image deformation.

We use F1-Measure [55], Tree-Edit-Distance-based Similarity (TEDS) [6], WAvg.F1 [16] and GriTS [56] metric, which are commonly adopted in table structure recognition literature and competitions, to evaluate the performance of our model for recognition of the table structure.

In order to use the F1-Measure, the adjacency relationships among the table cells need to be detected. F1-Measure measures the percentage of correctly detected pairs of adjacent cells, where both cells are segmented correctly and identified as neighbors. When evaluating on the WTW dataset, we use the cell adjacent relationship metric [57]. This metric is a variant of F1-Measure that maps a groundtruth cell with a predicted cell according to the Intersection over Union (IoU). Here we use IoU=0.6.

TEDS measures the similarity of the tree structure of tables. While using the TEDS metric, we need to present tables as a tree structure in the HTML format. Finally, TEDS between two trees is computed as:

where $T_{a}$ and $T_{b}$ are the tree structure of tables in the HTML formats. EditDist represents the tree-edit distance [58], and $\lvert T\rvert$ is the number of nodes in $T$.

Since taking OCR errors into account may lead to an unfair comparison due to the different OCR models used by various TSR methods, we also employ a modified version of TEDS, called TEDS-Struct. The TEDS-Struct assesses the accuracy of table structure recognition, while disregarding the specific outcomes generated by OCR.

While using the WAvg.F1, the precision, recall and F1 value of cell detection and cell pair relation prediction need to be calculated at IoU thresholds of [0.6, 0.7, 0.8, 0.9]. The weighted average F1 (WAvg.F1) value of all IoU thresholds is defined as:

The recently proposed GriTS metric compares predicted tables and ground truth directly in matrix form and can be interpreted as an F-score over the correctness of predicted cells. Exact match accuracy considers the percentage of tables for which all cells, including blank cells, are matched exactly.

In our experiments, we align the official provided text contents to the predicted table cells according to the IOU metric. Ultimately, the output for each table image encompasses the physical coordinates of predicted cell bounding boxes, accompanied by spanning information its corresponding content. It is worth noting that the iFLYTAB dataset does not provide text content annotation. Consequently, during the evaluation of iFLYTAB, we assign a distinctive marker to each text line, signifying its individual content. The evaluation code has been made available to the public and can be accessed at the following link: ¹¹1https://github.com/ZZR8066/SEMv2.

Label of Splitter Distinct from previous methods, we formulate table separation line detection as an instance segmentation task and endeavor to predict an individual mask for each table row/column separation line. Two labels necessitate generation to guide the training of the splitter, namely the table row/column separation line masks $\bm{\tilde{F}}^{\text{row}}$/$\bm{\tilde{F}}^{\text{col}}$ and table row/column line instance $\tilde{\bm{p}}^{\text{row}}$/$\tilde{\bm{p}}^{\text{col}}$.

Following the SEM [10], the table separation line mask $\bm{\tilde{F}}^{\text{row}}$/$\bm{\tilde{F}}^{\text{col}}$ are designed to maximize the size of the separator regions without intersecting any non-spanning cell content, as shown in Figure 10.

The $\tilde{\bm{p}}^{\text{row}}$/$\tilde{\bm{p}}^{\text{col}}$ is utilized to distinguish different table row/column separation lines. As depicted in Figure 11, we define $\tilde{\bm{p}}^{\text{row}}$/$\tilde{\bm{p}}^{\text{col}}$ as the projection of the start points of table row/column separation lines on the y-axis/x-axis.

Label of Merger Since we obtain the label of the splitter, we can partition the table into a series of basic table grids. According to the row/column information provided by the original table structure annotation, we can parse which table grids belong to the same table cell.

The ResNet-34 [50] as our backbone is pre-trained on ImageNet [59]. The number of feature channels $C$ and $D$ is set to 256 and 512 respectively. The pool size $R\times R$ of RoIAlign in the embedder is set to $3\times 3$. The hyperparameters $\alpha$ and $\gamma$ of sigmoid focal loss $L_{\text{f}}$ are set to 0.25 and 2. The threshold for binarization operations is set to 0.5.

The training objective of our model is to minimize the table row/column separation line segmentation loss ($\mathscr{L}_{s}^{\text{row}}$/$\mathscr{L}_{s}^{\text{col}}$), the table row/column separation line instance classification loss ($\mathscr{L}_{\text{inst}}^{\text{row}}$/$\mathscr{L}_{\text{inst}}^{\text{col}}$), and the cell merge loss ($\mathscr{L}_{m}$). The objective function for optimization is shown as follows:

We employ the ADADELTA algorithm [60] for optimization, with the following hyper parameters: ${\beta_{1}}=0.9$, ${\beta_{2}}=0.999$ and $\varepsilon={10^{-9}}$. We set the learning rate using the cosine annealing schedule [61] as follows:

where ${\eta}_{t}$ is the updated learning rate. ${\eta_{min}}$ and ${\eta_{max}}$ are the minimum learning rate and the initial learning rate, respectively. $T_{cur}$ and $T_{max}$ are the current number of iterations and the maximum number of iterations, respectively. Here we set ${\eta_{min}}=10^{-6}$ and ${\eta_{max}}=10^{-4}$.

In our experiments, we do not train the model using the training sets of all the datasets, but rather train it on the training sets of each dataset and test it on the test sets of each dataset. Following the SEM [10], SEMv2 is trained with table images in original size on the SciTSR and PubTabNet. However, in the case of the iFLYTAB dataset, we resize the table images using a randomly selected ratio within the range of $[0.8,1.2]$. Different from the training strategy employed by TSRFormer [38], which follows a multi-stage process, we train the SEMv2 in an end-to-end manner. We initialize SEMv2 with the model trained on the iFLYTAB dataset and then fine-tune it on the cTDaR TrackB1-Historical dataset. We crop table regions from original images in the WTW dataset for both training and testing. Our training setup includes a single NVIDIA TESLA V100 GPU with 32GB RAM memory and a batch size of 8 for the SciTSR, PubTabNet and cTDaR. For iFLYTAB and WTW, we utilize 8 NVIDIA TESLA V100 GPUs with 32GB RAM memory and a batch size of 48. The whole framework was implemented using PyTorch.

In this section, we visualize the predicted results of both the splitter and the merger to show how SEMv2 recovers the table structure. In our work, we formulate table separation line detection as an instance segmentation task, as exemplified in Figure 12(a-b), where different colors represent SEMv2’s predictions for distinct instances of table separation lines. By finding the maximum score on each column/row of the table row/column separation line mask, we can obtain the table row/column separation lines, as depicted in Figure 12(c-d). As illustrated in Figure 12(e), the table grid structure can be derived by intersecting the table row/column separation lines. Through the merger, the spanning cells structure can be restored, ultimately obtaining the final table structure, as shown in the red dashed boxes in Figure 12(f).

To verify the effectiveness of each component, we conduct ablation experiments through several designed systems as shown in Table 2. The model is not modifed except for the component being tested. As depicted in Table 2, T6 represents our baseline system, essentially representing SEM [10] without the textual branch. On the other hand, T1 corresponds to our proposed SEMv2. As shown in Table 3, by comparing T1 and T6, it is evident that our approach exhibits superiority over SEM in terms of both performance and efficiency.

The effectiveness of the splitter. In contrast to the majority of previous “split-and-merge” based methods, we formulate table separation line detection in the “split” stage as an instance segmentation task rather than a semantic segmentation. To evaluate the efficacy of our proposed splitter, we devis the systems T1 and T2 as shown in Table 2. Specifically, T1 represents our proposed SEMv2, whereas T2 replaces the splitter with the one used in the previous state-of-the-art method, SEM[10]. As shown in Table 3, although the performance of T1 is only marginally better than T2 on datasets comprising axis-aligned scanned PDF documents (e.g., SciTSR), T1 exhibits a significantly superior performance on the iFLYTAB dataset. This is because the iFLYTAB dataset features camera-captured images with severe deformation, bending, or occlusions. Additionally, we present the segmentation results from splitters of both T1 and T2 in Figure 13. It can be seen that both T1 and T2 can get high-quality masks on the digital document, but on the camera-captured document, the prediction result of T2 is relatively lower. It’s very hard for the mask-to-line post-processing module to handle such low-quality masks well. In contrast, our instance segmentation based method can easily obtain the shape of the table separation line in a row-wise or column-wise manner, which is more robust to such challenging tables.

The effectiveness of the Gather. To illustrate the effectiveness of the Gather module, as shown in Table 2, we designed a system T3, which eliminates RowGather/ColGather, and obtains $\bm{G}^{\text{row}}$/$\bm{G}^{\text{col}}$ by calculating the column/row mean value of $\bm{F}$. As shown in Table 4, T1 outperforms T3 by a large margin on both SciTSR and iFLYTAB datasets, which demonstrates the effectiveness of the Gather module for capturing horizontal/vertical visual clues.

The efficiency of the merger. As shown in Table 2, we design the systems T1, T4 and T5 that employ different mergers. T4 eliminates the merger, while T5 substitutes it with the one utilized in SEM[10]. As shown in Table 5, though T4 exhibits slightly higher Frames Per Second (FPS) than T1, its performance deteriorates significantly as it disregards table cells that span multiple rows or columns. The comparison between T1 and T4 also illustrates the indispensability of the merger. The merger in SEM predicts the merging of grids in a step-by-step manner. As the number of table cells increases, the costed time in the decoding stages of T5 rise, causing the FPS of T5 to be much lower than T1 and T4.

We compare our method with other state-of-the-art methods on five TSR datasets, including SciTSR, PubTabNet, cTDaR, WTW and iFLYTAB. The results are shown in Tables 6 7 8 10. Furthermore, we visualize the prediction results of our method as shown in Figure 15. Finally, we discuss the differences between SEMv2 and other methods that adhere to the “split-and-merge” principle.

SciTSR and PubTabNet As shown in Table 6, our method achieves competitive performance compared to state-of-the-art methods. The test dataset of SciTSR exhibits a few annotation errors. Following the RobusTabNet [12], we manually rectify these annotation errors. However, it may result in an unfair comparison with other methods. The more equitable comparison can be observed on the PubTabNet. It is worth noting that the LGPMA [9] emerged as the winner of the ICDAR 2021 Competition on Scientific Literature Parsing, Task B.

cTDaR TrackB1-Historical To verify the effectiveness of our approach on tabular objects in various scenes, we conduct experiments on the cTDaR TrackB1-Historical dataset. The table images in this dataset are historical documents with handwritten. As shown in Table 7, we compare the SEMv2 with the participant teams of ICDAR 2019 Competition on Table Detection and Recognition, TrackB2 [16]. Our method outperforms other teams by a large margin.

WTW To verify the effectiveness of our approach on wired distorted/curved tabular objects in wild scenes, we also conduct experiments on the WTW dataset. The results in Table 8 show that our method achieves comparable performance with state-of-the-art methods. To investigate the performance of SEMv2 in table structure recognition across various scenarios, we conducted tests on a series of subsets divided by WTW as shown in Table 9. For polygon detection methods like Cycle-CenterNet [11], SEMv2 performs better in scenarios such as curved, occluded and blurred, and extreme aspect ratio. However, in overlaid scenes, we found that SEMv2’s performance was inferior. In the overlaid subset, most of the table images exhibit significant angular rotation, which reduces the overall performance of table structure recognition. How to enhance the performance of SEMv2 in scenarios with obvious angle rotation is also a direction for our future research.

iFLYTAB The iFLYTAB dataset is distinct from SciTSR and PubTabNet in that it encompasses table images taken by cameras. These images are typically accompanied by intricate backgrounds and non-rigid deformations, which makes them more challenging. It is worth noting that detection-based methods (e.g. TabStructNet [23], LGPMA [9]) are subject to the constraints that tables are free of visual rotation or perspective transformation. This condition is difficult to satisfy when table images are camera-captured. Therefore, as shown in Table 10, we reimplement the closest method, SEM [10], for a fair comparison on the iFLYTAB dataset. Since the iFLYTAB does not provide the text content annotation, we remove the textual feature in our reimplemented SEM. It can be seen that SEMv2 outperforms SEM by a large margin. As depicted in Figure 14, we also evaluate the performance of SEMv2 on different subsets of the iFLYTAB dataset. Notably, the model’s performance is comparatively lower in the camera-captured scenarios due to the suboptimal image qualities. Furthermore, the model performance is less satisfactory on wireless tables than on wired ones, as the former lacks crucial visual information.

Split-and-merge Previous works such as TSRFormer [38], SEM [10], SPLERGE [8] and RobusTabNet [12] all follow the “split-and-merge” principle. The main difference between these methods is found in their individual split stages. SEM, SPLERGE and RobusTabNet design the splitter based on the semantic segmentation, which requires a complex mask-to-line algorithm to extract table row/column separation lines from the predicted masks. In contrast, SEMv2 designs an instance segmentation-based method in the split stage, which predicts a mask for each table row/column separation line. Considering that most table row/column separation lines are spread through horizontal/vertical direction, we propose a simple “mask-to-line” algorithm, as shown in Figure 7, that can accurately extract table row/column separation lines. TSRFormer formulates separation line prediction as a line regression problem instead of an image segmentation problem. Specifically, TSRFormer predicts reference points and using a DETR decoder to regress line coordinates. As shown in Table 8, TSRFormer and SEMv2 are comparable in performance.