Multi-Cell Decoder and Mutual Learning for Table Structure and Character Recognition

Previous studies [21, 36, 18] used a convolutional neural network (CNN) to extract image features and fed them to the decoder. CNN is useful for recognizing small characters while preserving locality such as character positions, reducing the size of image features, and improving the computational efficiency and performance of the decoder.

The number of convolutional layers contributes to recognition performance, and many derivatives have been explored to increase it. ResNet [6], which consist of a large number of residual blocks of multiple convolutional layers with simple skip connections, has been commonly used. In addition, ResNeXt [34] with group convolution and DenseNet [9] with more complicated skip connections between all convolutional layers were proposed.

One of the weaknesses of CNN is its poor ability to recognize global context by focusing strongly on local features. As a solution, a global context attention (GCA) block [2] was proposed, defined by Eq. (1).

where $i,j$ are the pixel indices, $\bm{x},\bm{y}$ are the input and output pixels, respectively. $W_{1},W_{2},W_{3}$ are the weight matrices of three linear layers. $\mathrm{LayerNorm}$ means layer normalization. The softmax function is defined as follows.

where $m,n$ are the pixel indices. The GCA block should be placed between some residual (or dense) blocks.

Transformer [31] achieves superior performance in both language modeling and visual recognition tasks. Compared to recurrent neural networks including long short-term memory [8], a Transformer itself does not involve recursion, allowing parallel processing of sequential input and output data. It should be noted that recurrent, sequential inference is performed unless the prediction length is fixed. However, Transformer does not require recursion to recognize the context of the sequence, avoiding vanishing gradients and providing better performance.

The key idea of Transformer is called scaled dot product attention. Let $X$ be a sequence of length $l_{x}$ and $d_{x}$ channels, and $Y$ be another input sequence. For self attention, $X,Y$ are the same sequence, and the Transformer pays attention to other parts of $X$ in processing $X$. For cross attention, $X$ and $Y$ are in different domains, and the Transformer pays attention to some parts from $Y$ in processing $X$. These mechanisms allow Transformer to learn the context of sequential data and the relationship between visual and language domains.

The attention layer first generates a query $Q$, key $K$, and value $V$ from $X,Y$ as defined in Eq. (3).

where $i,j$ are the sequence indices, $\bm{q},\bm{k},\bm{v},\bm{x},\bm{y}$ are the elements of $Q,K,V,X,Y$, respectively. $W_{Q},W_{K},W_{V}$ are the projection matrices.

The output $Z$ of the attention layer is defined by Eq. (4).

where $W_{Z}$ is the output projection matrix, and $d_{k}$ is the dimension of $\bm{k}$. This is the mechanism of the scaled dot product attention [31].

In practice, the attention layer is divided into several groups, each of which pays attention independently and combines the outputs at the end. This is called multi-head attention [31]. Through the above mechanism, Transformer can focus on specific values of $Y$ and incorporate them into the $X$ series.

Although Transformer has superior ability to recognize long sequences compared to recurrent neural networks, it is still known to perform poorly upon extremely long sequences. Local attention (LA) [1] is a technique designed to handle such long sequences in Transformer.

Let $M_{ij}$ be a mask to focus on the $j$th element from $i$th element of $X$. The output of the local attention layer is defined by Eq. (5), involving causal masking to prevent leakage from subsequent elements.

The mask matrix $M$ is given by Eq. (6).

where $i,j$ are the sequence indices, and $w$ is the width of the sliding window.

Transformer [31] itself have poor ability to know the position of each element in the sequences, and the position information must be provided explicitly. Instead of inputting simple position values, two approaches have been proposed, namely positional embedding [4] and positional encoding [31]. In general, the latter works better on small training datasets.

The output $\bm{p}(n)$ of positional encoding for the index $n$ is defined by Eq. (7).

$\bm{p}(n)$ must be added directly to the feature vector $\bm{x}(n)$ with $d$ channels.

If the sequence $X$ has two-dimensional positions $(i,j)$, 2D positional encoding proposed by Zhao et al. [40] may be a better choice. It normalizes the horizontal and vertical coordinates to $[0,1]$, encodes each with Eq. (7), and then combines them to obtain a single vector. The positional encoding of the $i,j$th pixel is given by Eq. (8).

where $H,W$ are the height and width for positional normalization. In this paper, we omitted this normalization.

Ensemble learning is commonly used to improve machine learning generalization performance and fitting accuracy by averaging or complementing the outputs of multiple inference models. However, it is computationally more expensive than single models due to the large number of parameters especially for deep learning approaches.

To achieve similar effects using only a single model, knowledge distillation [7] may be an alternative solution. This is a technique which uses a large, complex neural network, i.e., an ensemble model, as a teacher and a small, simple model as a student to obtain higher performance than simply training a student model using the ground-truth data.

Mutual learning [38] may be another solution. Here, multiple student models are trained simultaneously to teach each other, without training a teacher model in advance. In particular, each student model performs supervised learning using ground-truth data and minimizing Kullback–Leibler (KL) divergence [14] so that the distributions of each other’s classification outputs match.

Zhong et al. [42] introduced a tree edit distance based similarity (TEDS) metric for performance evaluation of both table structure and cell content recognition. After converting the recognition results and the ground truth into tree structures of HTML tags, the TEDS score is calculated according to Eq. (9).

where $T_{a}$ and $T_{b}$ are the HTML trees, $\mathrm{EditDist}{}$ is the edit distance function, and $|T|$ is the number of nodes in $T$.

There are two versions of TEDS, namely structural TEDS and total TEDS. The former is calculated for HTML trees excluding cell contents and represents the recognition performance for table structures only. The latter is computed on complete HTML trees including cell contents and indicates the total recognition performance.

In addition, Zhong [42] classified the tables into two subsets, namely simple tables and complex tables. The former are tables without cells which are merged vertically or horizontally, and the latter are the other tables.

The encoder consists of a CNN backbone and 2D positional encoding. The CNN extracts image features of 65x65 pixels from an image of 520x520 pixels. For the CNN, we adopted TableResNetExtra [36] with 26 convolutional layers and three GCA blocks. After 2D positional encoding, the image features are flattened into one-dimensional features with 512 channels for cross-attention at the decoders.

The HTML decoder consists of one embedding layer, three local attention blocks, and two output layers. Each attention block accepts a table structure sequence through a self-attention layer in the block. The attention block then incorporates image features into the table structure sequence through a cross-attention layer, and outputs the sequence through a feed-forward layer. Several skip connections and layer normalizations are inserted within the block. The output from the last attention block is converted into HTML tokens and cell bounding boxes by the two output layers.

During training, the decoder predicts left or right shifted HTML tokens from the input HTML tokens. The shift direction is specified by an additional one-hot vector. During inference, the decoder predicts the following token and iteratively expands the input sequence to obtain the complete HTML sequence.

In addition to HTML tokens, the decoder accepts some special tokens, namely SOS, EOS, and PAD. SOS is a token that triggers sequential inference and is inserted at the beginning of the tokens. EOS is a token that stops inference and is inserted at the end of the tokens. PAD is inserted after EOS to equalize the lengths of the tokens in the mini batch.

Following previous studies [18, 19], the HTML sequence was simply tokenized into HTML tags, except for the <td> tag representing the start of a cell. A <td> tag is tokenized as ‘<td’, ‘colspan="2"’, ‘rowspan="3"’, ‘>’ if it contains colspan or rowspan attributes. Otherwise, the tag is simply tokenized as ‘<td>’. It should be noted that FinTabNet [41] and PubTabNet [42] described in Section 5.1 are publicly available with such tokenization applied. We then merged the <td> and immediately following </td> tokens into one token.

Furthermore, we assigned some special tokens to frequent sequence patterns. If all header cells in the dataset had bold text, we removed the <b> and completed it in post-processing. These methods follow previous studies [18, 19].

The cell decoder consists of one embedding layer, one local attention block, and an output layer. Following previous studies [18, 19], the embedding layer accepts cell characters one by one. This is because cell contents typically consist of short sentences or unknown words or numbers, making it difficult to utilize pretrained language models.

The basic structure of a cell decoder is similar to that of an HTML decoder, with the following differences. First, a special token SEP is inserted between cell contents to trigger movement to the next cells. Second, the cell decoder accepts a combination of cell contents and their corresponding HTML features extracted from the output of the HTML decoder. These improvements allow the proposal to sequentially read the contents of multiple cells while referring to information from previous cells.

The previous study [18] exploited local attention for the HTML decoder and global attention for the cell decoder. This was because the cell decoder processed each cell independently, and the cell contents were short in general. On the other hand, in our proposed multi-cell decoder, the sequence of cell contents tends to be long. Consequently, we employed local attention.

We propose bidirectional mutual learning inspired by deep mutual learning [38] to train the HTML decoder. Here, two equivalent decoders are trained together to predict table structure in either a left-to-right (LtoR) or right-to-left (RtoL) direction. To reduce model parameters, we implemented the mutual learning in a single decoder by combining an additional one-hot vector that determines the direction with the embedded HTML tokens.

Let $\overrightarrow{\bm{x}}$ and $\overleftarrow{\bm{x}}$ be the LtoR and RtoL sequences respectively, and let $p(\bm{x})$ be the ground-truth and $q(\bm{x})$ the predicted probabilities. The loss $\overrightarrow{\mathcal{L}}$ for the LtoR decoder is defined by Eq. (10).

We utilized two large datasets, FinTabNet [41] and PubTabNet [42]. In addition, we used a subset named PubTabNet250 [18] for ablation studies. Table 1 shows the statistics for the datasets.

The proposed model was implemented in PyTorch using mmcv [22], mmdet [23], and mmocr [24] frameworks and trained on four NVIDIA V100 GPUs with batch size 8 in total. We used Ranger [33] optimizer. The learning rate was initialized to 0.001 for the first 25 epochs, and decreased to 0.0001 and 0.00001 for the next three and last two epochs, respectively.

Each tabular image was normalized and reduced to 520x520 pixels, padding the margins with zeros if necessary. The cell bounding boxes were normalized to have a minimum value of 0 and a maximum value of 1.

HTML tokens and cell contents were converted to 512-dimensional embedded representations. The four attention blocks in the HTML and cell decoders have the same 8-head, 512-channel architecture, and the sliding window size for local attention was set to 300 by default, following previous work [18]. The maximum lengths for table structure sequences and cell content sequences were set to 800 and 8000, respectively, including special tokens. We employed greedy search for sequential prediction.

To ensure a fair comparison with the previous studies, we did not utilize data augmentation or ensemble learning techniques. We also did not take advantage of early stopping.

We compared the performance of the proposed model trained on FinTabNet and PubTabNet with the claimed performance of existing models.

We conducted additional experiments for ablation studies using PubTabNet250 dataset for training and PubTabNet subsets for evaluation.