CED: Catalog Extraction from Documents

We collect three types of documents to construct the proposed dataset, including bid announcements¹¹1http://ggzy.hebei.gov.cn/hbjyzx, financial announcements²²2http://www.cninfo.com.cn and credit rating reports³³3https://www.chinaratings.com.cn and https://www.dfratings.com. We adopt Acrobat PDF reader to convert PDF files into docx format and use Office Word to make annotations. Annotators are required to: 1) remove running titles, footers (e.g., page numbers), tables and figures; 2) annotate all headings with different outline styles; and 3) merge mis-segmented paragraphs. To reduce the annotation bias, each document is assigned to two annotators, and an expert will check the annotations and make final decisions in case of any disagreement. Due to the length and structure variations, one document may take up to twenty minutes for an annotator to label.

After the annotation process, we use pandoc⁴⁴4https://pandoc.org and additional scripts to parse these files into program-friendly JSON objects. We insert a pseudo root node before each object to ensure that every document object has only one root. In real applications, documents are usually in PDF formats, which are immutable and often image-based. Using OCR tools to extract text contents from those files is a common practice. However, the OCR tools often split a natural sentence apart when a sentence is physically cut by line breaks or page turnings in PDF, as shown in Figure 1. To simulate real-world scenarios, we randomly sample some paragraphs with a probability of 50% and chunk them into segments. For heading strings, we chunk them into segments with lengths of 7 to 20 with jieba⁵⁵5https://github.com/fxsjy/jieba assistance. This makes heading segmenting more natural, for example, “招标公告” will be split into “招标 (zhao biao)” and “公告 (gong gao)” instead of “招 (zhao)” and “标公告 (biao gong gao)” . For other normal texts, we split them into random target lengths between 70 and 100. Since the workflow is rather complicated, we will open-source all the processing scripts to help further development.

In addition to the above manually annotated data, we collect 665,355 documents from Wikipedia⁶⁶6https://dumps.wikimedia.org/zhwiki/20211220/ for model pre-training. Most of these documents are shallow in catalog structures and short in text lengths. We keep documents with a catalog depth from 2 to 4 to reach higher data complexity, so that these documents are more similar to the manually annotated ones. After that, 214,989 documents are obtained. We chunk these documents in the same way as the manually annotated ones to simulate OCR segmentation.

Table 1 lists the statistics of the whole dataset. Among the three types, BidAnn has the shortest length and the shallowest structure, and the headings are similar to each other. FinAnn is more complex in structure than BidAnn and contains more nodes. Moreover, there are many forms of headings in FinAnn without obvious patterns, which increases the difficulty of catalog extraction. CreRat is the most sophisticated one among all types of data. Its average length is 8.5 times longer than BidAnn while the average depth is 4.59. However, it contains fewer variations in headings, which may be easier for models to locate. Compared to manually annotated domain-specific data, Wiki is easy to obtain. The structure depth is similar to that of FinAnn while its length is 1.5k shorter. Because of the large size, Wiki is well suited for model pre-training and parameter initializing.

It is worth noting that leaf nodes can be heading or normal texts in catalog trees. Since normal texts cannot lead a section, all texts are leaf nodes in catalogs. However, headings could also be leaf nodes if the leading section has no children. Such a phenomenon appears in approximately 24% of documents. Therefore one node cannot be recognized as a text node simply by the number of children, which makes the CED task more complicated.

The transition-based method is designed for parsing trees from extremely long texts. Since the average length of our CreRat documents is approximately 15k Chinese characters, popular global optimized tree algorithms are apparently too costly to be utilized here.

Action design plays an important role in our transition-based method. There are two buffers here: 1) the input queue $\mathcal{Q}$ providing one text segment $q$ at each time; and 2) the tree buffer $\mathcal{S}$ that records the final catalog tree, where the current stack top points to $s$. Actions are obtained by comparing $s$ and $q$ continuously, which results in the buffer changing. As the comparison process continues, actions compose a control sequence to build the target catalog tree simultaneously.

To solve the mentioned challenges, actions are designed to distinguish between headings and texts. Our actions can also capture the difference between headings from adjacent depth levels. In this way, we construct the catalog tree without regard to its depth and complexity. Additionally, we propose an additional action for text segment concatenation. Based on these facts, we design 4 actions as follows:

• •

  Sub-Heading: current input text $q$ is a child heading node of $s$;

• •

  Sub-Text: current input text $q$ is a child text node of $s$;

• •

  Concat: current input text $q$ is the latter part of $s$ and their contents should be concatenated;

• •

  Reduce: the level of $q$ is above or at the same level as $s$, and $s$ should be updated to its parent node.

An example is provided in Table 2. To start the prediction, a Root node is given in advance. The first heading Credit Rating Report is regarded as a child of Root. Then, Debt Situation becomes another heading node. After that, the Sub-Text action suggests that The balance is the child node of Debt Situation as the body text. Action Concat concatenates two body text. Next, action Reduce leads to the second layer from the third one. We can eventually build a catalog tree with such a sequence of actions. Furthermore, we present two constraints to avoid illegal results. The first one is that the action between Root node and the first input $q$ can only be Sub-Heading or Sub-Text; Another constraint restricts text nodes to be leaf nodes in the tree, and only Reduce and Concat actions are allowed when $s$ is not a heading. If the predicted action is illegal, we take the second-best prediction as the final result.

As Figure 2 shows, the given inputs $s$ and $q$ are encoded via a pre-trained language model (PLM). Here, we use a light version of Chinese whole word masking RoBERTa (RBT3) [2] to obtain encoded representations $\boldsymbol{s}$ and $\boldsymbol{q}$. After concatenation, $\boldsymbol{g}=\boldsymbol{s}||\boldsymbol{q}$ is fed into Feed-Forward Networks (FFN). The FFN is composed of two linear transform layers with ReLU activation function and dropout. Finally we adopt the softmax function to obtain the predicted probabilities as shown in Equation 4.2.

where $\mathcal{A}$ denotes all the action candidates. In this way, we can capture the implicit semantic relationship between two nodes.

During prediction, we take the action with maximal probability $p$ as the predicted result:

where $a_{i}\in\mathcal{A}$ is the predicted action. As discussed in § 4.1, we use two extra constraints to help force decoding legal action results. If $a_{i}$ is an illegal action, we sort the predicted probabilities in reverse order, and then find the legal result with the highest probability.

As for training, we take cross entropy as the loss function to help update the model parameters:

where $\mathbb{I}$ is the indicator function, $y_{a}$ is the gold action, and $a_{i}\in\mathcal{A}$ is the predicted action.

We further split the datasets into train, development, and test sets with a proportion of 8:1:1 for training. To fully utilize the scale advantage of the Wiki corpus, we use it to train the model for 40k steps and subsequently save the PLM parameters for transferring experiments.

We use the overall micro F1 score on predicted tree nodes to evaluate performances. Each node in a tree can be formulated as a tuple: (level, type, content), where level refers to the depth of the current node, type refers to the node type (either Heading or Text), and content refers to the string that the node carries. The F1 score can be obtained by comparing gold tuples and predicted tuples.

where $N_{r}$ denotes the number of correctly matched tuples, $N_{g}$ represents the number of gold tuples and $N_{p}$ denotes the number of predicted tuples.

Few studies focus on the catalog extraction task, thus we propose two baselines for objective comparisons.

1) Classification Pipeline: The catalog extraction task can be formulated in two steps: segment concatenation and tree prediction. For the first step, we take the text pairs as input and adopt the [CLS] token representation to predict the concatenation results. Suppose the depth of a tree is limited, the depth level can be regarded as a classification task with $\textrm{MaxHeadingDepth}+1$ labels, where “1” stands for the text node label. We use PLM with TextCNN [9] to make level predictions.

2) Tagging: Inheriting the idea of two-step classification from above, the whole task can be formulated as a tagging task. The segment concatenation sub-task reflects the BIO tagging scheme, and the level depth and node type are tagging labels. We use PLM with LSTM and CRF to address this tagging task.

Experiments are conducted with an NVIDIA TitanXp GPU. We use RBT3⁷⁷7https://huggingface.co/hfl/rbt3, a Chinese RoBERTa variation, as the PLM. We use AdamW [10] to optimize the model with a learning rate of 2e-5. Models are trained for 10 epochs. The training batch size is 20, and the dropout rate is 0.5. We take 5 trials with different random seeds for each experiment and report average results on the test set with the best model evaluated on the development set. For the classification pipeline and the tagging baselines, we set the maximal heading depth to 8.

From Table 3, we find that our proposed TRACER outperforms the classification pipeline and tagging baselines by 5.305% and 4.121% overall F1 scores. The pipeline method requires two separate steps to reveal catalog trees, which may accumulate errors across modules and lead to an overall performance decline. Although the tagging method is a stronger baseline than the pipeline one, it still cannot match TRACER. The reason may be the granularities that these methods focus on. The pipeline and the tagging methods directly predict the depth level for each node, while TRACER pays attention to the structural relationships between each node pair. Besides, since the two baselines need a set of predefined node depth labels, TRACER is more flexible and can predict deeper and more complex structures.

As discussed in § 4.1, we use two additional constraints to prevent TRACER from generating illegal trees. The significance of these constraints is presented in the last line of Table 3. If we remove them, the overall F1 score drops 0.794%. The decline is expected, but the variation is small, which shows the robustness of the TRACER model design.

Interestingly, the PLM trained on the Wiki corpus does not bring performance improvements as expected. This may be due to the different data distributions between Wikipedia and our manually annotated ChCatExt. The following transferring analysis section § 5.6 contains more results with WikiBert.

One of our motivations for building a model to solve the CED task is that we want to provide a general model that fits all kinds of documents. Therefore, we conduct transfer experiments under different settings to find the interplay among different sources with diverse structure complexities. The results are listed in Table 4 to 7.

We first train models on three separate source datasets and make direct predictions on target datasets. From the left part of Table 4, we can obtain a rough intuition of the data distribution. The model trained on BidAnn makes poor predictions on FinAnn & CreRat, and gets only 7.391% and 2.361% F1 scores, which also conforms with former discussions in § 3.2. BidAnn is the easiest one among the three sources of datasets, so the generalization ability is less robust. FinAnn is shallower in structure, but it contains more variations. The model trained on FinAnn only obtains a 69.249% F1 score evaluated on FinAnn itself. However, it gets better results on BidAnn (25.557%) and CreRat (14.420%) than the others. The model trained on CreRat gets 92.790% on itself. However, it does not generalize well on the other two sources. We also provide the zero-shot cross-domain results from Wiki to the other three subsets. Although the results are poor under the zero-shot setting, the pre-trained WikiBert shows great transfer ability. Comparing results horizontally in Table 4, we find that the pre-trained WikiBert could provide good generalization and outperforms the vanilla TRACER among 6 out of 9 transferring data pairs. The other 3 pairs’ results are very close and competitive.

To further investigate the generalization ability of pre-training on the Wiki corpus, we take an extreme condition into consideration, where only a few documents are available to train a model. In this case, as shown in Table 5, we train models with only k source documents and calculate the final evaluation results on the whole target test set. Each model is evaluated on the original source development set to select the best model and then the best model makes final predictions on the target test set. TRACER w/ WikiBert outperforms vanilla TRACER among 23 out of 27 transferring pairs. There is no obvious upward trend when increasing k from 3 to 10, which is unexpected and suggests that the model may suffer from overfitting problems on such extremely small training sets.

In most cases of real-world applications, a few target documents are available. Supposing we want to transfer models from source sets to target sets with k target documents available, there are two possible methods to utilize such data. The first one is to train on the source set, and then further train with k target documents; the other one is to concatenate the source set and k targets into a new train set. We conduct experiments under these two settings. The results are presented in Table 6 and 7. Comparing the vanilla TRACER model results, we find that concatenating has 10 out of 18 pairs that outperform the further training method. From k=3 to 10, there are 2, 3, and 5 pairs that show better results, indicating that the concatenation method is better as k increases. WikiBert has different effects under these two settings. In the further training method, WikiBert is more powerful (11 out of 18 pairs), while it is less useful in the concatenation method (8 out of 18 pairs).

Overall, we find that: 1) WikiBert achieves good performances, especially when the training set is small; 2) If there are k target documents available besides the source set, WikiBert is not a must, and concatenating the source set with k targets to make a new train set may produce better results.

The left part of Figure 3 shows the average results on each separate dataset with different training data scales. Although BidAnn is the smallest data, the model still gets a 63.460% F1 score and surpasses the other datasets. Interestingly, a decline is observed in BidAnn when the number of training documents increases from 40 to 80. We take it as a normal fluctuation representing a performance saturation since the performance standard deviation is 4.950% when the training data scale is 40. Besides, we find that TRACER has good performance on CreRat. This indicates that TRACER performs well in datasets with deeply nested documents if the catalog heading forms are less varied. In contrast, TRACER is lower in performance on FinAnn than BidAnn and CreRat, and it is more data-hungry than other data sources. For ChCatExt, the merged dataset, performance grows slowly with the increase of training data scale, and more data are needed to be fully trained. Comparing the overall F1 performance of 82.390% on the whole ChCatExt, the small scale of the training set may lead to a bad generalization.

From the right bar plot of Figure 3, it is interesting to see the F1 scores are 0% in level 1 text and level 5 heading. This is mainly due to the golden data distribution characteristics that there are no text nodes in level 1, and there are few headings in deeper levels, leading to zero performances. The F1 score on level 2 text is only 43.938%, which is very low compared to the level 3 text result. Considering that there are only 6.092% of text nodes among all the level 2 nodes, this indicates that TRACER may be not robust enough. Combining the above factors, we find that the overall performance increases from level 1 to 2 and then decreases as the level grows deeper. To reduce the performance decline with deeper levels, additional historical information needs to be considered in future work.