Context-Aware Chart Element Detection

Although the field of view of each anchor would increase with stacking of the convolutional neural network (CNN) layer, it is still limited to the local field, which has fragile context awareness. However, accurate element detection and labeling in chart images require a much larger field of context.(see Fig. 1). For instance, a text block can be classified as a legend label by the context that nearly comes from the whole image, where the legend label is beside some (legend) markers that share the same color with plot elements in the plot area. To address this issue, similar to SCNet [11], we introduce visual context enhancement (VCE) by incorporating the feature maps from the backbone feature pyramid as global visual features and bringing them into each region of interest. These global visual features are combined with each RoI-aligned local feature map to amplify local-global visual context. As shown in Fig. 2(c), we first average pool the feature maps from all stages of the feature pyramid to the same size $[N,256,7,7]$, then concatenate them together, where the shape becomes $[N,256*5,7,7]$. A convolutional neural network further abstracts these global visual features and reduces the number of channels. Then the global feature maps are attached to each RoI-aligned feature map. The feature map for each region of interest with size $[N,512,7,7]$ now consists of local visual features in the first half channels and global visual features in the rest of the channels. A SE-Net and a 1x1 convolutional layer are followed to fuse the local-global visual features channel-wise and reduce the number of channels, respectively. The final feature maps with size $[N,256,7,7]$, which align with the original RoI feature map dimension but contain rich local-global visual context, are used as visual feature representations for each region of interest.

In the general image domain, the detection of common objects such as people, vehicles, and animals depends mainly on visual features, and the positions of the objects in the image have less impact on accurate localization and classification. However, since chart images are highly structured data visualization formats, where object rendering always follows some specific patterns, e.g., x-tick labels are always below or beside the x-tick marks and chart titles typically appear on the edges of the image while rarely appearing in the center part, the relative positional context among the chart elements is very trivial. We propose a Transformer-based positional context encoder to obtain the positional context feature. As Fig. 2 shows, the Transformer-based architecture draws attention among objects’ bounding-box coordinates. Firstly, we normalize the 4-dim coordinates to $[0,1]$ by the height and width of the input images. Then a linear layer takes the normalized coordinates and embeds them into a 512-dim vector. Zero mask paddings are filled in the embedded bbox vector sequence, where the length of the final embedding bbox sequence is fixed to the max block size of 1024. The block size of 1024 is enough to cover the maximum sampling bbox number from the RPN results in training and testing, which is set to a maximum of 1000. We use each head’s output to represent the relative positional context information from the corresponding bbox to all other bboxes. Then, the encoding vector for each bbox is concatenated to its RoI visual feature vectors.

The adobe synthetic dataset was first proposed in 2019 [4] and refined in 2020 Chart-infographic Competition [5]. This dataset is synthesized using Matplotlib and contains 14400 images for 12 types of charts. The annotations include the chart data information and all elements’ label and location. Such annotations are valuable for chart classification, element detection, and data extraction tasks. However, the diversity of chart samples in this dataset is severely limited. The best results showed in ICPR 2020 Chart Competition [5] in chart classification, text role classification, tick mark label association, and legend marker detection are close to 100%, further indicating the limited variance of the data. Low data variance typically causes unstable performance and low generalization from the trained deep-learning model on samples outside the dataset. We only refer to the annotation standards in this dataset for element categorization refinement.

This dataset was released and updated with the Chart Competition in ICDAR 2019 [5], ICPR 2020 [6] and ICPR 2022 [6]. Unlike the Adobe Synthetic dataset, the PMC dataset is a real-world dataset collected from PubMed Central Documents and manually annotated. Taking into account the much more diverse and high-fidelity samples in this dataset, the PMC dataset became the primary dataset in most recent chart competitions. The most up-to-date PMC dataset is released in the ICPR 2022 CHART-Infographic competition [6], which contains 5614 images for chart element detection, 4293 images for final plot detection and data extraction, and 22924 images for chart classification. Although slightly limited by the number of available training samples due to the time-consuming human annotation process, this real-world dataset is most valuable, and much more challenging than most synthetic datasets.

In [24], the author proposed the ExcelChart400k dataset, which contains a total of 360k training samples. This dataset is generated using the Excel data sheet and was used to train the plot element detection model in [24]. However, the dataset is focused only on annotating data plots inside the plot area, and most basic elements are left without annotation. We use this additional dataset to qualitatively evaluate the results of our method.

After conducting early experiments, we observed that including the Adobe synthetic training dataset decreased the detection performance on real-world chart images. This was attributed to the uniform chart rendering patterns and limited sample variance in the synthetic dataset. Taking into account the high diversity and real-world chart data distribution, we use the PMC dataset as our primary dataset for training and quantitative evaluation purposes.

We compare with the results from the ICPR 2022 Chart Competition [6]. However, our method shares different detection routines, where the chart competition treats the text block detection task as a 1-class detection task (task 2) and leaves the recognition as an additional task (task 3). In the chart competition, teams could use the ground truth of previous tasks for the current task. Our method follows the regular detection routine (localization + recognition). After refining the categorization, the overall categories (see Table 1) expand from the original PMC dataset used in the competition. Taking these two facts, we compare by adapting our prediction results backwards to be compatible for evaluation and splitting the prediction result into task 2 and task 3. Table 2 shows the result of the official 2022 chart competition. Our method in task 2 outperforms the best result from ‘UB-ChartAnalysis’. In task 3, our method detects and recognizes all elements once, without taking the element localization ground truth as prior knowledge, and this may result in a lower recall for task 3 due to imperfect detection. Overall, we achieve the best detection result on task 2 and the comparable F-measure score on task 3.

To provide a better general overview of the performance of our method, we use the COCO object detection evaluation metric [21] to evaluate our method with several often used two-stage detectors (shown in Table 3). To our best knowledge, there isn’t any trained chart element detection model conducted on PMC datasets publicly available for inference and fine-tuning. We trained these two-stage detectors on the refined PMC dataset at our end. In Table 3, the first part includes results from popular two-stage detectors. DETR converges slowly in training due to the scale of our dataset and the Transformer architecture. The accuracy of small object detection from DETR also lags behind. An enormous amount of small elements in chart images result in limited overall accuracy. The Cascade R-CNN with Swin-Transformer performs best in the standard Cascade R-CNN model zoo. The second part shows our methods with several ablated setups on the local-global context module, which consists of visual context enhancement (VCE) and positional context encoding (PCE). Since the Swin-Transformer backbone can draw spatial attention and potentially obtain context-aware visual feature representations, we made an ablation by only integrating the positional context encoding (PCE) into the Swin-Transformer-based Cascade R-CNN. As expected, the Swin-Transformer-based Cascade R-CNN with the PCE performs better than the standard Swin-Transformer Cascade R-CNN. Although the Swin-Transformer can obtain context-aware visual features, adding VCE could further enhance the context extraction capability, giving better results. The focal loss with balanced smooth L1 loss can help with sample imbalance problems, and our method with this loss setup achieves state-of-the-art performance (see the last row in Table 3).

Although our goal is to offer a robust basic element detection method, we extend our experiment to bar plot detection. We fine-tune our method using the PMC bar chart subset and test on the PMC test set (see Table 4). The results of the first five models were from [25], and we evaluate our results with the same criteria. The first three columns are F-measure scores with different IoU thresholds - 0.5, 0.7, and 0.9. The last column, ‘Score_a’, is calculated using the ICPR 2022 chart competition [5] evaluation metric for task 6a. Our method achieves state-of-the-art performance on bar chart detection on the PMC dataset.