A Systematic Evaluation of Object Detection Networks for Scientific Plots

Object detection is one of the fundamental problems in computer vision with the aim of answering what objects are where in a given input image. Most of the object detection research in the past few years has been on natural images with real-life objects. For instance, in the PASCAL VOC dataset (Everingham et al. 2010), the four major classes of objects are people, animals, vehicles, and indoor objects such as furniture. In this work, we study object detection for a very different class of images, namely computer-generated scientific plots. Fig. 1b shows an example of a scientific plot: It is a bar plot depicting the number of neonatal deaths in Bulgaria and Cuba over two years. Object detection on this plot would be required to identify different visual and textual elements of the plot such as bars, legend previews and labels for ticks, axes & legend entries. Such object detection can then enable a question-answering task. For instance, for Fig. 1b we could ask “In which year does Cuba have lower neonatal deaths?”. Clearly, this has use-cases in data analytics, and has been studied in recent research (Kafle et al. 2018; Kahou et al. 2018; Methani et al. 2020).

It should be clear that scientific plots differ significantly from natural images. Firstly, plots have a combination of textual and visual elements interspersed throughout. The text can either be very short (such as numerical tick labels) or span multiple lines (such as in plot-titles). Secondly, objects in a plot have a large range of sizes and aspect ratios. Depending on the value represented, a bar in a bar-plot can be short or long, while in a line-plot a thin line could depict the data. Thirdly, plots impose structural relationships between some of the objects. For instance, the legend preview and the close-by title text denote a correspondence map. This also applies to a tick label and its corresponding bar in a bar-plot.

Given these differences, it needs to be seen if existing object detection methods are adequate for scientific plots. In particular, are they capable of (a) detecting short and long pieces of text, (b) detecting objects with large data-dependent range of sizes and aspect ratios, and (c) localising objects accurately enough to extract structural relationships between objects? To answer this question, we first evaluate state-of-the-art object detection networks on the PlotQA dataset (Methani et al. 2020) which has over $220,000$ scientific plots sourced from real-world data thereby having a realistic vocabulary of axes variables and complex data patterns which mimic the plots found in scientific documents and reports. We observe that, across these models, the average of mAP@$0.5$ is only around $87\%$, indicating success in detecting the relatively simple objects in the plot.

While the above results appear positive, a closer manual inspection revealed that these models make critical errors which lead to large errors in downstream numerical inference on the plots. This disparity is because we use an IOU of 0.5 while computing the mAP scores. While IOU values in the range of 0.5 and 0.7 are acceptable for natural images where large relative areas are covered by foreground objects, such values are unacceptably low for scientific plots. This is demonstrated in Fig. 1a with two example images from the PASCAL VOC dataset where the predicted box (red) is very different from the ground-truth box (cyan), but still acceptable as the IOU is within range. Contrast this with the case for the plot in Fig. 1b. For an IOU setting of 0.5 (left) and 0.75 (middle), the estimated values of the data-points would incorrectly identify that neonatal deaths in Cuba are lower in 2002 than the actual value in 2003. Only at the high IOU value of 0.9, this is correctly resolved. Thus, downstream numerical reasoning on plots requires much stricter IOU settings in comparison to object detection on natural images. For the PlotQA dataset if we use a stricter IOU of 0.9, then the mAP scores for all models drop drastically with the best model giving a mAP of only $35.70\%$. In particular, one-stage detectors such as SSD (Liu et al. 2016) and YOLO-v3 (Redmon and Farhadi 2018) have a single-digit [email protected].

The poor performance of current models at high IOU settings motivate improvements in the models. We first propose minor modifications to existing models. In particular, we propose a hybrid network which combines Faster R-CNN (Ren et al. 2015) with the Feature Pyramid Network (Lin et al. 2017a) from RetinaNet (Lin et al. 2017b) and the ROIAlign idea from Mask R-CNN (He et al. 2017). This significantly improves the performance giving an overall mAP of $77.22$%@0.9 IOU. However, careful analysis reveals two major limitations: (i) accuracy on text objects is very low which can lead to errors in downstream analytics tasks, and (ii) the inference time is very high (374 ms) which is unacceptable given the lower visual complexity of plots.

To further improve the speed and performance, we propose an architecture, named PlotNet, which contains (i) a fast and conservative region proposal method based on Laplacian edge detectors, (ii) an enhanced feature representation of region proposals that includes local neighbouring information, (iii) a linking component that combines multiple region proposals for better detection of longer textual objects, and (iv) a custom regression loss function that combines smooth $\ell_{1}$-loss with an IOU-based loss designed for improving localisation at higher IOUs. This significantly improves accuracy with a mAP of 93.44%@0.9 IOU. Further, it is 16x faster than its closest competitor and has 3x lower FLOPs. We also evaluate PlotNet on an extrinsic plot-to-table conversion task where we extract the plot’s underlying data into a table. Specifically, we replace Faster R-CNN with PlotNet in the Visual Element Detection stage of the pipeline proposed in PlotQA (Methani et al. 2020). This results in a relative improvement of 35.08% in the table F1 score.

In summary, the contributions of this paper are as follows:

1. 1.

   We motivate the need for object detection at extreme IOU values for the specific dataset of scientific plots which require accurate localization.

2. 2.

   We evaluate the robustness of nine different object detection networks (SSD, YOLO-v3, RetinaNet, variants of Fast and Faster R-CNN) to increase in IOU and identify that Feature Pyramid Network (FPN) and ROIAlign (RA) are good design choices for higher accuracy.

3. 3.

   We propose PlotNet that improves on mAP by over $16$ points while reducing execution time by over $16$ times from its closest competitor. Thus, PlotNet is faster than one-stage detectors and simultaneously more accurate than the best two-stage detectors.

The rest of the paper is organised as follows: We first discuss the different datasets and detail the experimental setup for evaluating existing object detection models on scientific plots. We also report the results and critique their performance on an example plot image from the PlotQA dataset. We then detail the architecture of PlotNet and compare it with other networks in terms of accuracy and speed. Note that, we follow an unconventional organisation of describing partial experimental results first as this lays the groundwork for motivating the design and evaluation of PlotNet.

The PlotQA dataset (Methani et al. 2020) contains over $220,000$ scientific plots across three categories of bar (both horizontal and vertical), line, and scatter plots. The dataset includes ground-truth bounding boxes for bars, dot-lines, legend labels, legend previews, plot-title, axes ticks and their labels. The underlying data is from data sources such as World Bank Open Data containing natural variable names such as mortality rate, crop yield, country names, and so on.

These models were trained with an initial base learning rate of $0.025$ with momentum stochastic gradient descent algorithm. The network’s classification and regression heads use a batch-size of $512$ ROIs. RetinaNet and SSD models were trained with a batch-size of $32$ with a learning rate of $0.004$. Based on evaluation on the validation dataset, we modified the parameters in the focal loss for RetinaNet to $\alpha=0.75$ and $\gamma=1.0$, against recommended values of $\alpha=0.25$ and $\gamma=1.0$. The model was trained with a batch-size of $64$ and a learning rate of $0.001$.

To better illustrate the performance of each model, we exemplify the bounding box outputs of the different models on specific parts of an example plot shown in Fig. 2. We make the following observations, model-wise.

• •

  SSD glaringly misses detecting one of the bars, and also has low localisation accuracy as evidenced in the misaligned bounding for the second bar in Fig. 2a. However, it correctly detects small tick labels, perhaps due to proposal generation performed at multiple resolutions.

• •

  YOLO-v3 detects all objects (including both bars), but with lower localisation accuracy. For instance in Fig. 2b, the upper bar and plot title have misaligned bounding boxes. To see if this problem could have been solved by imposing priors on aspect ratios of bounding boxes, we plotted aspect ratios of all objects across plots and found no distinct clusters, i.e., aspect ratios of bars, texts, etc. vary in a continuum. This makes it hard to choose appropriate priors for bounding boxes.

• •

  RetinaNet which is based on SSD also misses out on detecting one of the bars and also the y-tick label (Fig. 2c). The bounding box of the detected bar is more accurate than that in SSD, indicating the benefit of the lateral connections in generating features for the regression head. Across the three one-stage detectors, which have much higher speed than the two-stage detectors, RetinaNet is the clear winner (row (c) in Table 1). While not apparent in the illustrated example, RetinaNet’s focal loss with custom tuned parameters $(\alpha,\beta)$ instead of hard suppression, may also be contributing to its higher performance.

• •

  Fast R-CNN (FRCNN) breaks up one of the bars into smaller objects (aligning with lines on the background grid of the plot). It also misses several objects including a legend item, the plot title, and a tick label (Fig. 2d). This could be attributed to the proposal generation method which uses selective search (SS). This under-performance is also visible at low IOUs: [email protected] is lowest for Fast R-CNN (row (d) in Table 1) potentially due to poorly performing SS which remains unaffected by IOU.

• •

  Faster R-CNN (FrRCNN) improves over the recall of SS by detecting most objects due to more complex region proposal network (RPN). However, RPN creates multiple overlapping proposals, even after non-maximal suppression (NMS) (Fig. 2e). This lowers the bulky model’s mAP to just $4\%$, which is second-lowest.

• •

  Mask R-CNN uses Faster R-CNN as the backbone architecture but uses ROIAlign instead of ROIPool gives mixed results when compared to Faster R-CNN. It is able to detect the longer textual elements (e.g., title) but has poorer localisation accuracy on the bars. It also breaks up one of the bars into smaller objects. However, its localisation accuracy on the tick labels is better than Faster R-CNN.

In summary, most state-of-the-art models have low robustness to high IOU values on this different class of images.

We do the same qualitative analysis as before for the three hybrid models and make the following observations:

• •

  Fast R-CNN with FPN and ROIAlign (FRCNN (FPN+RA)) improves on Fast R-CNN by not breaking up the bar into smaller objects, due to the FPN which enables improved feature extraction for the regression head. However due to the use of selective search (SS), many objects continue to go undetected (Fig. 2g). Notably, for the detected objects, the localisation accuracy is high, perhaps due to the use of ROIAlign’s bilinear interpolation when mapping proposals into smaller cells.

• •

  Faster R-CNN with ROIAlign (FrRCNN (RA)) improves on Faster R-CNN due to the substitution of ROIPool with ROIAlign. This leads to more accurate localisation and thereby removal of multiple proposals by NMS (Fig. 2h). Interestingly, this model under-performs FRCNN (FPN+RA) as evident in the mAP values and in the example by the difference in localisation accuracy. This illustrates the importance of FPN in being able to nullify the limitations of SS and improve localisation.

• •

  Faster R-CNN with FPN and ROIAlign (FrRCNN (FPN+RA)) performs the best amongst all existing models. It combines the RPN of Faster R-CNN for better region proposals, with FPN which provide better features, and ROIAlign which provides better mapping of features to scaled cells (Fig. 2i). There is an additive effect of combining these three important ideas in object detection, as evidenced by the significant difference between this model and the rest.

In summary, the model with the highest performance is the one that combines the best ideas in object detection.

Given the low mAP scores of existing models at the requisite IOU of 0.9, we propose a new network (shown in Fig. 3) which is designed bottom up based on three key observations. First, we observe that networks which use existing region proposal methods such as selective search, RPN, and anchor based methods have low [email protected]. In particular, these methods either generate too many proposals or miss out some objects. We contrast this with the apparent low visual complexity of these plots which suggests that detecting region proposals should be easier. Based on this insight, we propose a region proposal method which relies on traditional CV-based methods such as edge detection followed by contour detection. We however retain ROIAlign and FPN components which improved the performance of models. However, we note that FPN adds a significant computational cost and its addition needs to be carefully evaluated. While doing so, we make our second observation that longer textual elements such as titles and legend labels get detected as multiple proposals which need to be linked. We propose a separate linking component which decides whether a given proposal needs to be merged with any of its neighbours. None of the existing models perform such linking. Third, we notice a sharp decline of mAP scores on increase in the IOU. To address this, we design a custom loss function, which has non-negligible loss values for high IOU ( $>0.8$).

In summary, in the design of a custom model which we refer to as PlotNet, we (i) use a computationally efficient CV-based region proposal method, (ii) supplement it with a link prediction method to detect contiguous text objects, (iii) use ROIAlign and neighbouring information to better map proposals into smaller cells, (iv) evaluate the necessity of FPN, and (v) evaluate the need for IOU-based loss functions. We now describe the different components of our model.

Most state-of-the-art object detection models use $\ell_{n}$ loss (e.g., $\ell_{2}$-loss, smooth $\ell_{1}$-loss) for performing bounding box regression. However, several studies (Yu et al. 2016; Berman, Triki, and Blaschko 2018) suggest that there are some disadvantages of doing so and instead an IOU-based loss function which better correlates with the final evaluation metric should be used. Indeed, some studies have showed that using $-\log IOU$ (Yu et al. 2016) and $1-IOU$ (Berman, Triki, and Blaschko 2018) as loss functions give better results by ensuring that (i) the training objective is aligned with the evaluation metric, and (ii) all the 4 coordinates of the bounding box are considered jointly. However, these loss functions fail to learn anything in case of non-overlapping boxes. To overcome this, more generalized loss functions like GIOU (Rezatofighi et al. 2019), DIOU, and CIOU (Zheng et al. 2020) have been proposed which add an additional penalty term to deal with the non-overlapping boxes.

In existing loss functions, the penalty is negligible for boxes which have a large IOU overlap with the ground-truth box. To enable the network to learn tighter bounding boxes, motivated by focal loss (Lin et al. 2017b), we propose a Focal IOU (FIOU) loss that more gradually decreases the penalty as IOU overlap with ground-truth box increases. Formally, FIOU is defined as:

FIOU focuses on higher IOU values by dynamically scaling the $\log(IOU)$ loss (Yu et al. 2016), where the scaling factor increases as the IOU increases. The higher the hyperparameter $\gamma$, the larger is the penalty at medium IOU levels. We experimented with a couple of $\gamma$ values and have found $\gamma=2$ to work best for the PlotQA dataset. We include a plot comparing these loss functions graphically in the supplementary section. We then define a custom loss function combining smooth $\ell_{1}$-loss (SL1) with FIOU: $\mathcal{L}_{Custom}=\mathcal{L}_{SL1}+\mathcal{L}_{FIOU}$. This custom loss function achieves state-of-the-art results as we report in the comparison across loss functions in Table 2.

Similarly, for every proposed ROI, the coordinates of the parent ground-truth box identified above are assigned as the regression targets. In particular, for visual ROIs, the regression target is set to the co-ordinates of the parent box. For textual objects, it is difficult to regress the ROI to match the entire span of the parent box. For example, in Fig. 4(a), for the ROI containing the word “Number” in the title, the ground-truth box would be the entire title spanning all the words (cyan box). To avoid this large difference from the proposal, we create the regression targets for “Number” by clipping the ground-truth box to have the same boundary as the proposed box along the horizontal direction. The task then is to grow the proposed ROI vertically and then later link it to its neighbour thereby creating the entire title box.

Lastly, for creating the ground-truth for linking, we assign a binary value to each ROI for each of the 4 directions. These 4 values indicate whether the ROI needs to be linked to its left, right, top, or bottom neighbours. In order to find the neighbours, we consider an area of $50\times 50$ around a ROI and check if any of the neighbouring ROIs have the same parent box. If so, we assign 1 to the link corresponding to the direction (top, right, bottom, left) of that neighbour.

We trained our model for $10$ epochs using Adam optimizer (Kingma and Ba 2014) with a learning rate of $0.0001$. We experimented with different loss functions for bounding box regression, and used cross-entropy loss for classification as well as link prediction.

We now discuss the performance of different variants of our model as reported in Table 2. Note that all the variants in Table 2 use FPN as we always get better results with FPN. The variants mentioned in rows (a) and (b) use smooth $\ell_{1}$-loss (SL1) as the regression loss and do not have the Linking Head (LH) and the AN-ROI layer, respectively. The variants mentioned in rows (c) to (j) comprise the LH and the AN-ROI layer but only differ with respect to the regression loss. We make the following observations:

Existing object detection networks do not work for scientific plots - they have very low accuracy at the high IOU values required for reasoning over plots. Our proposed PlotNet makes a series of contributions across region proposal, model design, feature extraction, and loss function. These contributions together give a significant improvement of 16.22 points at [email protected] IOU. PlotNet is also much faster with a 16x speedup in comparison with existing networks, including one-stage detectors. On the extrinsic challenging task of plot-to-table, PlotNet provides an improvement of 20 points in the F1 score. These significant results enable further exploration of automated reasoning over plots.

We are extremely thankful to Google for funding this work. Such extensive experimentation would not have been possible without their invaluable support.

The supplementary material is organised as follows:
We first briefly introduce the different approaches for object detection and discuss the key ideas of the existing CNN-based object detection models. We then explain the motivation behind our proposed CV-based region proposal method and the Focal IOU (FIOU) loss function used in PlotNet (our proposed model). We also discuss the results and observations of hyper-parameter tuning experiments. Lastly, we explain the extrinsic task of plot-to-table conversion and exemplify the idea with the help of a few examples.

The goal of object detection is two-fold: (i) identify instances of a predefined class, (ii) identify its location with a bounding box. Over the last few years, the traditional complex ensemble object detection systems have been replaced by convolutional neural networks. There are two different approaches for doing object detection: (i) we can either make a fixed number of predictions on the feature map or the image itself by dividing it into $S\times S$ grid and then predict the class of the object present and the regressed co-ordinates of the bounding box; or (ii) leverage a separate proposal network to find relevant objects and then use a second network to fine-tune the proposals and output a final prediction. Based on the above criteria, we have one-stage and two-stage detectors.

In the following paragraphs, we summarise various CNN-based models along with their key insights.

In this section, we elaborate the motivation behind our CV-based region proposal method. On an average, there are $\sim 30$ objects present in a scientific plot and using any anchor-based region proposal techniques which proposes $\sim 100K$ boxes, is clearly an overkill. In the plot shown in Fig. 6a, we can see that in the upper strip of the image, to capture the plot-title, we need a long rectangular box. Therefore, it is unnecessary to try and fit anchors of all possible sizes at every feature point of this image. Given the simplicity of the objects found in scientific plots, we leverage the traditional CV-based methods to generate an initial set of proposals. To this end, we tried different approaches: i) Color-based segmentation method: It draws a bounding box for the objects having the same color. This approach fails to work when the neighbouring objects have colors of a similar shade. ii) Connected components method: It draws a bounding box for objects that share a common edge. This approach fails to work in the case of adjacent bars and results in a single merged proposal. For e.g., in Fig. 6a, a traditional connected component algorithm will merge the bars denoting Bulgaria and Cuba into a single proposal, resulting in a loss of information. Therefore, we use a sequence of edge-detection algorithms to generate our final proposals.

We first use a Laplacian edge detector on the image of the plot, which uses Laplacian kernels to approximate a second derivative measurement on the image. This helps us to detect the edges in the image by highlighting regions showing a rapid change in intensity.

Once the edges are identified, we extract contours by joining all the consecutive points along the boundary of the edges, based on uniform colour and intensity. Lastly, we convert these contours into a bounding box by finding the minimal area up-right bounding rectangle for each of the identified contours. These boxes serve as regions of interest (ROI) which are passed as input to the network. Our proposed method is $\sim 150$ times faster than traditional selective search simply because applying Laplacian kernels requires very few arithmetic operations. The output of each stage is shown in Fig. 6. It can be observed that our region proposal method has a high recall as it detects all the objects present in the image.

In this section, we compare our proposed Focal IOU loss with other IOU-based loss functions. As shown in Fig. 7, existing IOU-based loss functions have negligible values if the IOU overlap between the predicted box and the ground-truth box is reasonably high (say, $>0.8$). As discussed in the main paper, while this is acceptable for objects in natural images, it is not suitable for objects in scientific plots that require stricter localization. To this end, we propose a custom loss function that gives non-negligible values at high IOUs. The idea is similar to Focal Loss (Lin et al. 2017b) that scales the contribution of each sample to the loss based on the classification error, i.e., higher the classification error, lesser is the contribution of that sample to the loss function. Focal IOU (FIOU) loss function follows a similar principle. In particular, it increases the weight of samples which are already well localized so that their contribution to the loss is non-negligible. FIOU thus focuses on higher IOU values by dynamically scaling the $-\log(IOU)$ which pushes the network towards learning very tight bounding boxes.

From Table 4, we can observe that by using R-$22$ with FPN, i.e., ResNet-22 with FPN as the feature extractor gives the highest mAP throughout at $0.9$, $0.75$ and $0.5$ IOUs (row (d)). Hence we choose the same in all of our further experiments. We observe that even a very shallow backbone network of $10$ layers (R-$10$) achieves the second-best performance (row (a)) which supports the fact that bulky networks are not necessary for detecting simple objects like the ones present in scientific plots. The performance degrades (especially for small objects like dot-line and legend-preview) when we use too deep bulky networks such as R-$50$ with or without FPN (rows (c) and (e)) which suggests that features for tiny objects are lost. Hence, they are either misclassified or incorrectly localized.

From Table 5, we see that $\gamma=2.00$ gives the highest mAP throughout at $0.9$, $0.75$ and $0.5$ IOUs (row (f)), and hence we choose the same in all of our further experiments. We also observe that as $\gamma$ increases from $0.85$ to $4.00$, AP for the textual elements such as legend-label, x, y ticks and labels, increases, and then drops while achieving its peak value at $\gamma=2.00$.

As discussed in the main paper, detecting the objects in a scientific plot can be used to extract the underlying data that the plot visually represents. To this extent, we take a step forward and describe the process of table generation from plots using the example given in Fig.8. We have used the rules introduced in PlotQA (Methani et al. 2020) for the same. Specifically, referring to the detected elements in the Fig. 8(b) and the generated table shown in Fig. 8(d), we see that tick-labels (2002, 2003, 2004, and 2005) on the $x$-axis correspond to the rows of the table whereas the different labels (Cuba and Bulgaria) listed in the legend correspond to the columns of the table. The $i$, $j$-th cell of the table denotes the value corresponding to the $i$-th x-tick and the $j$-th legend-label. The values of all the textual elements such as plot-title, legend-labels, and the $x,y$-tick labels can be obtained from a pre-trained Optical Character Recognition (OCR) engine¹¹1https://github.com/tesseract-ocr/tesseract. Keeping in mind the spatial structure of scientific plots, the mapping of legend-label to legend-preview, $x,y$ ticks to corresponding $x,y$ labels and bounding boxes of bars to their corresponding $x$-ticks and legend-labels, is carried out. The height of each bar is extracted by using the top-left and bottom-right bounding box coordinates and the immediate $y$-tick label above or below that height. Finally, the value of each bar is interpolated based on the previously extracted bounding ticks.

We can follow the above method to extract the underlying data from a dot-line plot given in Fig.9(a) into a similar looking table, as shown in Fig.9(d).

On comparing the ground-truth tables shown in Fig.8(c) and 9(c) with the corresponding generated tables shown in Fig.8(d) and 9(d), we can see both the generated tables are very close to their respective ground-truth tables i.e., almost all the numeric values in the generated tables lie within $2\%$ of the corresponding values in the ground-truth table. This suggests that PlotNet can detect all the objects that are present in scientific plots with very high localization accuracy.