The GitHub Recent Bugs Dataset for Evaluating LLM-based Debugging Applications

A significant portion of software engineering research revolves around the automatic detection (Manès et al., 2021), reproduction (Kang et al., 2023; Soltani et al., 2020), and removal of software bugs (Gazzola et al., 2019). Prior work has shown differences between artificial bugs and real-world bugs (Gopinath et al., 2014; Just et al., 2014b), making real-world bugs valuable for research. As a result, software bug benchmarks using actual bugs from open-source software have been proposed, as such benchmarks allow standardized and fair evaluations of various techniques that deal with bugs. Examples of such benchmarks include the widely used Defects4J (Just et al., 2014a), Siemens (Do et al., 2005), BugsInPy (Widyasari et al., 2020) and BugsJS (Gyimesi et al., 2019) benchmarks.

However, the permissive licenses that led to these benchmarks of real-world bugs also make these repositories a prime target as training data for code-based large language models. Large Language Models (LLMs), which are Transformer (Vaswani et al., 2017) models with a large number of parameters, have been showing substantial performance gains relative to traditional techniques in multiple software engineering domains, such as in program repair (Jiang et al., 2023; Fan et al., 2023) or test generation (Lemieux et al., 2023; Kang et al., 2023). Having a large number of parameters, LLMs need a similarly large training dataset (Hoffmann et al., 2022), and thus a large amount of software data from open-source repositories is gathered (Li et al., 2023). Indeed, our own findings show that bugs from the Defects4J benchmark are often included in the training data of the open-source LLM StarCoder (Li et al., 2023). While information on the training data for the most popular LLMs in use, such as ChatGPT from OpenAI, is not public (OpenAI, 2023), it is reasonable to assume that they would use similar training data.

The potential overlap between existing bug benchmarks and LLM training data raises a critical question: are the state-of-the-art results from LLMs due to the strengths of LLM generalization, or simply due to code memorization? While such a concern had been voiced in early literature (Chen et al., 2021), we are unaware of subsequent attempts to build a real-world bug dataset that is not likely to be part of LLM training data.

To this end, we propose the GitHub Recent Bugs (GHRB) benchmark, which consists of 76 real-world Java bugs that have been fixed after September 2021, which is the cut-off date of training data for OpenAI LLMs (Ope, [n. d.]), such as GPT-3.5 and GPT-4. We further confirm that these bugs were not used for the training of the open-source StarCoder LLM. As a result, researchers can evaluate LLM-based applications on the GHRB benchmark without concern about data leakage.

In summary, our contributions are:

• •

  We provide a partial evaluation into how pervasive data leakage may be for the widely-used real-world bug benchmark Defects4J, and report that there is likely significant data leakage when using LLMs.

• •

  We provide an automated framework to gather real-world bug benchmarks, and filter them so that only the most relevant are left for manual inspection.

• •

  We make our bug benchmark, GHRB, publicly available¹¹1https://github.com/coinse/GHRB, which facilitates evaluation of LLM-based techniques that involve bugs without data leakage.

There has been a steady shift towards evaluating software engineering research artifacts using real-world bugs (Liu et al., 2019; Jiang et al., 2023; Koyuncu et al., 2019; Chen et al., 2018; Sohn and Yoo, 2017). These benchmarks allow comparisons of techniques on the same basis (Liu et al., 2020). Further, evaluations based on these benchmarks, while not perfect, are likely to reveal the real-world performance of a given technique better than those based on artificially seeded faults.

The use of the same benchmarks has continued while LLM-based SE techniques have been rapidly introduced. Given the opacity of training details of OpenAI LLMs, it is difficult for researchers to assess whether LLMs have been trained on benchmark data. Instead, under the assumption that LLM developers are likely to gather similar training data, we provide an initial assessment of the degree to which existing benchmark data is included in open-source LLM training data. To do so, we use the ‘Data Portraits’ tool²²2https://stack.dataportraits.org/, which allows users to easily check whether any text was included in the StarCoder training data by highlighting the longest common subsequence of the input data that was also present in the training data. Using this tool, we evaluate the widely-used Defects4J v1.0 benchmark. Specifically, we evaluate whether the bug-revealing test and the buggy method, two common artifacts used in LLM-based SE applications (Kang et al., 2023; Koyuncu et al., 2019; Jiang et al., 2023), are included in the training data; we consequently evaluate the 345 bugs with buggy methods. We conservatively assume that if 90% of a test or method is included in StarCoder data, data about that bug was likely included, making it inappropriate for evaluation when using StarCoder.

Worryingly, we find 35% of tests and 39% of buggy methods were included in StarCoder training data, using the 90% criterion mentioned above. In combination, 59% of Defects4J bugs were likely compromised. A breakdown of evaluation results by project is presented in Table 1. Furthermore, some matched subsequences would ‘break’ around the actual fix location, as in Figure 1, suggesting that StarCoder was trained with the fixed version of these methods. The new projects introduced in Defects4J v2.0 were not safe either: the most recent bugs from all newly added projects 2.0 were also included following the same 90% criterion. While it is difficult to know for sure, our finding strongly suggests that OpenAI may have included these projects in its LLM training data in a similar fashion, raising concerns about the validity of LLM-based SE evaluation using Defects4J.

Researchers have been aware of this issue (Kang et al., 2023; Jiang et al., 2023; Fan et al., 2023), with many developing new evaluation datasets. However, up to now these efforts have been ad-hoc and uncoordinated. We believe it would benefit LLM-based SE research if there existed a standard, real-world benchmark that was relatively free from data contamination concerns.

This section describes the process used to collect bugs for GitHub Recent Bugs (GHRB), that are recent enough to avoid being included in LLM training data. Specifically, every bug should meet the following requirements:

The scripts used to collect and verify our data are available in our artifact as well, facilitating expansion of GHRB.

The GHRB benchmark consists of 76 bugs from 16 repositories. The chosen repositories vary in size and popularity, but all are primarily written in Java. Table 2 shows summary statistics of the dataset.

Similarly to our check in Section 2, we checked whether the oldest bug-revealing tests gathered as part of GHRB were included in the StarCoder training data. We found no overlap over our 90% criterion³³3The maximum overlap observed was 66%: note that non-zero overlap is inevitable as these projects were created before September 2021. used in Section 2, demonstrating that the bugs that GHRB contains are free from data contamination concerns when evaluating StarCoder-based applications. Since GHRB is based on commits merged after September 2021, GHRB is also ‘safe’ when used to evaluate OpenAI LLM-based applications as well.

The GHRB dataset additionally provides the following:

In addition to the collection of bugs described in the previous section, the artifact of GHRB provides the following interfaces to facilitate ease of use.

This work expands the bug reproduction dataset introduced by Kang et al. (Kang et al., 2023). In that work, the authors constructed a real-world dataset consisting of 31 real-world Java bugs for which bug-reproducing tests were included after the OpenAI training data cutoff point to mitigate data leakage concerns. The GHRB of this publication significantly expands on that data, consists of 76 bugs from 16 repositories up from 31 bugs from 6 repositories in prior work, and provides a command-line interface that facilitates the use of GHRB.

This work is heavily inspired by prior bug benchmarks, as mentioned in Section 1, of which Defects4J is the primary example. As we show in Section 2 there is data leakage risk when using it to evaluate algorithms using state-of-the-art LLMs. In contrast to Defects4J, GHRB exclusively consists of data created after the data cutoff point of OpenAI’s LLMs(Ope, [n. d.]), allowing LLM-based algorithms to be evaluated without concern of data contamination. Other benchmarks exist in other languages as well, and we hope to expand our dataset to include other languages in future work.

We introduce GitHub Recent Bugs (GHRB), a real-world Java bug dataset designed to mitigate data leakage concerns when evaluating LLM-based software engineering techniques. Our hope is to present a supplementary evaluation benchmark to the larger and established bug benchmarks, so that software engineering researchers can evaluate the potential generality of their LLM-based tools more conveniently.