A Better Approach to Track the Evolution of Static Code Warnings

Static bug detection tools have been widely applied in practice to detect potential defects in software. However, they are known to be underutilized due to various reasons. First, static bug detectors detect an overwhelming number of warnings, which may be far beyond what resources are allowed to resolve [1], [2]. Second, static bug detectors are known to detect many false positive warnings. The existence of a large number of false positives discourages developers from actively working on resolving the reported warnings [3], [4]. As a result, a significant portion of static code warnings remain unresolved by developers and can hinder software quality. Researchers have been working on techniques to improve the performance of static bug detectors [5, 6].

Recent studies show that by integrating static bug detectors in software development workflows, such as code review, developers demonstrated a higher response rate in resolving the reported static warnings [7, 8]. Tracking the evolution of static code warnings reveals which warnings are persistent, i.e., unresolved by developers, which warnings are resolved, and which warnings are newly introduced by the latest changes. Currently, there has been little effort to review the existing solutions to track the evolution of static code warnings. Avgustinov et al. [9] presented the first algorithm that combines various information of one warning. It compares two warnings in layers and eventually establishes mappings between two sets of warnings using three matching strategies(i.e., Location Matching, Snippet Matching and Hash Matching), which we refer to as the state-of-the-art (SOA) solution. The information of one warning they used is shown as Figure 1. However, they did not present an evaluation of the performance of the SOA solution in tracking the static code warnings. In other words, it remains unknown whether or not the SOA solution has an acceptable performance.

In this paper, we collected and uncovered the ground-truth evolution status between two consecutive commits using two widely used static bug detectors(PMD [10] and Spotbugs [11]) on two open-source software projects (JClouds and Kafka). We examined the SOA solution in tracking the evolution of static code warnings. Our investigation shows that the SOA solution achieves inadequate results, and we performed a manual analysis to uncover the insufficiencies of the SOA solution. Based on our findings, we proposed a better approach to track the static code warnings. The evaluation based on the crafted data set shows that our approach can significantly improve the tracking precision.

We propose an improved approach based on the manual analysis on JClouds and Kafka. To avoid a biased evaluation, in addition to the two systems, we also select two other open-source software systems (i.e., Spring-boot and Guava) to evaluate our improved approach and show how much improvement our approach has compared to the SOA approach. Totally, we collect 3,452 static warnings from SOA approach.

Since tracking the static code warnings is not a standalone task for each individual warning, it is, in fact, a mapping problem between two sets. Hence, we actually applied our improved approach on all static warning on a commit, which is a superset of the 3,452 warnings in the manually-labeled dataset. The remaining warnings while not in our crafted dataset have a pre-assumed satuts, “persistent”.

Table I lists the comparison results between the SOA approach and our improved approach. In the 3,452 warnings, the SOA approach has 1,178 warnings with a wrong evolution status. Compared to that, our proposed approach reduces the false positives significantly, from 1,178 to 253, i.e., the false positive rate drops to 10.0%. Our approach maps correctly for the labeled persistent warnings, which are mistakely labeled as resolved or newly-introduce by the SOA approach.

Tracking the evolution of static code warnings across software development history becomes a vital question due to the increasing interest to further utilize static bug detectors by integrating them in developers’ workflow. Also, such tracking is widely used in many downstream software engineering tasks. This study presents an investigation on the performance of the state-of-the-art approach in tracking static code warnings. In particular, a dataset of 3,452 static code warnings and their evolution status is crafted. Last, this paper present an improved approach, which is shown to outperform the SOA approach significantly in terms of the tracking precision.