Defect Prediction with Content-based Features

Software defects occur frequently in software development and often lead to costly and time-consuming activities to find and fix them. Gallaher and Kropp (2002) report that software defects cost the US economy nearly $60 billions a year. In addition, Hailpern and Santhanam (2002) found that finding and fixing them accounts for 50 - 75% of the total development cost in a software project. The later a defect is detected in the life cycle of a software product, the higher cost and effort are needed to fix it and re-deploy the fixed version of that product back to the field.

Many methods, techniques, and tools have been developed to support the early detection of software defects. One important line of research is defect prediction. Defect prediction approaches aim to identify the most defect-prone modules (binaries, source files, classes, or functions) in a given software system. Such prediction results can help software engineers to focus their manual defect detection effort like code review or testing on modules with higher likelihood of success, thus, improving the effectiveness and reducing the cost of their activities. Extensive literature reviews on existing defect prediction approaches can be found in Hall et al. (2012); Shihab (2012).

Most researches on defect prediction focus on the factors and metrics could be used to predict defects. Often called predictors or features, they are used as input of a prediction model which outputs the predicted number of undiscovered defects in a software module (regression models) or whether it is defective (classification models). These factors are general in nature and follow a common belief about software systems.

Traditional defect prediction approaches for source code often use metrics measuring the complexity of the design or the implementing code of a software system. For example, the most commonly used code metric is the number of lines of code (LOC). Other frequently used object-oriented design metrics are the depth of inheritance tree (DIT), the number of children (NOC), the lack of cohesion in methods (LCOM), or the coupling between objects (CBO) of a class.

In this paper, we explore a new approach based on two key assumptions. The first one is that a software system often implements several groups of functionality, each might have different levels of defect-proneness. For example, JEdit is a subject system studied in this paper. It is a word processor (document editor) with functionality for managing graphical user interface (GUI), presenting documents, processing edit commands from users, searching text, managing files (e.g. loading, saving, parsing), etc. Our study of JEdit suggests that while the code for GUI and editing commands is highly defect-prone (e.g. the most defective file is JEditTextArea.java with up to 45 post-release defects), the code for text search and parsing is much less defect-prone. Therefore, if we can infer the functionality implemented in a code module (e.g. a source file or a class) and the defect-proneness levels of such functionality from historical data, we can predict defects in that module.

We make the second assumption in our approach that functionality implemented in a code module could be inferred from its content, i.e. from identifiers, comments, annotations, string literals, keywords, embedded documentation, etc. For example, developers often name classes and methods using identifiers suggesting their functionality. For example, JEdit has some classes named JEditTextArea, OptionsDialog, and Buffer which clearly indicate the functionality they implement. In code comments, developers can also explain and discuss the functionality of their code such as the implemented algorithms or the roles of variables and parameters.

Based on those ideas, we explore four new types of features extracted from code content for defect prediction. The first one includes textual terms extracted from all text elements in code like comments or identifiers. We use standard tokenizing and stemming techniques to extract those terms and use the bag-of-word model to represent them. For example, the identifier OptionsDialog is tokenzied into two terms Options and Dialog which are further stemmed as opt and dialog, respectively.

Although textual content of source code could contain most information of the implemented functionality, the amount of extracted terms can be large and noisy. Therefore, we use topic modeling Blei et al. (2003) as a feature reduction technique for the extracted terms. This produces topics, the second type of features explored in this paper. Prior studies suggests that topics extracted from source code of a software system correspond to its technical concerns (e.g. functionality) and those topics can be used to predict defects Nguyen et al. (2011). For example, in JDT, a compiler framework, code written for semantic analysis is more likely to have errors than code written for lexical analysis.

To further address the noise in textual features like terms and topics, we investigate programming-semantic features including the data types of variables and objects, and the packages containing those types. Compared to text features, types provide a higher level of abstraction. In object-oriented design, a particular data type, e.g. a class, is defined to perform certain tasks, which contributes to the actual functionality of the code using that type. For example, objects of type java.io.File represent particular files or folders. They provide methods to obtain information about files and to manipulate those files in the system. Hence, if a source file uses java.io.File type, it likely works with the file system (e.g. create, delete, read, or write files) or performs file I/O functionality. In contrast, objects of type org.eclipse.jdt.core.dom.ASTParser provide functions to parse Java code in abstract syntax trees. Thus, if a source file contains objects of this type, it is likely to have parsing/compiling functionality. That means, we could infer functionality of a code unit based on the presence of some particular data types.

As a large software system might contain thousands data types, we consider package organization as a feature reduction technique for type features. In object-oriented programming, package organization can improve code modularity by grouping related modules (e.g. classes, source files) into packages. As related classes are grouped together, the resulting packages can provide higher levels of abstraction. For example, in Java API libraries, package java.io provides input/output functions while package javax.sql provides data access and processing functions. Like modeling infers the technical concerns of source code from textual features, package organization could infer technical concerns from data type information. However, while topic modeling is an unsupervised learning task, package organization was done by human, i.e. developers that design the target system, hence it would be more accurate.

While extracting text, topic, type and package features, we construct the feature vectors for a source file using the count of each term, the (log-transformed) counts of terms assigned to each topic, the presence of each type, and the presence of each package, respectively. Those vectors can further be combined into a unified one for all types of features.

As a software system can has high numbers of the four newly proposed features, we apply feature selection techniques to improve the prediction performance, in both accuracy and running time. For example, we only select terms and type features that have high ranked correlation or mutual information with defects. We also apply principal component analysis (PCA), a standard feature reduction technique to reduce the potential correlations among selected features.

Because this paper focuses on the features rather than the models for defect prediction, we only use the simple linear regression model (LR) for the prediction task. It should be noted that other prediction models like logistic regression, decision tree, or support vector machine, can also be used.

We have conducted an extensive empirical evaluation on a public defect dataset including 42 releases of 14 real-world software systems. This evaluation contains more than 2,000 experiment runs, exercising different options of the prediction model, such as the number of selected term features, the number of topics extracted, the number of selected type features, etc. We also compared our features with the best reported traditional code metrics.

The results show that all our proposed features are predictive of defect-proneness and provides better prediction results than traditional code metrics like number of lines of code (LOC) or Chidamber-Kemerer (CK) metrics. More importantly, feature selection and reduction techniques could further improve prediction accuracy. Finally, combining all four types of features is better than using them alone.

The key contributions of this paper include:

1. New types of defect predictors including terms, topics, types, and packages extracted from source code, and

2. An empirical evaluation to compare those predictors with traditional code metrics.

In Section 2, we introduce our features in details, including techniques for extracting them from source code and selecting the best ones. Section 3 describes our evaluation settings and Section 4 reports its results. Section 5 presents related work and conclusions appear last.

In this section, we describe in details the extraction process for term, topic, type, and package features. We also discuss several feature selection and reduction techniques for those features. While term features can be extracted using simple text analysis techniques, topic features are inferred using LDA, a widely used topic modeling technique by Blei et al. (2003). Extracting types and packages is more complicated, involving code parsing and partial program analysis (PPA).

We conducted five experiments to evaluate our proposed features and methods of feature selections. The first fours examined each feature type in turn, the final experiment was conducted to evaluate combined features. Each of the first three experiments was split into phases to discover different optimal settings for each feature type. Every experiments were performed on all 42 released of 15 projects. To evaluate baseline systems, in the final experiment, we used the CK and OO metrics provided for some projects in the ALR set while for all others we used only available CK metrics.

Defect prediction has attracted great research interest in software engineering. Researchers have searched for: 1) software metrics that are predictive of defect-proneness, and 2) prediction models that deliver accurate results.

Software metrics used in defect prediction studies mainly fall into three categories: product metrics (e.g. static code metrics), process metrics (e.g. code churn and previous bugs), and socio-technical metrics (e.g. developers).

One of the earliest code metrics used in software defect prediction is lines of code (LOC). Simple, easy to extract, frequently used, this metrics is still being discussed to this day. In a number of studies, LOC has been reported to correlate with the number of faults and performs quite well, although other studies show that LOC has only modest predictive power. In general, LOC appears to be useful in predicting software defects (a survey of LOC’s use can be found in Hall et al. (2012)). Beside LOC, popular complexity metrics such as McCabe’s cyclomatic complexity have been shown to be useful in predicting software defects. Khoshgoftaar and Allen (2003) used 16 code complexity metrics to predict defects in a large legacy system for telecommunications and achieved accuracy of almost 80% when using modules from one release as training data to predict fault in the consecutive release of the system. In addition to general code metrics, object-oriented design measures have been widely used in defect prediction studies. Basili et al. (1996) investigated the usefulness of of the Chidamber and Kemerer (CK) metrics in predicting bugs. They found five metrics correlating with the defect count: weighted methods per class, coupling between objects, the depth of inheritance, the number of children, and the response for a class. In a later study Nesi et al. (1999) showed that coupling between objects, lack of cohesion among methods, and response for a class were highly predictive of fault-proneness of a class. Overall, OO metrics have been reported to outperform general complexity metrics Hall et al. (2012).

Process metrics such as code churn, the number of changes, previous bugs, are extracted from the software development history. Graves et al. (2000) studied the change history and found that large and recent changes contributed the most to defects. Nagappan and Ball (2005) showed that system defect density could be predicted using a set of relative code churn measures that relate the amount of churns to other variables such as component size and the temporal extent of a churn. They later found that change bursts (i.e. frequently changed code) could be used has good predictors of bugs. Hassan (2009) proposed to use the entropies of changes as measures of code change complexity and found that such measures outperformed the absolute numbers of changes.

There are arguments in favor of both product and process metrics. While process metrics appear to deliver more accurate predictions than product metrics, the latter is easier to obtain because it is derived from the code itself and does not require any information from the development process. When both types of metrics are available, it is useful to combine them for better predictions. In addition to these two categories of metrics, socio-technical metrics that are computed from the information on organization structure, developers, and social networks have also been shown to be useful in predicting effects.

Some content-based features has been explored previously for defect prediction. Jiang et al. (2013) use all words and special operators as features in their classifier of buggy changes. However, as they did not use any feature selection/reduction techniques and nor provide any detailed analysis on the defect-proneness and predictive power of their features, our work is a complementary treatment and contains a deeper analysis of those textual features.

Chen et al. (2017) reports an empirical study using topics extracted from source code to explain defects. As their study focuses on explaining defect-proneness using topics, they did not perform defect prediction nor analyze the effect of feature selection on the performance of the prediction tasks. Thus, our work provides a deeper study of the relationship between topics and defects. Nguyen et al. (2011) only study topic features on one subject system and one configuration of topic modeling. In this paper, we explored more types of features and ran experiments on many more configurations and subject systems.

In this paper, we explore a content-based approach for defect prediction. Our approach extracts text, topic, type, and package features from the textual and semantic content of the source code and uses them as defect predictors. Empirical evaluation shows that i) our content-based features are predictive of defect-proneness and have higher predictive power than traditional code metrics. In addition, selecting, reducing, and combining those features provides better prediction performance than using them individually.

Future work. In this paper we design the features based on direct processing of textual and semantic content of source code. However this method of feature engineering might not cover all possible aspects of source code semantics. Recent advanced unsupervised text processing and pre-trained language models that produce distributed representation of terms and sentences was successfully applied to the natural languages. Thus, it is a natural next step to apply these techniques directly to source code. In addition, we could extend that concept to learn the distributed representations of more abstract code features such as data types, functions, classes or entire programs. These new representations could be the key to improve the performance of defect prediction systems.