Utilizing Source Code Syntax Patterns to Detect Bug Inducing Commits using Machine Learning Models

To evaluate the efficiency of any techniques in detecting Bug Inducing Commits (BICs) or Just in Time (JIT) defect prediction, most of the related studies Borg et al. (2019); Rosen et al. (2015) used datasets labeled using the SZZ algorithm Śliwerski et al. (2005b). Wen et al. (2019) reported that about 63.7% of bug-inducing commits (BIC) identified by the SZZ Algorithm are not real. There are several updated versions of the SZZ algorithm to deal with its incorrect labeling da Costa et al. (2017); Davies et al. (2014); Kim et al. (2006b). We do not use a dataset labeled using the SZZ algorithm in this study because of its imprecision. As our primary goal is to verify the effectiveness of features extracted from developers’ coding syntax patterns, we utilize labeled datasets published by recent similar studies as our baseline and added our TS and TP-based features to investigate the improvements in detecting bug inducing commits (BICs). Data sets selected in this investigation fall into the following categories:

Developers’ coding syntax pattern represents the behavior of a software developer, which affects the overall structure of a source code statement. We identify thousands of different commit patterns from source code patches of the eight subject systems of this investigation. About 18k of these patches whose token pattern length is within 100 characters are publicly available to view by the readers at our GitHub repository¹ for an easier understanding of those patterns. We also extract a sequence of these patterns for preparing feature values, which could be used in our ML-based classification model. As tokens of a programming language are the basic building blocks of each feature type extracted from the XML representation of source codes, we name the categories of features as Token Sequence (TS) and Token Pattern (TP). Commits encoded by these TS and TP represent the syntax style of coding by different developers.

We show an overall demonstration of extracting features (TS and TP) using a straightforward piece of program in Java language in Figure 2. Few additional examples of more complex program constructions, such as adding AND, OR in if-statement, nested if statement, and switch-case statement, are also available to view in four different other files ($./simpleDemonstrations$) at our GitHub repository¹. Our public repository also contains all the project files ($./projectFiles$), including the source file of commit patches and their equivalent XML representations for further investigation and reproduction of this study.

We first downloaded all the commit patches using the commit id in our labeled dataset and the git command for corresponding GitHub repositories. Then we retrieved the source code segments from those patch files. After that, we utilized a tool (srcML⁵⁵5https://www.srcml.org/), which converts source code into an XML format representing its coding structure in a normalized form. Suppose we have a source code of a commit-patch in a file named test.java, as shown in Figure 2. Processing the file test.java with the srcML tool will generate an XML file test.xml. The XML representation of the source file provides a hierarchy of the source code statement structure and token organization. This example shows that the file test.java contains two main blocks, one for $if$ and the corresponding $else$ for the $if$. In the XML representation of the source file, we can see those two blocks keeping the identifier hierarchy and sequence similarity. Processing the XML file, we can extract the normalized names of different tokens in the source code and the hierarchy of those tokens. Examples of the extracted token sequence (TS) and token pattern (TP) are shown in Figure 2. We have given each token pattern an id such as $pattern\_id\_1,~{}pattern\_id\_2,~{}$…….$~{}pattern\_id\_n$ where n is the total number of patterns extracted from the source file. Here, $pattern\_id\_1$ means the extracted pattern of normalized tokens, such as if_stmt-if-condition-expr-name is one of the patterns shown in Figure 2. There could be thousands of such patterns extracted from a software system that will reflect how the developer of that software system performs coding. To prepare the dataset for a commit, we will consider how many such patterns are in the source code of that commit patch. Similarly, in the same figure, we can also see an example of the extracted token sequence. We processed that token sequence with the n-gram Cavnar and Trenkle (1994) technique, which is mainly used in extracting features from natural language text. We utilized n-gram (with n=1 to n=5) in our extracted token sequence to produce Bag of Word (BOW) Jason Brownlee (2017); Le and Mikolov (2014); Zhao and Mao (2018), which is then used as another source of features, and we named these features as Token Sequence or TS. We repeated this process for the source code of every commit patch in all the subject systems to extract the features from TS and TP.

Applying ML-based models depends on the availability of labeled datasets (sample commits from each of the labeled classes) and extracting relevant feature values from the codebase history to train and test the model. After identifying the TS and TP patterns from the source code of the commit patch, it is important to encode them as feature values, which could be used in ML-based detection models. In this study, we used simple encoding of TS and TP by considering their number of occurrences in the source code of the commit patch. For example, in Figure 2, we demonstrated how the TS and TP patterns are extracted from the Java source code. Each code fragment may have several TS and TP. We extract all the TS and TP from all the commit patches of our labeled datasets. After that, we selected unique TS and TP to be used as feature identifiers. We will use the count of those feature identifiers in each commit to represent that commit while using ML-based detection models. A sample of TP is extracted for each commit, and their encoding mechanism is demonstrated in Table 3 and Table 4. In Table 4, we show the commit id, its extracted TP, and the associated label for each of the commits. Here, L=0 indicates a clean commit, and L=1 indicates a BIC. We then took unique TP from all the TPs to encode them and took the number of occurrences of each TP to represent each of the commits in Table 4.

Machine learning algorithms provide its result based on the feature values given for each data instance. The features we are considering for training and testing a machine learning model play a vital role in the performance of that model. A weak or irrelevant feature list can degrade the overall accuracy of classification. We wanted to evaluate whether the best features for a subject system are also better for the other subject systems. We applied Recursive Feature Elimination (RFE) of SciKit Learn Pedregosa et al. (2011) python library to rank all the features in the dataset of a subject system. RFE is a recursive approach to finding a list of the best features for a dataset with its classification labels. A user first needs to specify the number of best features expected to keep and the number of features to eliminate in each step. RFE then eliminates the weakest feature (or features) in each recursive step until the expected number of features is obtained. The parameter settings we have used for RFE are shown in Figure 3, where we used the Random Forest Classifier to estimate the importance of features and eliminate one feature in each step.

As we set $n\_features\_to\_select=1$, it will rank all the features from rank 1 to n (n=total number of features). Then return the top feature as the best feature for the dataset classification. We performed this for each feature combination, such as GS, TS, TP, GS+TS, etc. As GS has 12 features, this step will rank those features from rank-1 to rank-12 based on their importance in classifying the dataset. Similarly, GS+TS features have $12+10514=10526$ features, and they will get from rank-1 to rank-10526.

After obtaining the ranking of each of the features from the RFE algorithm, we applied the Random Forest classifier to select the set of best features for each of the subject systems. We first determine the Precision, Recall, and F1 Score of BIC detection using only the top (rank-1) feature in a subject system. Then we added one feature in each step from the following position in the rank list and determined the classification performance. We discarded the added feature if it did not improve the performance obtained in the previous step. Therefore, after completing the iteration from feature 1 to the total number of features, we will have the best features for that subject system. We repeated this approach for each of the feature combinations and each of the six manually labeled subject systems.

Our investigation utilizes five machine learning classification models to detect bug inducing commits from the two (OpenStack and QT) of the eight subject systems. We use OpenStack and QT datasets for applying multiple ML models as these datasets are most used in similar studies and contain a large number of data samples compared to the other manually labeled six subject systems. We use only the Random Forest (RF) classification model on the manually labeled datasets as they contain a smaller number of data instances. We find the features extracted from source code statements improve the prediction of defective software commits. All the classification models except the DBN are available in the Python SciKit Learn Pedregosa et al. (2011) library. We utilized the DBN classification model made available by albertbup (2017).

1. i.

   Random Forest (RF) Classifier implements ensemble learning methods where it builds multiple decision trees and merges them to get a better and more stable prediction. At the root of each tree, it first divides the data in such a way that the differences in each part of the data become as lower as possible, then it again divides the branches of the tree to get more specialization on the data. We can control the depth of such a tree by the parameter variable max_depth. The algorithm repeats this process with some different random selections and creates a set of random decision trees. During prediction, it predicts the case using each of the decision trees in the forest, then finalizes the decision with the majority decision of the trees.

2. ii.

   K-Nearest Neighbors (KNN) Classifier is one of the most commonly used ML algorithms known for its simplicity and effectiveness Taunk et al. (2019). It is a supervised learning method that uses a labeled training dataset to categorize data points for predicting the category of the test dataset.

3. iii.

   Gradient Boosting Classifier (GBC) is also an ensemble learning method, where the model learns by optimizing the loss function. It is used for both classification and regression. We use the default configuration of the Python SciKit Learn Pedregosa et al. (2011) library for utilizing this algorithm to detect bug inducing commits.

4. iv.

   Perceptron (PCT) is the simplest type of neural network model used for binary classification. It takes a row of input data to predict its output class. It utilizes the weighted sum of the input and a bias value. During the training process of PCT, it updates its weights to minimize the error of classification in training datasets.

5. v.

   Supervised Deep Belief Network (DBN) Classification is the utilization of an advanced deep learning algorithm Yang et al. (2015); Hinton and Salakhutdinov (2006); Hinton (2007); Hinton et al. (2006) for detecting more meaningful features to increase the classification performance of a labeled dataset. It contains two phases, i) feature selection, and ii) machine learning based classification. In the feature selection phase, it utilizes provided features to extract more meaningful feature values and then applies classification based on the extracted features.

We have applied machine learning models using time-sensitive detection techniques, which means training data must be from past timestamps than the testing data. Tan et al. (2015) show that cross-validation can provide false higher precision in predicting a data instance that is sensitive to timestamp, and a situation may happen then train the ML model with the data from the future and tests it using the data from the past. We first sorted the dataset in ascending order based on the commit date and took the first 70% of the data for training the model and the remaining 30% for testing. All the commit instances in the training dataset are from the past compared to the testing data instances.

As we extracted Token Pattern (TP) features utilizing the SrcML tool, which does not support Python for converting the source fragments into an XML representation. Therefore, we extracted only token sequence (TS) features from OpenStack as it is written using Python programming language. We extract both the TS and TP features from all the other seven subject systems listed in Tables 1 and 2. We investigated the ML-based classification model’s performance by taking different combinations of GS, TS, and TP features. In this investigation, our baseline feature is GS, and we are proposing TS and TP features to improve BIC detection performance. Suppose we would like to combine GS and TS features for the subject system Accumulo where the number of GS and TS features is 12 and 10,514. Thus, the number of combined total features (GS+TS) in Accumulo will be 10,526. We will rank those features based on their importance, where the most important feature will get rank-1, and the least important feature will get rank-10526. Similarly, if we would like to investigate GS+TS+TP for Accumulo, we will have 12+10,514+741= 11,267 features and rank them from 1 to 11,267th based on their importance in the dataset classification of Accumulo. As we have three basic feature types, we got seven feature combinations (GS, TS, TP, GS+TS, GS+TP, TS+TP, GS+TS+TP). We wanted to investigate whether more than one type of feature improved the classification performance by combining different feature types. Each of the ranked seven feature combinations is prioritized using the technique described in Section 2.4.

We first calculate some metric values to compare the performance of different feature combinations proposed in this study. Calculating these metric values depends on the counts of true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN). These counts indicate how well the machine learning (ML) model detects bug-inducing commits (BICs). For example, TP and TN are the counts of correctly identified BIC and non-BIC. Similarly, FP and FN indicate the incorrectly detected BIC and non-BIC using the ML model. Once we have these counts for all the feature combinations from the dataset of all the subject systems, we use the following formula to calculate Precision, Recall, and F1 Score.

When an ML model has a lower count of FP, it provides a higher value for precision, indicating the higher reliability of the model. Similarly, a model with lower FN provides a higher recall value indicating higher accuracy of the detected results. However, an attempt to increase the precision value typically decreases the precision and vice versa Developers.Google (2020 (accessed August 26, 2021). A BIC detection model that provides a higher precision with lower recall or higher recall with lower precision is not acceptable in a real-life practical software development industry. Therefore, we calculate the F1 Score by taking the harmonic mean of precision and recall to evaluate the results of this study. A higher F1 Score in a comparison scenario indicates a better result, considering the precision and recall values.

We also calculate the AUC (Area Under the ROC Curve) Developers.Google (2020 (accessed August 26, 2021) score, which provides an aggregate performance measure across all possible classification thresholds. The value of the AUC score is between 0 to 1, where 1.0 represents a model whose all detections (100%) are correct; one whose all of the detection are incorrect has an AUC of 0.0.

We use Scikit-learn Pedregosa et al. (2011) library available in Python programming language to calculate Precision, Recall, F1 Score, and AUC Score to determine the performance of detecting BIC and clean (non-BIC) commits. We extract basic features for all the subject systems from GS, TS, and TP. We have combined those features and got four extra (GS+TS, GS+TP, TS+TP, GS+TS+TP) feature lists. We obtain the results by prioritizing features (eliminating unimportant features using RFE) from all seven feature combinations. To compare the performance of BIC and non-BIC detection without eliminating unimportant features, we also reported the results using all the features from GitHub Statistics (GS-ALL, no feature eliminated) for all the subject systems. Based on the results of this investigation, we are answering our research questions as follows.

How can we determine whether developers’ coding syntax patterns could be responsible for inducing bugs in a software system?

This research question was the most important motivation for doing this study. We wanted to see whether the coding syntax style could induce buggy or faulty code fragments in the software system. The detailed extraction and encoding process of TS and TP features discussed in Sections 2.2, and 2.3 explain the different steps of identifying the coding syntax styles/ patterns and encoding the labeled commits with those identified TS and TP values. For instance, if a different developer writes the test.java file demonstrated in Figure 2, it could have a different coding structure, which will lead to obtaining a completely different list of TS and TP feature values. As the source code patterns encoded by TS and TP values improve BIC detection performance in all eight software projects compared to the conventional feature values (GS), we can say that these patterns might also be responsible for inducing bugs in software systems. Evaluation of the results of this study verifies this assumption as follows.

We show our results (Precision, Recall, and F1 Score) from the six manually labeled subject systems in Table 1 in Figures 5, 7, and 9 and two automatically labeled subject systems in Table 2 in Figure 4. These figures clearly show that we can detect BIC (and the non-BIC) in all software systems using only the TS or TP feature with higher performance measures (F1 scores) than the most commonly used GS-ALL and GS features. As ML-based detection models can distinguish BIC and non-BIC using the feature values representing the developer’s coding syntax style/ pattern, we can provide an affirmative answer to this research question. However, we can also argue that the ML-based model mainly utilizes those specific coding syntax styles/patterns to detect BIC from the software commits. Therefore, these coding syntax patterns must be dealt with very carefully while doing change operations in the codebase.

The results obtained using the six subject systems in Table 1 show that considering all the evaluation metrics (Precision, Recall, F1 Score, and AUC scores in all the subject systems), the performance obtained by only GS features is lower than the performance obtained by the use of TS and TP features. Besides, using all the 12 feature values (GS-ALL) extracted from GS has a lower performance than the prioritized features from GS, where we removed some unimportant features in all the subject systems. Comparing the BIC detection performance between GS-ALL and prioritized GS indicates that if we take only important features from GS-ALL, it will improve the performance, but this is not enough to detect all the BICs. On the other hand, using feature values extracted from TS and TP can improve the detection performance at a statistically significant level. This scenario provides the answer to this research question. In all the subject systems, either TS or TP or both features improved performance more than the GS-ALL (non-prioritized) and GS (prioritized) features.

The results of two automatically labeled subject systems in Table 2 also provide improved F1 Scores using all five different ML-based classification models in Figure 4. The horizontal line at 0.00 demonstrates the F1 Score using GS features only, and the points above and below the 0.00 line indicate the percentage (%) of improvements and decrease in F1 Scores, respectively. We show the gain (in %) in F1 Scores using all the different feature combinations of GS, TS, and TP. Our results show inclusion of TS and TP features improves the F1 Score in all five ML-based classification models. Some feature combinations in PCT and KNN provide a decrease in F1 Scores, but these are only six cases out of 40 total test cases in two subject systems.s

The distribution of the AUC scores shown in Figure 12 also supports our findings. Most of the AUC scores obtained using TS and TP features are higher than GS-ALL and GS features. We find the highest distribution of AUC scores using TS and GS+TS feature values. TP features also provide higher AUC scores but could not outperform GS and GS+TS. Although other feature combinations such as GS+TP, TS+TP, and GS+TS+TP also provide improved AUC scores compared to GS-ALL and GS features, it gives a similar distribution of AUC scores to other TS and TP feature combinations.

Do the features extracted from developers’ coding syntax patterns provide significantly better performance compared to the other feature values?

We performed the Wilcoxon Signed Rank Test to see if TS and TP-based feature values significantly improved the performance parameters (e.g., Precision, Recall, F1 Score). This statistical significance test will compare the results using TS and TP feature values with those obtained using feature values from GitHub Statistics (e.g., GS-ALL, GS ). For instance, we would like to examine whether the Precision/ Recall/ F1 Score obtained by GS+TP features is significantly better than the results obtained by GS features. Thus, we applied both GS+TP and GS features independently in our six subject systems and obtained Precision, Recall, and F1 Scores from each implementation, as shown in Table 7. Therefore, we got six pairs of observations for each of the Precision, Recall, and F1 Score, and we used them to calculate the differences between each observation pair. Then, we used the differences between these observation pairs to perform the Wilcoxon Signed Rank Test utilizing the SciPy library Virtanen et al. (2020) available in the Python programming language. Finally, we performed a significance test for each Precision, Recall, and F1 Score separately.

A summary of the results obtained from the significance test of the features is given in Figure 11, which shows the results of a feature list are significantly different from the results of corresponding features using a $|$ (vertical bar) symbols. To compare which result is better, we can refer to the distribution of the results shown in Figures 6, 8, 10, and 12. When we consider the precision of BIC detection, we can see that prioritized features from GS, TS, TP and any combinations of these features perform significantly better than using all the 12 features from GitHub Statistics (GS-ALL). TP features alone or a combination of TP and GS, providing significantly better results than GS-ALL and GS if we consider the recall results. The significance test of the F1 Score also shows any other feature combinations are significantly better than the GS-ALL features and GS+TP are better than GS. Therefore, we can conclude that features extracted from TP in all the scenarios provide significantly better results when combined with GS than the GS-All or GS features alone. Thus, our findings in this study provide enough evidence to believe that more source code-related features (TS, TP) can increase the detection accuracy significantly in identifying BIC using machine learning models.

How generalized are the extracted features from one software system to the others?

Do the features extracted from developers’ coding syntax patterns enhance the explainability of BIC detection from software systems?

Pornprasit et al. (2021) published a state-of-the-art tool named PyExplainer to explain the underlying condition for machine learning predictions. We use their tool to explain our BIC detection using the conventional (GS) and our proposed Token Pattern (TP) features. The results of the most frequent conditions from PyExplainer in the QT subject system are shown in Table 8. We show the top five most frequent feature conditions from GS and TP features that detected buggy commits using Random Forest (RF) algorithm in the table, and all the other conditions are publicly available to access in the GitHub repository of our investigation. In the table, we can see that it is very difficult to make proper reasoning for buggy commit detection considering GS features from the identified conditions by PyExplainer. For example, the top five most frequent conditions obtained from the result of PyExplainer are some range of values of some GS features such as Awareness, Age, Line Added, and Number of functions, but it does not show any specific reasoning that what a developer can do to avoid bug proneness in a software project for a certain given condition. A software development environment with a set of developers may have a fixed set of given conditions, such as developer awareness of the software project, age of the developer, number of lines to be added or changed, etc. These given conditions might be difficult to modify instantly or within a short period. Therefore, if we detect buggy software commits using those unchangeable conditions as feature values, it can provide enough meaningful or explainable reasoning to the software developer about how they can avoid the bug proneness of a particular change.

On the other hand, if we see the result from PyExplainer using Token Pattern (TP) features for the top five most frequent conditions for detecting buggy commits; we can see that most of the commits are detected buggy because of the presence of specific source code syntax patterns (i.e., number of declared variable names, expression with operators in IF-conditions, different level of nested expression and function call, etc. ). In any given condition of a software development environment, these conditions of a source code syntax can easily be updated to avoid or minimize bug proneness of a software project. These scenarios provide evidence to argue that, Token Patterns (TP) as feature values in the ML-based classification model enhance the explainability of BIC detection from software systems compared to the GS features. We plan to do more investigation about this scenario with some other subject systems in the future.

Subject systems in this study are written in Java, CPP, and Python programming languages. Therefore, we can argue that the results of this study may not be generalizable to the subject systems of other programming languages not used in this investigation. However, the subject systems used in this investigation are widely used in similar studies and are of diverse varieties. Thus we are confident about the outcome. Furthermore, since we wanted to compare our newly introduced features (TS, TP) to the most common features (GS) by exploiting the manually and automatically labeled data of eight of those systems, we used these widely used diverse varieties of Java, CPP, and Python systems. Regarding systems of other programming languages, we note the technique used may provide similar results for similar programming paradigms. Of course, further investigation is necessary, which remains our future work.

Wen et al. (2019) reported that automatically labeling buggy and non-buggy commits leave many incorrect labeling. On the other hand, getting enough data instances to apply ML classification models in manually labeled buggy and non-buggy software projects is difficult. For example, only 642 data instances of the buggy and non-buggy commits are available in six software projects from the manually labeled dataset Wen et al. (2019) used in this study. A small number of data instances or incorrectly labeled data instances both have a detrimental effect on the generalizability of results obtained by ML models. Therefore, we used the six manually, and two automatically labeled software projects of buggy and non-buggy commit to performing our investigation. We accomplish this investigation in two steps. First, we apply our technique in manually labeled datasets of the buggy and non-buggy commits by Wen et al. (2019) and show that our proposed features improve the performance of the ML model to detect buggy commits. We then apply five different ML classification models in two automatically labeled datasets having 4,014 data instances. In both experimental setups, we show that using our proposed feature values improves the performance of ML models to detect buggy commits. Our goal is not to contradict the results of any state-of-the-art JIT defect prediction method. Instead, we applied five different machine-learning models and eight different feature combinations. We showed in almost all the ML algorithms that using our proposed features improves the detection performance measures of bug-inducing commits. Therefore, we believe selecting any other dataset in this investigation should have similar findings. However, in the future, we want to extend this study using other software projects of different programming languages to substantiate the conclusion.

We also tried applying K-Nearest Neighbour (KNN) and Logistic Regression to detect BIC from the manually labeled datasets in Table 1. However, as these datasets contain a smaller number of data instances than a large number of features (different combinations of GS, TS, and TP), KNN and LR failed to accurately estimate the precision and recall of different implementations in detecting BIC. Therefore, we only used the Random Forest (RF) classifier to identify BIC from the manually labeled datasets and utilized four more ML classification models in the automatically labeled datasets in Table 2. As different ML algorithms work differently, it may be questionable whether our results obtained using the manually labeled datasets are generalizable for different ML algorithms or not. Hall et al. (2012) reported that ML models’ performance mainly depends on the nature of data, the quality of features, and proper parameter tuning. This study’s main goal was to compare the ML-based model performance by using the features GS, TS, TP, and their different combinations. We also applied five different classification models on the larger datasets shown in Table 2. The results in figure 4 show that in most cases, the use of software TS and TP (source code syntax pattern ) based features improve the BIC detection performance in all the ML models. If a different ML algorithm is used, it should equally affect all the feature combinations. Therefore, the comparison scenario obtained in this study should remain the same in different ML-based detection models. Although, more investigations using more massive datasets and other ML models are required to verify this scenario, which we will do in future studies.