SoK: Machine Learning for Continuous Integration

UT is the first step in the chain of events in the CI pipeline [16]. This test checks and validates the developed code in an isolated environment when a developer check-in (commits) new or modified compiled code and builds it [17]. To keep the master branch free from error, developers commit all the changes on a developers’ branch at this step [18]. Two out of the 44 selected studies presented ML-based solutions to reduce the risk of buggy software releases by predicting the outcome of the unit tests without explicitly running them. Lee et al. [19], proposed a method for predicting the outcomes of unit tests using ML, and demonstrate its effectiveness in estimating the state of the alarms raised by static checkers. Vig et al. [20] proposed a method for estimating the required test efforts by predicting the outcome of unit tests using ML. They compared different ML methods and feature sets to evaluate the effectiveness of their method.

IT is a CI step where the newly developed modules are integrated with the existing ones on the same branch after they have been validated through unit testing [21]. The purpose of IT is to verify that all the modules of a system or subsystem work together seamlessly. In this phase, the utilization of ML techniques can aid in forecasting the outcomes of test cases [21], and identifying the branch coverage of test cases to detect bugs with fewer test runs without missing any changes that need to be tested [22].

After validating and testing the integration of new or modified software units with other software units, the current version needs to be tested entirely based on the previously designed test cases [23]. The goal of all 25 RT studies can be classified into “Test Optimization” studies by employing Test Case Prioritization (TCP) and Test Case Selection (TCS) strategies, predicting the outcome of the test suites for committed codes as “Defect Prediction” studies, and “Detecting Flaky Tests”. Flaky tests are the tests that produce an inconsistent pass or fail results and make the CI pipeline unreliable [24]. In the field of TCP and TCS studies, the objective of ML models is to prioritize the test cases or select a subset of them that are more likely to uncover bugs and defects in the software under development [25]. This is achieved by analyzing previous changes in source code and the results of previous test runs to identify patterns that indicate a higher likelihood of uncovering defects.

According to the outcome of the previous CI phases, developers can be relatively sure about the functionalities and performance of developed software units. So, this version of the software is eligible for merging into the master branch. The BV test is to ensure the stability of integrated code before releasing the software and handing it over to a system testing team [26]. Due to the high computational cost of building software products [27], predicting the build outcome is the main target of the presented ML-based solutions in the BV phase. Build prediction aims to reduce the resource usage and building time of software by predicting the outcome of the build process. The predictions are made based on analyzing within-project data or cross-project data [28]. Nine out of 11 studies in this CI phase used batch data of the CI pipeline to train their models. However, the authors of [26] and [29] trained their ML models using a stream of data and changes in data, respectively.

In this step, all CI-related and quality aspects including the performance, functionalities, and compatibility of different software units are tested. This phase can be performed by a portion of the users or an internal system testing team [30]. For example, beta testers (users who receive the developed software before official releases) are asked to provide their experience using the the fully integrated software product [31]. ML-based solutions in this phase target the discovery of installed software for ensuring compliance, security, and efficiency of the system under test [32], and the optimization of test suites for detecting performance defects in developed software products [31].

According to the reviewed studies, a majority of the papers in the field of CI testing (24 out of 44), employed Open-Source (OS) projects. The popularity of the OS projects is due to the availability of different types of OS project data, including source code, test code, and their metadata, as well as access to the data of cloud-based tools such as Travis CI and GitHub. For example, Pan et al. [33], evaluated their model on 242 OS projects, and Parry et al. [24] assessed their model on 26 open-source Python projects.

Additionally, Table I shows different data preparation techniques used to ensure the quality of the input data for the ML models. The first step in data preparation for ML is to balance the data using various re-sampling algorithms. These include methods such as those found in Weka [21], oversampling techniques like SMOTE to increase the number of instances in the minority class [35], and under-sampling techniques like TOMEK [36]. Additionally, researchers can utilize a combination of both over- and under-sampling methods through the use of the SMOGN technique [37]. On the other hand, the currently available data sets in the context of CI need to be cleaned by removing unrelated data [19] or noises [32]. Yaraghi et al. [25], employed a novel method for data preparation and created a dependency graph in which nodes refer to a source file and each edge shows a dependency relation from the source node to the destination to determine the code coverage of test cases.

It is important to normalize the feature values in a dataset due to the diversity in their distributions and range. This can be done by using statistical methods such as re-scaling the values to fit within a specific scale, like [0,1] for normalization [22], standardizing by dividing the feature values by the standard deviation of all values [24], or applying techniques such as log transformation to bring the values to the same magnitude and reducing the effect of extremely high or low values [38]. Scaling is commonly employed (17 out of 44 studies) in the context of CI due to the diversity of feature values and the availability of numerical features. Additionally, for converting text-based values to input that is understandable for ML methods, such as system logs [32] and the source codes [19], tagging and tokenization techniques are often used (9 out of 44). Word2vec [19] and Columbus [32] methods are employed in the identified state-of-the-art studies for extracting tags from the texts. TF-IDF (term frequency-inverse document frequency) is a technique used to weigh the importance of words in a normalized way in texts [21]. It can help address the issue of imbalanced class distribution in text data by giving more weight to terms that appeared less frequently.

As shown in Table I, all of the current state-of-the-art studies utilize supervised learning methods. Additionally, it should be noted that all papers published between 2020 and the time of writing this paper have employed supervised machine learning algorithms. This trend suggests that supervised methods are becoming more popular among researchers in recent years. The popularity of supervised learning can be attributed to the acceptable accuracy of the models and the availability of labelled data from automated tools in CI environments.

Also, Table I shows that tree-based algorithms such as Decision Trees and Random Forests have been commonly employed as classification methods in state-of-the-art solutions. Tree-based algorithms have also been used in 19 studies from 44 selected studies. The reasons for the popularity of tree-based algorithms are threefold. First, a CI environment continuously produces a huge volume of data, and ML models need to be retrained frequently [25]. Since training tree-based algorithms require low computational resources, it is suitable to employ them in practical CI environments [39]. Second, the performance of the tree-based algorithms is generally high in classifying unseen data [26]. For instance, for automating the TCP task in CI, Yaraghi et al. [25] compared their tree-based (XGBoost) model with BERT [40], the state-of-the-art language representation model. They found that their method achieved relatively similar results as BERT while the training cost of their model was lower than BERT. Third, the tree-based algorithms can be well-interpreted, and thus they are understandable for humans [41].

According to Table I, besides tree-based algorithms, Neural Network (NN) algorithms, including Long Short Term Memory (LSTM), Multi-Layer Perceptron (MLP), Recursive Neural Network (RNN), and Deep Neural Network (DNN) have also been commonly employed in six studies. One of the main reasons for the popularity of NN algorithms, despite their need for high computational resources for training, is their ability to automatically extract features from data sets and make accurate predictions [31].

The K-fold method is widely used for evaluating ML models. It involves dividing the data into K equal segments (folds), and then in each of the K evaluation rounds, the model is trained on K-1 segments of the data and evaluated on the remaining unseen segment. According to the selected studies, 19 out of 44 have used this technique. In general, evaluation methods are classified into random and sorted sampling. Under random sampling, each row of data has the same possibility to be selected as training, testing, and holdout data set, while in sorted sampling the testing set must be newer than the training data set. In sorted K-fold methods, the unseen testing data folds must be selected from the earlier folds in comparison with the training data set.

F1-score, precision, and recall are commonly used evaluation measures in literature, with 16, 20, and 23 studies utilizing them respectively. They are the most frequently used metrics in the field. The description and formula of these measures are presented in Table II. Also, AUC (“Area under the ROC Curve”) has also been used, in which ROC is a graph on the plot of True Positive Rate (TPR or Recall) as y-axis and False Positive Rate ($\frac{FP}{FP+TN}$) as the x-axis.

The evaluation measure of efficiency (cost) has received significantly less attention compared to measures of effectiveness. Specifically, only Yaraghi et al.[25] reported cost-related evaluation metrics, which know as Cost-cognizant Average Percentage of Faults Detected ($APFD_{c}$). Malishevsky et al. [42] presented another version of $APFD_{c}$ that combines the cost of detecting and the severity of faults as presented in Eq. (1). In this metric $n$ is the number of test cases in test suite $T$ with costs $t_{i}$ and $f_{i}$ is the severity of faults. $TF_{i}$ is the first test case that reveals the fault $i$.