Analyzing the Effects of CI/CD on Open Source Repositories in GitHub and GitLab

GitHub is an open-source repository for developers to store, collaborate, and manage their code in a centralized location [17]. To collect data from GitHub, we primarily utilized the API provided by the platform. We adopted several different approaches while using the API. Initially, we collected information on repositories that implemented CI/CD and then further mined this subset of repositories to gather data on pull requests, issues, and related metadata. To achieve this, we utilized the GitHub API to search the path of each publicly available repository for the file path: “.github/workflow” [18]. Fig. 2 shows an overview of this data collection method.

This search was conducted to identify projects that contained CI/CD since this file is required to implement CI/CD within a repository. This method was effective until we reached the limit on the number of results. As per the documentation, only 1000 results can be returned for each search [19]. Thus, we needed another approach to collect the required CI/CD projects after reaching the limit of 1000 repositories.

To collect data from GitHub, we adopted another approach of scraping a list of developer name/repository name pairs using GitArchive. After compiling a substantial list, we utilized the GitLab API to check each repository for CI/CD. Next, we parsed each repository into a ”using CI/CD” or a ”not using CI/CD” bucket. To prevent personal or school projects from skewing the data, we pruned each bucket to include only repositories with at least two developers. We also ensured that each repository was active in 2022, thus ensuring the data is current and valuable for this research. This guarantees that the results produced by this research project are relevant and applicable to the current time. Fig. 3 shows an overview of this data collection method.

We used another search method using the GitHub API to collect the remaining data from GitHub. We formulated the search by filtering repositories by language (e.g., python, java) and sorting them based on the number of stars. This method proved highly effective, as we observed a strong correlation between the number of stars and the implementation of CI/CD pipelines in the project repositories. The more stars a repository had, the more likely it was to have implemented CI/CD. This search method enriched our dataset significantly, boosting our confidence in the quality of the data collected. Fig. 4 provides an overview of this data collection approach.

We conducted a deduplication process on all the collected repositories to ensure that each repository was collected only once and categorized correctly based on whether it implemented CI/CD. Then, we ran a validation function on each repository to ensure it was in the appropriate bucket. This step was important to ensure the accuracy and integrity of the data. After the deduplication process, it was confirmed that each repository was in the correct category and that there were no duplicates.

Issues With GitHub Data Collection. The primary challenges encountered during the GitHub data collection were related to the hourly rate limit imposed by the API when searching for repositories. This limitation was not ideal and prolonged the data collection process significantly. Another issue was that each API query could only return up to 1000 results, meaning multiple searches had to be conducted for each bucket, ”using CI/CD” and ”not using CI/CD.” This led to considerable time being spent validating and de-duplicating the data.

The process of collecting data from GitLab differed significantly from collecting data from GitHub. Initially, we attempted to utilize GitLab’s own resources for data collection [22]. However, we found that it took approximately 5 minutes to collect data from a single repository, which was deemed too time-consuming for the project’s timeline. Consequently, we refined our approach by using a python wrapper instead of the API natively [23]. Although the wrapper did not support global searching of a specific file, we devised a workaround. We used the python GitLab API wrapper to search for keywords found in the repository name across GitLab. Therefore, we searched for relevant data using keywords such as ’ci’, ’cd’, ’git’, and ’workflow’. Fig. 5 provides an overview of this data collection approach.

The method we used for data collection was not optimal in returning results. We had to spend a considerable amount of time to ensure that the collected data met the research’s standards. We invested a significant amount of time ensuring that the repositories had multiple developers, were categorized correctly based on their use of CI/CD and that only unique entries were collected. This time investment was critical to ensure that we collected high-quality data. Although we attempted other implementations, this approach proved to be the only reliable way to collect the required amount of data at scale within the research’s timeline.

Issues With GitLab Data Collection. The process of gathering data from GitLab is significantly more arduous, particularly in terms of searching for relevant information, which consumes a considerable amount of time. In comparison to collecting data from GitHub, obtaining GitLab data takes considerably longer. Despite the need to employ multiple methods to collect GitHub data, it is still roughly three times faster than gathering GitLab data. The disparity in speed is largely due to GitLab’s requirement to perform an API search for each issue and timestamp on commits individually, as opposed to GitHub, which can retrieve all necessary information with a single API call. Consequently, the API call-intensive process imposed by GitLab restricts the number of repositories that can be collected in a day and incurs significant waiting time as the API replenishes requests.

An additional problem encountered when searching for repositories on GitLab is that the API lacks the comprehensive functionality offered by GitHub’s API. Compared to GitHub, the GitLab API is less user-friendly and has fewer resources available to facilitate the search for repositories on the platform. In addition, unlike GitHub, the GitLab API does not support the search for file paths, highly-rated projects, language-based projects, or projects that use CI/CD. Consequently, a new approach was necessary to gather data on projects utilizing CI/CD on GitLab.

While searching for a new data collection method, we discovered that many researchers began by scraping repository URLs and then enriching the data using the GitPy tool to create a temporary local clone of the project. The logs required for the research were extracted from the local clone, and the project was then deleted [20]. Another method we found during our search involved crawling GitLab to collect the required data [21]. However, neither of these methods was ideal for collecting large amounts of data. Consequently, we decided to develop a new data collection method from scratch, as described above.

We collected data throughout this research, which comprises about 12,000 repositories. For GitHub data, there were at least 3000 repositories in each category; for GitLab data, roughly 1300 repositories were in each category. The collected data enabled us to analyze to answer the research questions. However, in this section, we will demonstrate the data to provide the reader with insight into the collected data. Table I represents the number of repositories with and without CI/CD.

We collected a few attributes associated with each repository. We cataloged each repository based on its file path in GitHub or GitLab. Additionally, we stored the number of active or retired issues within the lifespan of each repository. We collected the issues to analyze whether CI/CD creates an environment where fewer issues are created. Furthermore, we collected the average time between commits. We calculated the commit velocity of the repository by determining the mean between the time of each commit, and we calculated each commit to obtain an accurate measure. As shown in Table II, we designed the layout of our collected data. We further parsed the data above to enable calculations to take place. We subsequently analyzed the data using various statistical measures, and we describe the results in the next section.

After analyzing the data on a macro level, we took a much more micro approach to find the small details hiding within the data set. We used this micro approach to mine the repository data, giving us a better look at the distribution of the number of issues found in projects that use CI/CD and those that do not use CI/CD. We analyzed the entire data set and formulated the mean, standard deviation, and median of the number of issues within it as shown in Table III.

Throughout this study, we also collected and analyzed GitLab data. We demonstrate the results of the distribution between the number of issues for each in regard to CI/CD usage in Table IV.

Upon examining the data, it is evident that repositories without CI/CD processes have a lower number of issues. This conclusion is drawn from the lower mean of issues found within the repository, combined with a standard deviation that is significantly lower than that of the repositories containing CI/CD. Therefore, we can conclude that, on average, repositories without CI/CD have fewer issues compared to those with CI/CD in the development process.

After analyzing the number of issues in repositories from both GitHub and GitLab, we further investigated the commit velocity between the two platforms. It is worth noting that the mean and median values presented in Tables V and VI are calculated in hours. Upon examining the GitLab data, it becomes apparent that the commit velocity shows a similar trend to what we observed on GitHub. On average, the commit velocity is significantly lower when CI/CD is used in the pipeline, compared to when it is not used.

Upon analyzing the commit velocity data from both GitHub and GitLab, we observed that the projects using CI/CD have a faster commit velocity compared to those that do not use CI/CD. This trend is demonstrated in Tables VII and VIII, as expected.

The most significant finding regarding the use of CI/CD and commit velocity is the median time between commits. On average, when using CI/CD in their pipelines, the median commit velocity is significantly lower compared to those that do not use CI/CD. Based on this analysis, we can conclude that implementing CI/CD results in an increase in commit velocity.

The data clearly shows that implementing CI/CD increases the commit velocity on average, which is the primary goal of CI/CD. On average, the implementation of CI/CD results in a significant improvement in commit velocity of 141.19%, as shown in Table IX.

In delving deeper into RQ2, as previously discussed, the use of CI/CD has an impact on the number of issues found within the project repository. Our research and analysis revealed that when CI/CD is utilized, there are generally more issues created within the repository. Although CI/CD improves and accelerates commit velocity, it is interesting to note that the number of issues within the repository would increase. However, this is not surprising as the pipeline may naturally create errors as it executes. As the project is continually deployed, the likelihood of issues increases with more commits made. The swiftness of these commits may contribute to more issues found within the repository. On average, as shown in Table X implementing CI/CD results in a significant increase in the number of issues found within the code repository by 321.21%.

This research question evaluates the usefulness of GitLab compared to GitHub based on the findings of this study. Our research indicates that GitHub provides more benefits than GitLab regarding commit velocity and issues found within the repository. In addition, the analysis reveals that GitHub has a higher commit velocity compared to GitLab.

When considering this question, it is also noteworthy to mention the difference in data collection between GitLab and GitHub. GitLab provides fewer tools for analyzing data within the repository compared to GitHub, which offers a more extensive suite of tools to facilitate data collection and analysis.

Finally, it is worth noting that on GitHub, a higher number of stars on a project indicates a higher likelihood of the project implementing CI/CD pipelines within the repository. However, this trend was not as evident with GitLab. The number of projects using CI/CD versus those not using CI/CD did not show a significant difference based on the number of stars on the project.