Gathering GitHub OSS Requirements from Q&A Community: an Empirical Study
^††thanks: *Corresponding author: [email protected] ^††thanks: This work is sponsored by the National Key Research and Development Program of China under Grant 2018YFB1004202.

Data collections comprise of two parts: collecting GitHub data and collecting SO data (see Fig. 2). We use GitHub API v3 to retrieve issue data. The quantity of search results limited to 1000 so we use stratified sampling to select samples randomly every six months and obtain 3266 issues as raw data in total for manually annotation. Preprocessing on raw dataset are necessary in order to support the follow-up study activities, e.g., filtering out non-English issues, non-requirements issues and the issues in which SO links are contained in comment part. We obtain the required issue data that describe the information of the cross-community requirement gathering from the GitHub repository and explicitly contain SO links describing the associations between the issue in GitHub and the Q&A in Stack Overflow.

A lot of semantics information of issue data that are required for analysis are missing or unclearly described in raw datasets. For example, whether the issue describes a functional requirement or non-functional requirement. We need to read and understand the issue data, get the expected semantics information, annotate and add these information manually in issues datasets. If the issue is used to feedback repositories’ bugs, defects, enhancements, new feature suggestions, and other software development related issues, we will determine this issue is used to gather requirements and record other attributes of requirements. For some issues with brief content and few text descriptions, we can further judge by visiting the issue URL and referring to the comments and context of the issue. We can also click on the SO links citing in the issue, jump to SO website to browser the corresponding post contents. Finally, we obtain 1034 software requirements with annotation for subsequent data analysis.

In order to analyze the requirement associations and development collaborations across communities, we need to correlate the issue data from GitHub with the Q&A dataset in SO. We extract the complete SO links from the issue data, obtain the post_id information from the links as input to query the SOTorrent dataset in order to acquire the post’s information such as question, answers, comments, post creation time, etc. associated with the issue.

We analyze the above data in GitHub and SO and get some meaningful insights about the designed three research questions. The details show in next section.

In general, labeling issues is more conducive to issues’ understanding and management. Issue tracking system in GitHub can define a label list to classify issues, facilitate issue management, and favor the resolution of issues [23]. By analyzing the issue’s label list, we find that the most frequently used labels are: enhancement, bug, feature. The enhancement label represents the enhancement of the original function, and the feature represents the addition of a new function. In addition to these labels, 36 issues are labeled with other labels such as help wanted, component and so on, 119 issues have no labels. We manually classify this part of the data according to enhancement, bug, and feature. Issues outside these three categories will not be discussed. The final distribution of requirements categories is shown in the Fig. 5 (a) below.

The previous research [11] shows that the most used tags in GitHub issues are bug (46.9%), enhancement (17.8%), and feature (5.9%), as is shown in Fig. 5 (b). But in our research, we find that among cross-community requirements, only about 32.67% of the requirements corresponds to bug, while 52.42% corresponds to enhancement, and 14.92% corresponds to feature. Thus, we find that more and more enhancement requirements have been gathered from SO. We consider that due to the increasing discussion in SO, it has gradually become a huge knowledge base derived from the Internet crowds. More people directly find some similar questions by searching when they encounter problems, instead of asking questions directly, so more and more posts are cited in GitHub for gathering enhancement requirements.

We classify requirements into Functional Requirement (FR) and Non-Functional Requirements (NFR), which are the basic classification of software requirements. FR define a function that a system or system element must be qualified to perform and describes the behavior of the system as it correlates to the system’s functionality. And NFR is restricted to a set of specific qualities other than functionality: such as usability, reliability, maintainability, extensibility, scalability and security. For example, a question: “What information must be stored in the database.” can be categorized as a FR. “The memory used by the software does not exceed 10Mb.” can be manually classified as an NFR.

The description about FR and NFR is mentioned in reference [24][25]. According to reference [24], we think that Feature belongs to FR, and we more agree with reference [25] to believe Bug belongs to the reliability of NFR. We only need to annotate the Enhancement to determine its type.

As shown in Table I, we found that most of the requirements discussed in SO are non-functional requirements (73%), and there are fewer functional requirements (27%). These non-functional requirements include maintainability, efficiency, availability, reliability, etc., we find most of NFRs gathered from SO are questions about system safety and performance testing. OSS developers in GitHub mainly focus on functional requirements, while non-functional requirements are difficult to gather. Through the Q&A community, it is easier to gather non-functional requirements from the crowds, which is of great help to improve the security, reliability, and stability of the software system. The developers can not clearly express these non-functional requirements, so they ask questions in SO. After full discussion, some developers move this question to the corresponding GitHub software repository. Finding 2: More than half software requirements gathered from SO is enhancement requirements, followed by bug and feature. We annotate requirements from the dimensions of FR and NFR, and find that most cross-community requirements are non-functional requirements about security and testing.

We categorize users in SO and GitHub, and investigate different type of participants’ contribution of gathering cross-community requirements. In SO, we classify users according to reputation values and privileges of user. In GitHub, we classify users according to the association between the issue author and the repository which the issue belongs to. The following are the details and results:

Users’ contribution of gathering cross-community requirements in SO: Reputation is a rough measurement of how much the community trusts you. High-reputation users contribute more to community development. In SO, the primary way to gain reputation is by posting good questions and useful answers. It represents the higher reputation users have, the more discussions users participate in, the more active users are, and the more experience and knowledge they gain. We classify SO users according to privileges and get the following four categories: new user, active user, established user and trusted user. The description of privileges and the corresponding user’s reputation value are shown in Table II.

We calculate the total number of various types of users in SO and the various types of users’ contributions of gathering cross-community requirements. As shown in the Fig. 5, trusted user in SO contribute the most to gathering cross-community requirements although the amount is the least. The contribution of established user and active user in gathering cross-community requirements is proportional to their respective numbers. Obviously, new users are not active whether in gathering cross-community requirements or participating in other discussions on SO. This shows that the new practice of gathering requirements across communities is mainly driven by active users.

Users’ contribution of gathering cross-community requirements in GitHub: The relationship between issue participants and repository is called author association in GitHub. There are five associations defined in GitHub: OWNER, MEMBER, CONTRIBUTOR, COLLABORATOR, NONE. Table III shows the detailed description of author association. According to the description of association, we classify both OWNER and MEMBER as OWNER, NONE as USER, and investigate the number of cross-community requirements gathered by different types of users in GitHub.

As shown in Fig. 7, most cross-communities requirements gathered by OWNER (55.5%), it shows that repository owners are more concerned about their own OSS repositories. Moreover, USER has no association with the repository, but there are more contributions (22.8%) of gathering cross-community requirements than collaborators (7.8%) and contributors (13.9%).

This shows that the new practice of gathering requirements across communities is mainly driven by users with higher permission in GitHub repository. However, the contributions made by other users who are not associated with the repository cannot be ignored.

Fig. 8 illustrates the development collaborations of gathering OSS requirements across GitHub and SO. The main cross-community behavior of users is about gathering requirements from SO to GitHub. We can dispose this behavior into two parts, one is that users in SO discuss requirements about GitHub OSS projects, and the other is that users create new issues in GitHub and link to SO posts. We investigate the user behavior of gathering requirements across SO and GitHub, and find two specific behaviors: Referring GitHub OSS repository’s name in SO posts, such as using repository’s name as a post’s tag; Citing multiple SO links in a GitHub issue of gathering cross-community requirements.

We define a new statistical measurement called Response time of gathering requirements, it represent the time period from the creation time of SO post of discussing requirements to the creation time of GitHub issue. To clearly illustrate our results, we used box plots in our research. In the box plot, five lines from top to bottom indicate the maximum value, the upper quartile (75%), the median of the sample, the low quartile (25%) and the minimum value. All data points above the top line or below the bottom line are outliers.

Referring GitHub OSS repository’s name in SO posts of discussing cross-community requirements: In Fig. 1 of section 1, the SO post’s author (dmz73) used the GitHub repository name (ModelMapper) as a tag of the question, which may help OSS developers (Dabd) to find this requirement faster. We explore how many SO posts cited in GitHub also refer GitHub repositories and whether such behavior shorten the response time of gathering cross-community requirements. We find that in all 939 issue-post associated data, there are 251 SO posts referring GitHub repository’s name, 52 of which refer in title, 109 in body, and 90 posts use repository’s name as tags.

We analyze response time of cross-community requirements with or without repository’s name to answer this question. Calculate the distribution of response time of gathering requirements referring the repository’s name.

As shown in Fig. 9 the distribution of response time of requirements referring GitHub repository’s name are more intense, while other requirements’ distributions are sparser, with an average value around 1120 days. Combining the data analysis in the box plot in Fig. 10, we observe that the median of response time of requirements referring the GitHub repository’s name is around 4 days, therefore about half requirements response time is less than 4 days. 75% requirements’ response time is less than 300 days. Except for a few outliers, this type of requirements response time is within 1000 days. The median of other requirements’ response time is around 1000, the first 25% and the last 25% of the median are distributed evenly. Only about 25% of other requirements’ response time is less than 300 days. In summary, about 75% requirements referring repository’s name are significantly shorter than other requirements.

Therefore, the requirements referring repository’s name has a shorter response time than other requirements. This shows that when discussing the OSS projects in the SO community, if you add a direct reference to GitHub repository, such as repository’s name, repository’s URL, etc., it will be more helpful for developers to understand and gather the cross-community requirements.

In addition, in Q&A community, there is generally a tag system that can classify the questions in more detail and get specific search results, which is convenient for users to quickly browse and locate a certain type of content. Of the 251 posts referring repository’s name, 90 of them use repository’s name as a tag, which shows that about 1/3 of users have agreed to this behavior and actively used in practice. Therefore, we believe that using the GitHub repository name as a tag will make it easier for developers to search by tag and quickly locate requirements.

Citing multiple SO links in a GitHub issue of gathering cross-community requirements: Another interesting behavior is issue’s author citing multiple Q&A, in order to enrich the understanding of software requirements and help developers comprehend in more detail about the cross-community requirements in GitHub issues. Fig. 11 shows an issue of gathering requirements with multiple links, GitHub developer cites three links in an issue, two of which are from SO. After browsing the issue content, we find that the first SO link is used to gather requirements, and the second SO link provides a possible solution. In order to better understand this behavior, we conduct the following investigation: How many links are cited in cross-community requirements? How many of them are SO links? What is the purpose of citing these SO links? The specific analysis is as follows:

We analyze the number of links and SO links cited in these issues containing cross-community software requirements. In our research dataset, there are 1873 links and 1334 SO links in 939 issues. As shown in Fig. 12, there are at most 23 links in the same issue, of which cited up to 18 SO links. We can infer from Fig. 12 that about half (537) of the cross-community requirements contain only one link which is SO link, and the other half contain multiple links, of which the distribution of the number of SO links is shown in Fig. 12. In addition, we find the behavior with more than 14 links in the same issue is less common.

Many developers list multiple requirements, some potential solutions, examples from SO when they create issues, which is also the main reason for including multiple links in an issue. This is one of the benefits of gathering requirements from Q&A community, which can easily provide many references. To understand the purpose and prevalence of quoting multiple links in an issue, we use regular expressions (solution, workaround, trial, approach, case, example, etc.) to filter the issue content and simply process the data. The results are shown in Fig. 13, there are 1.4% issues contains multiple requirements, 15.4% issues provide examples and 16.8% issues provide solutions. This is a good cross-community behavior, which can easily provide ideas and clues for OSS developers and help them solve problems quickly.