Team Resilience under Shock: An Empirical Analysis of GitHub Repositories during Early COVID-19 Pandemic

Using GitHub repositories as surrogates for teams is common practice in literature (Maldeniya et al. 2020; Ortu et al. 2017; Vasilescu et al. 2015a; Jarczyk et al. 2014). For example, (Maldeniya et al. 2020) studies how repositories as virtual teams respond to increased attention. (Vasilescu et al. 2015a) investigates how tenure and gender diversity of developers relate to team productivity and turnover. Technique details in team selection can be different due to the complexity of data (Kalliamvakou et al. 2014) and distinct research objectives. We limit the number of active members and contribution histories of repositories to filter teams where members work remotely and consistently in Section 4.1.

The performance of GitHub repositories can be measured in different aspects: productivity (Vasilescu et al. 2015a; Meyer et al. 2014; Vasilescu et al. 2015b; Mockus, Fielding, and Herbsleb 2002), team size (McDonald and Goggins 2013), code quality (Mockus, Fielding, and Herbsleb 2002), etc.. The productivity of a repository is also a combination of different activities. It could be measured by the quantity of the activities such as commits (Vasilescu et al. 2015a) or pull requests (Forsgren 2020), time spent on code (Meyer et al. 2014), accepted changes (Vasilescu et al. 2015b) (i.e., the number of merged pull requests), file changes (Casalnuovo et al. 2015), or the response time to problem reports submitted by users (Mockus, Fielding, and Herbsleb 2002). We select the number of pushes as a metric of productivity, following the official report of GitHub (Forsgren 2020), and the number of active members as an additional metric to characterize the growth of a repository during the pandemic.

A cluster of work has been done to explore the properties related to the performance of virtual teams (and GitHub repositories in particular), including the demographic diversity in gender (Vasilescu et al. 2015a), country (Ortu et al. 2017), languages spoken (Daniel, Agarwal, and Stewart 2013), experiences and expertise (Vasilescu et al. 2015a), social connections (Casalnuovo et al. 2015; Blincoe et al. 2016), the ability of multitasking (Vasilescu et al. 2016), and the emotional status (Graziotin, Wang, and Abrahamsson 2014) of developers. We propose a comprehensive set of features that characterizes many of these properties in Section 4.2 to understand team resilience under the shock.

Since the pandemic can be seen as an external shock out of the developers’ control, we adopt a causal inference framework to estimate the effect on each team. There are several causal inference frameworks developed to evaluate the impact of a shock, they don’t fit this problem very well. Difference-in-Differences models (Angrist and Pischke 2008; Ye et al. 2020) are commonly used to analyze “natural experiments,” yet the pandemic influences the entire GitHub community, leaving no control group to construct the counterfactuals. Regression Discontinuity in Time (Hausman and Rapson 2018) has been proposed to evaluate the impact of a policy that universally affect all subjects at a cutoff time. However, as with all Regression Discontinuity methods, the estimated treatment effect is usually focused on the narrow window close to the cutoff. The shock from the pandemic is much more complex, because both the infection rate and the workspace policies are rapidly changing over time, and we need to estimate the treatment effect over a longer time span. To this end, our approach is mostly close to the interrupted time series analysis (McDowall, McCleary, and Bartos 2019). Instead of fitting linear models for forecasting, we adopt advanced machine learning algorithms to predict the counterfactuals in Section 4.

We base our analysis on the event log data archived by GHArchive³³3http://www.gharchive.org/, retrieved on March 18, 2021. from January 2015 to December 2020. The dataset includes more than 20 types of events provided by GitHub, attributed to individual developers and public repositories.⁴⁴4https://developer.github.com/webhooks/event-payloads/, retrieved on October 9, 2021. Although the events correspond to the activities of individuals (code submission, issue tracking, coordination, etc.), a few event types should not be counted towards their contributions to a team, such as a watch event. A developer is not an active contributor to a repository if they simply “watch” it without performing other activities. Similarly, when a developer forks a repository, they are not contributing to the original repository unless they merge the code back in. Therefore, we exclude watch and fork and use all other event types to measure the activeness of repositories.⁵⁵5Star events also belong to the excluded category but they are not included in our dataset. In a given period of time, a repository is regarded as active if there is at least one logged event. A developer is regarded as an active member of a repository if they have at least one logged event explicitly attributed to this repository. In this way, we obtain a dataset containing 27.6 million users that have been active contributors to at least one repository and 160.5 million repositories that have been contributed by at least one member during the six years.

We first count the number of active repositories and the number of active developers per repository on a monthly basis. The latter provides a measurement of the effective size of the team in that month. Furthermore, we measure the productivity of a repository by counting two events directly related to code contribution (Forsgren 2020). Specifically, pushes are used to upload all local branch commits to the corresponding remote branch; pull requests are used to tell others about changes pushed to a branch with the aim of getting the changes merged to the base branch. In our dataset, there are 115 million opened pull requests and 1.5 billion pushes in total. We aggregate these statistics over active repositories every month to measure the monthly productivity of the GitHub platform.

Intuitively, the productivity of both individual teams and the platform as a whole may have been impacted when the pandemic hit. Since it is a worldwide pandemic, it is hard to define the exact date of the shock or to pin down the dates to individual developers or repositories even if one could link the identities of developers. Indeed, the behavior of a team may be already influenced even before the wave hits their own locations. Considering that COVID-19 first outbroke in late January of 2020, we regard the whole year of 2020 as possibly affected by the pandemic when estimating the shock effects. In particular, the first three months of 2020 overlap with the first outbreaks of COVID-19 and the declaration of the pandemic by the WHO (on March 11, 2020),⁶⁶6https://www.who.int/director-general/speeches/detail/who-director-general-s-opening-remarks-at-the-media-briefing-on-covid-19—11-march-2020, retrieved on January 15, 2021. which reflect the immediate shock effect of the pandemic. The months shortly after (i.e., April to June) are more reflective of the teams’ responses to remote work, as when the pandemic unfolds, working from home has been adopted as a common response in most countries. The months thereafter are more reflective of their longer-term responses to the pandemic, and by then remote work should have been widely adopted and COVID vaccines had not been deployed.

We anticipate that the influence of the pandemic on GitHub teams may vary depending on the stages. For each of the aggregated productivity metrics, we construct a time series with monthly measurements since January 2015 and use the observations before 2020 (60 months in total) to forecast the values in the next 12 months (i.e., January-December 2020) with an exponential smoothing and STL decomposition method (Robert, William, and Irma 1990).

Figure 1 visualizes the observed values of four metrics compared with their forecasted values (what if there were no pandemic). The differences can be interpreted as the overall effects of the pandemic on the GitHub repositories. As shown in Figure 1(a), the number of active repositories shows an increasing trend over years, and the predicted number of active repositories during 2020 follows the same trend. Interestingly, the observed numbers of active repositories after the pandemic started are significantly larger than expected since March 2020 when COVID-19 becomes a worldwide pandemic, which shows that the shock does have an effect on the activeness of the repositories. It is likely that many new teams start to use GitHub, or existing inactive teams have reactivated their GitHub repositories for remote collaborations. This finding is aligned with the trend that the pandemic has made teams shift to remote collaborations, and GitHub provides an efficient platform for this shift.

Although the number of active teams has increased because of the pandemic, the other measures show different patterns. For example, Figure 1(b) shows that although there is an increasing trend in the number of opened pull requests before the pandemic and it is projected to further increase in 2020, the actual observations are mostly below the expectation. This indicates that while more teams have joined (or rejoined) GitHub, the productivity of existing teams may have actually dropped. An alternative possibility is that the newly activated teams might be just smaller (and thus less productive). However, Figure 1(c) does not suggest a clear trend of the effect of the pandemic on the size of active teams, other than it becomes much more volatile during the pandemic. The number of active members of a repository actually increased during the first five months (although not significantly) and then dropped below the forecasts. The pandemic effects on the number of pushes per repository present similar patterns as team size, although in certain months the difference between observed and forecasted are significant.

The different patterns on pushes and pull requests are rather interesting. While both measure productivity, the former is more of individual activity, thus more correlated to the number of members in a team; the latter represents a collaborative activity (someone else in the team is expected to merge the updates into the base branch). The visual examination suggests heterogeneity in the shock effects over time, on different outcomes, and potentially on different types of teams. This motivates us to conduct a more rigorous analysis to examine the shock effect on individual repositories.

We define team members as those who make any type of contribution to the repository, following the survey findings in (Vasilescu et al. 2015a). As stated before, developers who are just watching or forking a repository are excluded.

We select two sets of repositories in this analysis. The reference set contains 71,928 repositories that were active in every quarter of 2018 with at least three active members. This excludes teams that were built temporarily for short-term projects (such as course teams). These repositories must have logged push events by 2018. Similarly, there are 87,914 repositories in the target set had at least three active members in every quarter of 2019 and push records by 2019. The criteria are set to identify consistently maintained repositories that are already familiar with remote work.

For repositories in the reference set and target set, we characterize their team properties in the 4th quarter of their selected year (i.e., 2018/2019 for the reference/target set) in the following aspects, respectively.⁷⁷7We use the GHTorrent Mysql dump dated 2021-03-06 retrieved from http://ghtorrent-downloads.ewi.tudelft.nl/mysql/ to extract the metadata of users (account creation time and country) and their followers. Other information is from the GHArchive data.

Meanwhile, considering that the activity and outcome of developers show a weekly pattern (Forsgren 2020), we also construct a 7-dimensional vector to describe the distribution of working days in the 7 days of a week. Entropy is used to characterize this distribution.

Additionally, we include the following variables to control for the confounds of the team outcome during the shock. We measure the number of pushes, pull requests, and issues in the 4th quarter of and the monthly pushes by the selected year. Literature has shown that external attention could influence team behaviors (Maldeniya et al. 2020), to control for this, we count the overall number of watches and forks by the selected year and the recent attentions in its 4th quarter.

With the properties of the reference repositories in the 4th quarter of 2018, we then train machine learning models to predict their outcomes for a given month in 2019. Specifically, we train one model for each of the six selected months (i.e., January-June, denoted as month $1$-$6$) to predict each of the two outcomes (i.e., productivity and growth).

We adopt a Gradient Boosting Decision Tree model (GBDT) and a Random Forest model (RF) for this prediction, implemented with scikit-learn (Pedregosa et al. 2011). After dropping repositories with missing features, we have 42,018 repositories in the reference set. We use an 80-20 train-test split for this set, and tune hyper-parameters on training with 5-fold cross-validation. To compare, we use a seasonal naïve model as the baseline, which predicts the outcome of a repository in month $i$ of 2019 with the outcome of the same repository in month $i$ of 2018.

We train a model for each of the month $i$ ($i\in\{1,2,..,6\}$) and present the performance of the models in Table 1. Results show that both the GBDT and RF models obtain a much higher $R^{2}$ and a much lower $MSE$ than the baseline seasonal naïve model across all six months. Similar observations could be made from the prediction models for team size. As GBDT performs the best in most settings, we adopt it in the rest of the paper. The $R^{2}$ scores are mostly $>$0.6 and even higher for a shorter time horizon, which is reasonable given the difficulty of the prediction task.

Note that a high $R^{2}$ (or a low $MSE$) does not ensure the robustness of downstream analyses, which may still be affected by the potential errors in the counterfactual predictions. These errors are not observable and thus cannot be directly measured. To address this issue, we generate a reasonable estimation of the unobserved errors of the counterfactual prediction in Section 4.5, which can be used to analyze the potential effect of these errors on downstream analyses.

With the 6 GBDT models trained (using Q4 of 2018 to predict months 1-6 of 2019), we are able to make counterfactual predictions of the outcomes of the target repositories in months 1-6 of 2020 (based on their features in Q4 of 2019). Note that repositories with missing features are dropped, which leaves us with 48,612 repositories in the target dataset. The effect of the pandemic in a given month (1-6) is then measured by the difference between the observed and the counterfactual outcomes.

We plot the distributions of the individual treatment effects in Figure 3, along the time horizons. The mean of the distribution in a particular month is essentially the average treatment effect of the pandemic in 1-6 months on the selected teams (teams that are already active before the pandemic). When the mean is below zero, it means the productivity or size of the teams has declined because of the pandemic. Comparing to Figure 1(d) and  1(c), the average treatment effects on these selected teams are much more stable over months. As a reference, we also show the residuals between the observed and predicted outcomes in 2019 (calculated based on the 20% test data). These residuals indicate the individual treatment effects of “no pandemic”, and therefore they should be centered at zero if the machine learning predictors are unbiased. As we can see from Figure 3, when there was no pandemic, there was indeed no obvious difference between the observed/predicted team outcomes. When the pandemic hits, there are noticeable negative effects on team productivity, especially in the first three months of 2020. Interestingly, these negative effects are not clear on team size in the same months (1-3), while from April (4), we start to observe a clearly declining pattern (in team size), especially in May (5) and June (6). For each month, we ran Two-sample Kolmogorov-Smirnov tests and the null hypothesis (“the pre- and post-pandemic distributions are identical”) is rejected for all settings in Figure 3.

It is intriguing to observe that team productivity reduced immediately following the shock (1-3) and then recovered to normal a few months after (4-6). The number of active team members, on the other hand, did not immediately drop when the pandemic hit (1-3), but it drops gradually when the teams respond in a longer term (5-6). A possible explanation is that the core members of a team kept contributing (and even took more responsibilities in a longer term), while the activeness of peripheral members are more likely to be impacted by the pandemic - and some of them may not come back.

Although the average effects are now more stable over time, we still observe a high variance in treatment effects on individual teams. This motivates us to further analyze what kind of teams are more likely to be affected by the pandemic and what teams are more likely to bounce back quickly.

Are the counterfactual prediction models trustworthy for the downstream analysis of heterogeneous treatment effects, especially when the prediction task is quite difficult? The key to answer this question is to understand the effects of the potential prediction errors. In other words, if we can derive findings that are robust to the potential errors of the counterfactual predictions, the analysis is not limited by the goodness of the prediction models.

The challenge is how to estimate the prediction errors. Recall that all teams are “treated” during the pandemic, we can not directly obtain the prediction error because they are counterfactual. We adopt a principled statistical method, split conformal prediction (Lei et al. 2018), to infer the errors of the out-of-sample counterfactual predictions from the distribution of residuals on the test data.⁸⁸8The conformal inference can be made under the assumption of exchangeability of the two datasets, which holds as we can predict future outcomes for teams in the target set with models trained with the reference set.

More specifically, given a model trained for a specific month, denote the prediction error by $E$ and the number of repositories in the test set by $n$ (i.e., 20% of the reference set), the goal is to obtain a conformal interval of $E$ given a miscoverage level $\alpha$ ($\alpha=5\%$). We infer the conformal interval of $E$ in the target set with the following two steps (see Algorithm 2 in (Lei et al. 2018)). (1) We draw the residuals of predictions for each repository in the test set and get their absolute values. (2) We rank the absolute residuals and get the $k$-th smallest value $d$, where $k=\lceil(n+1)(1-\alpha)\rceil$. The probability that $E$ falls within $[-d,d]$ is no less than $1-\alpha$ (i.e., 95%). This conformal interval gives us a reasonable estimation of the error distribution of the counterfactual predictions and can be used for resampling residuals for bootstrapped regressions in the following analysis.

We report the bootstrapped regressions on productivity in March 2020 and that on team size in June 2020, in which we observe obvious shock effects (Figure 3). Specifically, we report the median and the 95% confidence interval of the coefficient for each independent variable in these two settings. We illustrate the significance of coefficients with an example in Figure 4. When the dependent variable is the shock effects on team productivity in March 2020, the coefficients of the variable entropy of working days in the 1,000 bootstrapped regressions have a median of $-0.2$ and a 95% confidence interval of $[-0.26,-0.15]$, showing that the negative relation between that variable and the outcome is significant.

We then examine the effect of certain variables more closely from the tables and note the following observations.

Team size: larger teams are more resilient to the shock in terms of maintaining their sizes/growth. Size doesn’t show a significant relation to maintaining productivity.

Multitasking: Teams with a higher number of dedicated members (active only in this repository) are less resilient to the shock in terms of both productivity and growth. A high proportion (98% in (Vasilescu et al. 2016)) of prolific developers contribute to multiple projects. With productivity levels controlled, a high number of dedicated members may indicate the irreplaceability of their roles, which reduces the flexibility of the team under shock. On the other hand, members contributing to more repositories at the same period could mean a higher level of engagement, which makes the team more resilient in terms of productivity (but not growth).

Tenure: The resilience in productivity increases when the max coding tenure increases, indicating the benefit to have senior contributors/experts. The median of coding tenures, however, shows a negative correlation with the productivity resilience. One possible explanation is that the senior developers are more likely playing roles of management and coordination in the team (just like senior developers in IT companies), while junior developers may spend more time to learn (Lee et al. 2013) and make contributions to build their reputation (Lakhani and Von Hippel 2004). It is not surprising to observe a positive correlation between the median of coding tenures and the team size resilience, as teams with less junior developers may have more room for junior developers to join. Teams that have more senior people and more new members are more resilient in maintaining their sizes, while the latter may help recruit new members and the former may help retain existing members. The max platform tenure shows a negative effect on team size resilience. Diversity in platform tenures shows negative effect on productivity resilience, possibly due to different levels of adaptability to the remote collaboration mode.

Prestige: Member prestige (measured by the number of followers) has no significant effect on productivity resilience except for the max number of followers. The max number of followers of a team member is positively related to resilience in growth, possibly because that popular users help attract their followers to new projects (Blincoe et al. 2016). However, both the median and the diversity of prestige show a negative correlation with the resilience in growth, suggesting that even if a team has some very popular members, having a large number of junior members still helps attract new members. The new members attracted by the most popular member may not be able to immediately contribute to productivity, which helps explain the negative correlation between the max number of followers and productivity resilience.

Activity: Teams with a higher level of activity (push or issues) are more resilient in both productivity and growth. Teams with a higher level of pull requests are less resilient in productivity while more resilient in growth, probably because a higher level of collaboration helps retention. Teams with more watchers are less likely to maintain their productivity under the shock.

Diversity: Open source software development often benefits from worldwide collaborations. However, our results show that the country diversity plays a negative role in resilience in both outcomes under the shock. Teams with members from more countries are more likely to experience a reduction in productivity and size, possibly because teams with members from many countries tend to be loosely connected (Wuthnow 1998), or because the policies and contexts in different countries add more uncertainties and complications to the challenges under a worldwide pandemic.

The programming languages that one masters represent their technical background. Programming language diversity has a negative effect on growth, likely because it is hard to recruit members with similar technical backgrounds.

Work schedule: The length of off segment with a high threshold (64) is negatively correlated with both productivity and team size resilience. Off segment with a low threshold (16) is positively related to resilience in growth. A longer off-segment means fewer members work in these hours. At a higher cutoff threshold, this can be due to different time zones of members or a spanned-out working schedule. At a lower threshold, this indicates most members maintain a clear boundary between work and life, which is beneficial to the growth of a team. The entropy of working days in a week shows a negative effect on productivity resilience, which implies that a spread-out work schedule doesn’t help maintain productivity under shock. This however has a positive relation to the resilience in team size.

Others: Team age increases resilience in growth but not in productivity. The proportion of emoji posts has no significant effect, which may imply that a more accurate way of measuring the emotional status of teams is needed.

We then examine the consistency of the significance and directions for the coefficients of team properties in multiple regressions across the six months (see Appendix). Note that the effect of the shock on a team can be both short-term and longer-term, and the signs and significance levels of corresponding factors may vary across different time horizons.

Most features present a consistent sign (but some with various coefficients and significance levels) over the time horizons. Some present interesting patterns. The number of countries, the entropy of working days, activity levels in push and issues are fairly robust across time horizons for productivity resilience. Team size, the median of platform tenure, number of dedicated members, member prestige, number of countries, number of programming languages, and off segments are fairly robust for resilience in team size. The proportion of emoji posts has a significantly positive effect on team size resilience in the first four months (months 1, 2, and 4 for productivity), but the effect fades in later months. This indicates that emotional intelligence may be an effective factor to resist the immediate effects of a shock, but its effect might not last long enough. Team productivity levels before the pandemic have a negative effect on the resilience of team size in the first three months but the effect becomes positive in later months. Highly productive teams might be busy maintaining their activity levels in a short term, but when the situation stables, they will be back on track for quick growth. Older teams seem to be more resilient overall, although the effect size is small.