A Case Study on Software Vulnerability Coordination

Coordination can be defined as a phenomenon that originates from dependencies between different activities, and as a way to manage these dependencies between the activities [36]. Software engineering is team work. As team work implies social dependencies between engineers, there is always some amount of coordination in all software engineering projects involving two or more engineers. There are also technical dependencies in all software products. Early work on software engineering coordination focused on the reduction of such technical dependencies in order to improve task allocation and work parallelism [14]. However, not all technical dependencies can be eliminated. For this reason, later work has often adopted a socio-technical perspective for studying software engineering coordination.

The augmentation of technical characteristics with social tenets is visible at a number of different fronts. One relates to the research questions examined. Examples include synchronization of work and milestones, incremental integration of activities, arrangement of globally distributed teams, frequent deliveries, and related project management processes [53, 63, 81]. Another relates to the research methodologies used. While interviews and surveys are still often used [14, 53, 81], techniques such as social network analysis have become increasingly common for examining software engineering coordination [12, 101]. Despite of these changes, one fundamental theoretical premise has remained more or less constant over the years.

Coordination requires communication, and coordination failures are often due to communication problems [14, 115]. The consequences from coordination failures vary. Typical examples include schedule slips, duplication of work, build failures, and software bugs [12, 14]. Communication obstacles, coordination failures, and the resulting problems are well-known also in the security domain.

A good example would be incident and abuse notifications for vulnerable Internet domains: it is notoriously difficult to communicate the security issues discovered to the owners or maintainers of such vulnerable domains [16]. Analogous problems have been prevalent in vulnerability disclosure, which refers to the practices and processes via which the vulnerability discoverers make their discoveries known to the vendors whose products are affected either directly or via a third-party coordinator. Although there have long been recommendations and guidelines [20], it is still today often difficult to communicate vulnerability discoveries to vendors [85]. A substantial amount of effort may be devoted to coordinate the disclosure and remediation of high-profile vulnerabilities, but difficulties often occur for more mundane but no less important vulnerabilities. Some vendors are reluctant to participate in vulnerability disclosure—and some vendors even avoid patching their software products altogether. Even legal threats are still today not unheard of. While vulnerability disclosure is a prime example of coordination (failures) in the software security context, it has only a narrow scope.

As Steven M. Christey—the primary architect behind the current CVE tracking—has argued, the term vulnerability coordination is preferable because vulnerability disclosure only captures a small portion of the activities required to handle and archive vulnerabilities [19]. In fact, vulnerability coordination is a relatively frequent activity among the integrative software engineering activities carried out by open source developers [2]. Bug reports must be handled in a timely manner for security issues. Evaluation is needed to assess the versioned products affected. Of course, also fixes must be written, but fixes may further require careful backtracking within version control systems, debugging, reviews from peers, testing done by peers and users alike, and coordination between business partners or third-party open source software projects. Then, erratas, security advisories, or other notes must be written, identifiers must be allocated to uniquely track the notes and to map these to other identifiers, preparations may be required for answering to user feedback, and so forth. These are all examples of the software engineering activities that are typically present in the software vulnerability context.

For empirical software engineering research, the identifiers used to track vulnerabilities are particularly relevant. The most important identifier is CVE, in terms of both research and practice. For open source projects, having a single canonical identifier helps developers and users to coordinate their efforts. Given the inconsistency issues affecting open bug trackers [86, 108], CVEs help at addressing questions such as: “well it sounds like this one but maybe it’s that other one?” [100]. For a security professional, CVEs are ubiquitous entries in a curriculum vitae. For research, CVEs provide the starting point for connecting the distinct engineering activities into a coherent schema. While there is an abundant amount of existing research operating with CVE-based research schemas, thus far, only one attempt [95] has been made to better understand the primary schema; software and security engineering coordination is required for CVEs themselves.

Software development teams tend to produce software products that reflect the communication structure of the teams, to paraphrase the famous law formulated by Melvin E. Conway [23]. Although the law is not directly applicable to the context of CVE coordination, the oss-security case supports a weaker variant of the same basic assertion. This variant might be defined as a statement that a communication structure tends to reflect the type of software engineering work and the medium used for communication.

The type of work done and the medium both characterize also the social network structure of the CVE coordination through the mailing list. Both limit also the applicability of common theoretical interpretations. For instance, knowledge sharing is a typical way to motivate research on open source social networks [50, 57, 109], but sharing of knowledge was not the main purpose of the oss-security mailing list during the period studied.

The primary purpose of the list was to coordinate the assignment of CVE identifiers for discovered and usually already disclosed software vulnerabilities. This coordination was done by explicitly requesting CVE identifiers, which were then assigned by MITRE affiliates based on a brief evaluation. In terms of communication practices, this coordination practice often culminated in short replies, such as “use CVE-2016-6526” [78], made by MITRE affiliates to previously posted CVE requests. Participants typically kept MITRE’s [email protected] email address in the carbon copy field when they posted a request, further using a subject line that identified the message as a request. While longer discussions were not unheard of, these communication practices indicate that knowledge sharing has only limited appeal for framing the case theoretically. Exchanging information about abstract identifiers does not necessarily imply thorough discussions about the technical details of the vulnerabilities coordinated. In other words, data does not equate to information, and information does not equal knowledge.

The coordination work through the list produced clear core social network components [95]. This observation is classical in the open source context [22, 24, 57, 110], but the contextual interpretation is still relevant. Due to the way CVEs are assigned, the cores centered to MITRE affiliates. By using a network representation described later in detail, the core social network components are illustrated in Fig. 2. Excluding the use of MITRE’s email address without an explicit sender, there are two identifiable individual participants with degrees higher than $800$, meaning that these two participants both coordinated at least eight hundred CVEs each. Both participants are MITRE affiliates. What is more, there are only a few CVEs that link the three core participants to each other. This observation indicates that task allocation and related coordination techniques apply also to the case studied.

These core social network components are a good example of so-called communication brokers who help at resolving coordination obstacles in software engineering [116]. Such brokers integrate information from multiple sources. This integrative engineering work often leads to a social network structure with relatively high level of centrality, low level of clustering, and star-like centers within which the integrative work is located [82]. This theoretical characterization applies well also to the social networks for the CVE coordination through the mailing list.

Thus, the type of software engineering work carried out is an important factor characterizing the corresponding social network structures. In terms of development mailing lists, non-core participants may post as many messages as core participants [25], but this observation does not generalize to all cases [34], including the case studied. Although coordination requires communication, coordination tends to result in different network structures than communication required for other work (or leisure) activities. A communication medium used for coordination presumably also affects the emerging network structure. For instance, social networks constructed from bug tracking systems indicate that many individuals may report bugs even though the actual development may be concentrated to a small core group [120]. A communication medium also places restrictions over who can participate, how much can be communicated, and what is communicated [12, 84]. While oss-security is open for anyone to participate, the communication volume and the content of messages exchanged are both important for further elaborating the case.

The integrative coordination work done on the list reflected not only social but also loose technical dependencies between different tracking infrastructures. The classical consumer-producer abstraction for software engineering coordination [62, 63] is useful for framing these technical dependencies theoretically. According to this abstraction, there exists a prerequisite constraint: the work activity of a producer must be usually completed before work can start on the consumer side [36, 62]. In terms of the case studied, the participants (producers) who requested CVEs provided sufficient technical information to justify the requests for the MITRE affiliates (consumers). If the information was sufficient for a request, the prerequisite constraint was satisfied, and the vulnerability in question later appeared in NVD with the CVE requested. Instead of in-depth knowledge sharing, the sufficient information was usually delivered via hyperlinks to other open source tracking infrastructures within which a given vulnerability had already been discussed. Consequently, the discovery, disclosure, and patching of the vulnerabilities coordinated had almost always occurred before information appeared on the mailing list.

The preceding discussion motivates three questions worth asking about the delays between CVE requests on the mailing list and the later appearance of these identifiers in the central tracking database. Given the medium used and the type of software engineering coordination done, these questions can be framed by separating the two terms in the concept of socio-technical coordination. The first question addresses the social dimension:

1. RQ₁

   Have the CVE coordination delays been affected by the social networks and communication practices between the participants on the oss-security mailing list?

The remaining two questions address the technical dimension in socio-technical coordination. Given the integrative work done and the consumer-producer abstraction, it is worth asking the following question about the delays:

1. RQ₂

   Did traces to other tracking infrastructures affect the coordination delays during the period observed?

The third and final research question approaches the technical dimension from a more direct perspective:

1. RQ₃

   Were the coordination delays affected by the technical characteristics of the vulnerabilities coordinated?

The analytical meaning behind the three questions is illustrated in Fig. 3. It should be noted that causal inference is not attempted; hence, no arrows are drawn in the figure between the three explanatory dimensions.

The dataset is compiled from two sources. All email messages posted on oss-security were obtained from the online Openwall archive [77]. These messages are cross-referenced with CVE identifiers to the second data source, NVD [74]. The sampling period runs from the first welcome message in February 2008 to the emails posted in December 31 2016. This sampling interval corresponds with the historical period during which the mailing list was used for CVE assignments.

There were a little over sixteen thousand messages posted during this time interval. For each message delivered in the hypertext markup language (HTML) format, a twofold routine is used for pre-processing the archival data (see Fig. 4). The first pre-processing subroutine extracts the actual messages from the HTML markup, further omitting all lines referring to quotations from previous messages posted on the mailing list. To some extent, also forwarded messages are excluded based on simple but previously used heuristics [118]. Due to the markup language, this pre-processing routine is likely to yield some inconsistencies. However, the consequences for the empirical analysis should be relatively small because no attempts are made to pre-process email threads. This choice is largely imposed by the use of the online archive. In particular, the fields Message-ID, In-Reply-To, and References are not delivered via the online interface, which prevents the use of common (meta-data) techniques for parsing email threads [34, 112]. Although content-based alternatives exist for reconstructing threaded email structures [28, 98], only the senders of emails are considered in order to improve data quality and to simplify the data processing.

The second subroutine is used for identification. The From email header field is used for identifying the individual participants by using the first match from a simple Python regular expression. Although the matching itself is simple, it should be emphasized that all senders are identified according to their names rather than their addresses. This option is often preferred because individuals’ email addresses tend to vary [12, 73, 109]. The choice is again also imposed by the online archive, which obfuscates the email addresses for spam prevention and related reasons. Consequently, additional pre-processing techniques [71] for mapping identified names to addresses cannot be used.

Furthermore, Levenshtein’s [54] classical distance metric between two names, say $L(x,y)$, is a decent choice for accounting small inconsistencies [122]. If $s_{1}$ and $s_{2}$ denote two full names with lengths $l_{1}$ and $l_{2}$, a similarity score is computed through $\delta=1.0-L(s_{1},s_{2})/\max\{l_{1},l_{2}\}$ [18]. If the scalar $\delta$ then exceeds a threshold value $0.8$, two names are taken to refer to the same individual. Although the threshold is subjective, it captures some typical cases, such as “John Doe” who occasionally writes his name as “John, Doe” when sending emails. The few outlying cases that exceeded the $\delta=0.8$ threshold were further evaluated manually. This additional check revealed no obvious approximation mistakes. In addition, “Christey, Steven M.” and “Steven M. Christey” were merged manually.

Also CVEs are identified with a regular expression, which is a typical approach for searching vulnerabilities from heterogeneous sources [4, 58]. The expression takes into account both the old and deprecated candidate (CAN) syntax as well as the recently made syntax change that allows an arbitrary amount of digits in the second part of CVE identifiers [66, 67]. Even though forwarded messages and direct quotations to previous emails are approximately excluded, it should be noted that there can be many-to-many relations between messages and CVEs. For instance, a lengthy security advisory posted on the list may contain a large amount of individual CVE-referenced vulnerabilities. Such cases are included in the sample. Finally, only those CVEs are qualified that have also valid entries in NVD. Most of the invalid entries in the database refer to identifiers that were assigned but which were later rejected from inclusion to the database. Given that these disqualified CVE assignments cause different biases [19], all rejected identifiers were excluded from the empirical sample. The matching of these invalid entries was done by searching for the string REJECT in the summary field provided by NVD.

There are multiple ways to measure the efficiency of software engineering coordination [53]. In the context of software bugs in general, a good example would be the validity of bug reports. Bug tracking infrastructures typically reserve multiple categories for classifying invalid reports. These categories include classes for already fixed bugs, irreproducible bugs, and duplicate reports, among other groups. If a large amount of bug reports end up into such classes, coordination would be generally inefficient.

As coordination deals with dependencies, it is no surprise that also social network structures of bug reporters have been observed to affect the probability of valid bug reports  [120]. Although the generalizability of this observation seems limited [10], there are good reasons to suspect that an analogous effect is present in the software vulnerability context. Because attribution is particularly important for vulnerabilities and monetary compensations are relatively common, it is no wonder that invalid—or even outright fake—reports have been a typical menace affecting vulnerability coordination [19, 52]. For this reason, vulnerability reports from established discoverers likely increase the validity of the reports as well as the trust placed on the reports. This trust provision is also reflected on the CVE assignment authorities granted for a few individual security researchers.

Due to the sensitiveness of the information coordinated, the availability of open data is limited about the internal vulnerability tracking systems used by software vendors, MITRE, and other actors. Many open source projects also limit the visibility of security bugs, or otherwise try to constrain the exposure of public information.

These data limitations have affected also the ways to quantify longitudinal vulnerability information. For instance, the efficiency of vulnerability disclosure can be approximately measured with a time difference between a disclosure notification sent to a software vendor and the vendor’s reply [7, 90]. Analogously, patch release delays can be measured by fixing the other endpoint to the dates on which vendors released patches for the vulnerabilities disclosed to the vendors [64, 107]. Similar delays can be measured also in terms of initial bug reports and later CVE assignments based on the reports [113]. Further examples include the timing of security advisories  [93], the dates on which signatures were added to intrusion detection and related systems [11], and exploit release dates for known vulnerabilities [1, 4, 15]. All these different dates convey different viewpoints on the coordination of vulnerabilities.

In this paper, analogously, the empirical interest relates to the following per-CVE time differences (in days):

given

vulnerabilities observed. The timestamp $T_{\textmd{NVD}_{i}}$ records the date on which the $i$:th CVE was first stored to NVD. The second timestamp $T_{\textmd{{oss-security}}_{i}}$ refers to the earliest date on which this CVE was posted on the mailing list. The restriction in (2) excludes already archived CVEs that were later discussed on the mailing list. Thus, the integer $y_{i}\geq 0$ approximates the length of the path (b) in Fig. 1. In theory, also the path (a) in Fig. 1 could be measured, but the data from the online archive makes it difficult to identify the initial CVE requests. This limitation does not affect the validity of the delay metric, but it does affect the interpretation given for it.

The metric in (1) approximates the time delays between the initial CVE assignments done through the oss-security mailing list and the usually later appearance of the requested CVEs in NVD. Therefore, $y_{i}$ focuses on the internal coordination done by MITRE affiliates, the NVD team, and other actors involved in the coordination. One way to think about this internal coordination is to consider the delay metric as an indirect quantity for measuring how fast work items in a backlog or an inventory are transferred to complete deliveries [62, 88]. Due to data limitations, however, it is currently neither possible to observe the internal within-MITRE (or within-NVD) coordination directly nor to explicitly measure the work items in the CVE backlog. Therefore, the delay metric used provides a sensible but not entirely reliable way to approximate the typical coordination delays that affected one particular CVE coordination channel.

The socio-technical coordination and communication characteristics are proxied by observing so-called task-based [116] or person-task [101] social networks. Thus, the underlying social network structure is bipartite, meaning that there are two types of vertices. An edge connecting any two vertices always contains both types; there are no edges that would connect a vertex of one type to a vertex of the same type. In addition, the network structure is unweighted and undirected. More formally:

where $G_{s}$ denotes an observed social network constructed from the messages that were posted on the mailing list, from the first message posted in 2008 to the last message in 2016. The two disjoint vertex sets $P$ and $A$ refer to participants and CVE identifiers, respectively. If a participant $p\in P$ has sent a message containing a CVE identifier $a\in A$, an undirected edge, $(p,a)=(a,p)$, is present in the edge set $E$. Therefore, individual participants are linked together through CVE identifiers (see Fig. 5). Due to the bipartite structure, it holds that $P\cap A=\emptyset$ and

where

Furthermore, another network is constructed from the uniform resource locators (URLs) embedded in the hyperlinks present in the emails that contained also CVEs. The structure is again undirected, unweighted, and bipartite:

where $D$ denotes a set of domain names extracted from the URLs sent by participants in $P$, $B$ denotes another set of CVEs, and $F$ is an edge set containing edges from the domain names to the CVEs. If a participant $p\in P$ sent an email containing a $b\in B$ and a domain name $d\in D$ extracted from a URL in a hyperlink, an undirected edge $(b,d)=(d,b)$ is present in the set $F$. Thus, CVEs are linked also to domain names through the participants who posted messages containing both CVE identifiers and hyperlinks. Therefore, $B\subseteq A$ and $n=|A|\geq|B|$ because the subset of CVE identifiers stored to $B$ are required to also have mappings to domain names.

The network $G_{s}$ is a social network in the traditional sense; even though participants are linked to each other through abstract identifiers, the participants are still human beings. The network $G_{d}$, in contrast, resembles more the so-called domain name system graphs within which domain names are connected to each other via Internet protocol (IP) addresses or by other technical relations [94, 103]. Consequently, it would be possible to manipulate $G_{d}$ by resolving the addresses of the domain names or by considering only the second-level domain names [96]. Given the historical context, however, no attempts are made to resolve the domain names, many of which are nonexistent today. Instead, only three semantic validation checks are enforced: (a) the length of each $d\in D$ is asserted to be at least three characters; (b) each $d$ is required to contain a dot character; (c) and no entry in $D$ is allowed to refer to a semantically valid IPv4 address.

The three research questions are evaluated by regressing the coordination delays $y_{1},\ldots,y_{n}$ against the metrics enumerated in Table 1. Six models ($\operatorname{M}$) are used for the statistical computations; the integer $k$ denotes the cumulative number of metrics included in the consecutively estimated models, including the intercept. The table shows also the scale of the metrics; there are a few continuous ($\operatorname{\mathcal{C}}$) metrics but most of the metrics are dichotomous ($\operatorname{\mathcal{D}}$) dummy variables. The metrics enumerated can be further elaborated according to the models within which these first appear.

The coordination delay vector $\textup{{{y}}}=[y_{1},\ldots,y_{n}]^{\prime}$ represent count data: the observations count the days between CVE assignments on the mailing list and the publication of the CVEs assigned within the primary global tracking database. Therefore, Poisson regression [88, 93] and regression estimators for survival analysis [1, 107] have been typical statistical estimation strategies in the research domain. Previous work in comparable settings [7, 26, 88, 90] indicates that the ordinary least squares (OLS) regression often works also sufficiently well when the dependent metric is passed through a transformation function $f(x)=\log(x+1)$. This simple OLS approach is taken as the baseline for the empirical analysis. For all regression models estimated, the explanatory metrics with continuous scale are also transformed via the same function in order to lessen skew.

Although frequently used in applied research, the OLS approach contains also problems. In a sense, the transformation function is redundant and should be avoided in order to maintain the statistical properties of count data. It also adds small but unnecessary complexity for interpreting the parameter estimates. Furthermore, the residual vector $\boldsymbol{\epsilon}$ from the OLS model is assumed to be independent and identically distributed from the normal distribution with a mean of zero and variance $\sigma^{2}$. This basic assumption can be written as $\operatorname{E}(\boldsymbol{\epsilon}~{}|~{}\textup{{{X}}}_{j})=\boldsymbol{0}$ and $\operatorname{E}(\boldsymbol{\epsilon}\boldsymbol{\epsilon}^{\prime}~{}|~{}\textup{{{X}}}_{j})=\sigma^{2}\textup{{{I}}}$, where $\operatorname{E}(\cdot)$ denotes the expected value, $\textup{{{X}}}_{j}$ the $j$:th model matrix for a given $\operatorname{M}_{j}$ from Table 1, and I an identity matrix. If the assumption is satisfied,

where $\boldsymbol{\beta}$ is a regression coefficient vector and $N(\cdot)$ refers to the multivariate normal distribution. This distributional assumption allows exact inference based on $t$ and $F$ distributions. In many empirical software engineering applications the problem is not as much about the (asymptotic) normality assumption as it is about the unreliable variance patterns typically present in the typically messy datasets [44]. Thus, often, $\operatorname{E}(\boldsymbol{\epsilon}\boldsymbol{\epsilon}^{\prime}~{}|~{}\textup{{{X}}}_{j})=\sigma^{2}\textup{{{V}}}$, where V is a generally unknown non-diagonal matrix that establishes some systematic pattern in the residuals. Such tendency is generally known as heteroskedasticity.

For instance, some vulnerability time series are known to exhibit time-dependent heteroskedastic patterns [91, 106]. In the present context heteroskedasticity is to be expected due to the count data characteristics, and possibly also due to the heavy use of dichotomous variables. Either way, it is important to emphasize that asymptotic inference is still possible under heteroskedasticity; the consistency of $\boldsymbol{\beta}$ remains unaffected but the estimates are inaccurate. An analogous consequence results from multicollinearity; the stronger the correlation between a variable and other variables, the higher the variance of the regression coefficient for the variable.

When heteroskedasticity is an issue, more accurate statistical significance testing can be done by adjusting the variance-covariance matrix, $\operatorname{Var}(\boldsymbol{\beta}~{}|~{}\textup{{{X}}}_{j})=\sigma^{2}[\textup{{{X}}}^{\prime}_{j}\textup{{{X}}}_{j}]^{-1}$, with well-known techniques based on the unknown V estimated from data. The OLS results reported use the MacKinnon-White adjustment [61] conveniently available from existing implementations [123]. Another point is that rather analogous problems are often encountered with count data regressions. For instance, the negative binomial distribution is often preferable over the Poisson distribution [88]. Also the gamma distribution is known to characterize related difference-based vulnerability datasets [39]. Furthermore, conventional methods for survival analysis face some typical problems. Although not worth explicitly reporting, it can be shown that the Cox regression’s so-called proportional hazards assumption fails for most of the metrics in $\operatorname{M}_{1},\ldots,\operatorname{M}_{6}$, for instance. Due to these and other reasons, it beneficial to use a regression estimator that makes no distributional assumptions. Quantile regression (QR) is one of such estimators.

Ordinary least squares provides estimates for the conditional mean. Quantile regression estimates quantiles. The $\tau$:th quantile is defined by

where $F(\cdot)$ denotes a cumulative distribution function, while the \textcolorblackinfimum operator is used to denote the smallest real number for which the condition $F(x)\geq\tau$ is satisfied. If $\tau=0.5$, for instance, QR provides a solution $\hat{\boldsymbol{\beta}}$ for the conditional median of y conditional on $\textup{{{X}}}_{j}$. Estimation minimizes the sum of absolute residuals:

where $\beta_{0}$ is the intercept, $\textup{{{x}}}^{\prime}_{i(j)}$ is the $i$:th row vector from the $j$:th model matrix $\textup{{{X}}}_{j}$, and $\rho_{\tau}(x)=x[\tau-I(x<0)]$ with $0<\tau<1$ and $I(\cdot)$ denoting an indicator function [3, 47]. The optimization of (11) is computed with a so-called Frisch-Newton algorithm available in existing implementations [48]. The benefits from QR are well-known and well-documented. Among these are the lack of distributional assumptions and the robustness against outliers. Although asymptotic assumptions are still required for significance testing under heteroskedasticity, no parametric assumptions are made regarding the residual vector [45]. For applied research, quantile regression has also an immediate appeal in that the whole range in the conditional distribution of the explained variable can be observed. This potential is also relevant for vulnerability delay metrics, which typically (but not necessarily) tend to be distributed from a long-tailed distribution.

There are a couple of additional points that still warrant brief attention. First, potential non-linearities should be accounted for also with QR. As the dataset contains many variables but only five of these have a $\operatorname{\mathcal{C}}$ scale (see Table 1), non-linear modeling of the explanatory metrics can be reasonably left for further work, however. The $\log(x+1)$ transformation applied to these five variables also lessens some of these concerns (particularly regarding MSGSLEN and MSGSENT). Second, multicollinearity is always a potential issue with typical empirical software engineering datasets. Although the concern is not as pressing as with software source code metrics [30], analogous datasets containing social network and communication metrics allow to also expect potential multicollinearity issues [8]. Instead of adopting techniques such as principal component regression—which would make the interpretation of Fig. 3 difficult—the QR models are re-checked with the least absolute shrinkage and selection operator (LASSO). For quantile regression, LASSO amounts to optimizing

where $\lVert\cdot\rVert_{1}$ denotes the $L_{1}$-norm and $\lambda$ is a non-negative tuning parameter [3]. Although cross-validation and other techniques can be used for selecting the tuning parameter [88], the QR-LASSO regressions are estimated with $\lambda\in[1,2,\ldots,100]$ using the full $\operatorname{M}_{6}$. This range captures most of the regularization applicable to the dataset.

The rationale behind (12) follows the rationale of LASSO in general: when $\lambda$ increases, the quantile regression coefficients shrink toward zero. This regularization makes LASSO useful as a variable selection tool for high-dimensional and ill-conditioned datasets. When $\lambda$ is sufficiently large, some of the coefficients shrink to zero, leaving a group of more relevant coefficients. The model selection properties are not entirely ideal, however. When a coefficient for a correlated variable is regularized to zero, the coefficients of the other correlated variables are affected; that is, LASSO tends to select one correlated variable from a group of highly correlated variables [17]. It is worth to remark that also the common alternatives contain problems. In addition to issues related to multiple comparisons, the expected heteroskedasticity presumably causes problems for a stepwise variable selection algorithm particularly in case the algorithm uses statistical significance to make decisions. Instead of relying on such algorithms, a more traditional is adopted for the modeling and inference.

Applied regression modeling has two essential functions. It can be used to make predictions based on explanatory metrics, or it can be used to examine “the strength of a theoretical relationship” between the explained metric and the explanatory metrics [32]. Already because prediction is not a sensible research approach for a historical case study, this paper leans toward the examination of theoretical relationships based on classical statistical inference. The two functions cannot be arguably separated from each other, however. Given the notorious data limitations affecting vulnerability archiving [19], the inevitable noise introduced by the collection of online data, and many related reasons, a watchful eye should be kept for asserting the presence of a signal in terms of prediction. The magnitudes of the regression coefficients, the effect sizes, are used to balance the final subjective judgment calls regarding the research questions RQ₁, RQ₂, and RQ₃.

The six models $\operatorname{M}_{1},\ldots,\operatorname{M}_{6}$ are fitted consecutively. This hierarchical modeling approach [8] is also known as a bottom-up or specific-to-general modeling strategy [60]. For comparing the six OLS models, the adjusted coefficients of determination (adj. $R^{2}$) and Akaike’s information criterion (AIC) values are used. Higher and lower values, respectively, indicate better performance according to these two common evaluation statistics. Both penalize the performance by the number of parameters. Although pseudo-$R^{2}$ measures are available for quantile regression [47], AIC is used to summarize the general signal-to-noise-type of performance of four quantile regressions estimated for each of the six models. The following quantiles are used in order to have probes across a wide range of the conditional distribution of the coordination delays:

It should be noted that neither censoring nor transformations are applied for the delays when estimated with QR. Consequently, the predicted values may be also negative, which is theoretically impossible. As prediction is not a goal, this limitation can be accepted.

The specific-to-general strategy is used also for formal testing with two analytical dimensions. First, the nested structure is exploited to test parameter restrictions between models. The testing is done by comparing (restricted) $\operatorname{M}_{j-1}$ against (unrestricted) $\operatorname{M}_{j}$ for $1<j\leq 6$. The procedure follows the standard (Wald’s) logic to compare nested models [47]. In fact, also the test statistics are delivered via $F$-like statistics [45]. Second, QR provides also means to test whether the coefficients are equal for a given $\operatorname{M}_{j}$ across the whole (13) or subsets thereof. For instance, to evaluate whether the effect of WEEKEND remains constant across the proxied conditional range of the coordination delays, the twentieth coefficients from $\{\hat{\boldsymbol{\beta}}_{\tau_{1}},\hat{\boldsymbol{\beta}}_{\tau_{2}},\hat{\boldsymbol{\beta}}_{\tau_{3}},\hat{\boldsymbol{\beta}}_{\tau_{4}}\}_{j}$ for the $j$:th model would be used to test that $\hat{\beta}_{20j\tau_{1}}\simeq\hat{\beta}_{20j\tau_{2}}\simeq\hat{\beta}_{20j\tau_{3}}\simeq\hat{\beta}_{20j\tau_{4}}$. As it is reasonable to speculate that the effect of the explanatory metrics differ particularly at the tails, the coefficients in the following sets are used for the between-quantile testing:

The bootstrap procedure available from the implementation used [48] is used to compute the statistical significance for the between-model tests. To accompany the MacKinnon-White adjustment for the OLS estimates, the same bootstrap procedure is used also for reporting the statistical significance of the coefficients from the QR regressions. Unfortunately, analogous procedure has not been implemented for the between-quantile tests. The conventional approach [45] is thus used instead.

Finally, the regression analysis is accompanied with a brief classification experiment to further assess the general performance. Following the existing bug tracking research [35, 124], this experiment is conducted by splitting the coordination delays into low-delay and high-delay groups according to median. Although this sample splitting is a good example of data manipulation that can be avoided with quantile regression [46], it provides a simple additional assertion regarding the overall performance across the six models. Classification accuracy is a sufficient evaluation metric for this simple purpose. Estimation is done with a readily available and well-known random forest classifier [49, 56]. Ten-fold cross-validation is used during training. Given that prediction of new data is not a realistic scenario for the historical oss-security case studied, testing is done with a randomly picked set containing ten percent of the CVEs observed. The same test set is used for all six classification models.

The sample contains $n=5,780$ identifiers once the exclusion criteria in (2) is enforced (see Table 3). These were discussed by about five hundred unique participants who posted hyperlinks containing $4,642$ unique domains. The mean and median delays were $77$ and $15$ days, respectively. The values for (13) are $4$, $15$, $92$, and $226$ days. As can be seen from the histogram in the outer plot of Fig. 7, the delay distribution indeed has a long tail. This tail contributes to the large standard deviation of $147$ days. Thus, there is a large majority group of CVEs that were coordinated rapidly and a small but important group of CVEs for which the coordination was significantly delayed. For a few outlying CVEs, the coordination has taken even over four years. To balance this remark, it can be noted that about 10% of the cases observed attain a value zero, meaning that these CVEs appeared in NVD during the same day when these were requested on the mailing list.

As can be seen from the inner plot in Fig. 7, the transformation function $f(x)=\log(x+1)$ does not yield normally distributed delays, although the shape of the delay distribution is still better suited for OLS regression after the transformation. To examine whether immediate multicollinearity issues are also visible, Fig. 9 displays the absolute correlation coefficients between all explanatory variables. (As with the regression analysis, the transformation function is used also for the continuous variables marked with the symbol $\operatorname{\mathcal{C}}$ in Table 1.) The interpretation is simple: the bigger a circle and the darker a color, the stronger the correlation between any two metrics. Given this simple visual decoding guide, there are two notable correlations that warrant a multicollinearity concern.

The first is between MITREDEV and MSGSENT. The correlation between these two metrics reflects the typical “use this identifier” replies made by the MITRE affiliates. In other words, such replies tend to result in a lower entropy for the messages referencing CVEs. The second notable correlation is between INFDEG and MSGSLEN. Also this correlation is logical: posting many hyperlinks increases the length of the messages. Despite of these correlations, the regression estimates do not change notably when only MITREDEV and INFDEG (or, equivalently, MSGSENT and MSGSLEN) are included in the models. The same applies to the few visible correlations between the CVSS and CWE metrics. For these reasons, the results reported in the subsequent section are based on the original model specifications summarized in Table 1.

To examine the longitudinal variation, the outer plot in Fig. 8 shows the delays across the period observed. This illustration clearly shows that the delays started to increase from 2010 onward, but the increases were mostly caused by outlying CVEs. The arithmetic mean delays shown on the upper $x$-axis indicate that the average annual delays have been below one hundred days. The annual median delays are all below $60$ days. These averages align roughly with the so-called grace periods (the time vendors are given to patch their products before public release of information) typically used in the security industry during vulnerability disclosure. Although the lengths of these grace periods vary, an upper limit of about three months seems to capture most explicitly reinforced policies [64, 90]. Finally, the inner plot in Fig. 8 shows that the increasing delays and the increasing variation corresponded with the increasing amount of CVEs coordinated. The increased coordination volume in turn corresponded with the increased number of participants [95]. These longitudinal changes allow to expect that the baseline model $\operatorname{M}_{1}$ explains a relatively large share of the total variation in the delays. This expectation provides a good way to start the dissemination of the results from the regression analysis.

The statistical performance is somewhat modest regardless of the model. Analogous to interpreting effect sizes [42], adjectives such as modest are subject to interpretation, of course. Values $R^{2}<0.3$ are neither atypical in the vulnerability research domain [7] nor uncommon in empirical software engineering experiments in general. Such values are commonly seen also in social sciences, including economics [43]. In this sense, the about 28% of the total variation explained by the OLS model $\operatorname{M}_{6}$ indicates modest but typical performance for regressions involving human beings. This observation can be seen from Table 4, which shows the adjusted coefficients of determination, the AIC values, and the differences between the AIC values of the consecutively estimated models. The latter two are shown also for each of the four per-model QR regressions.

Four further points can be made about the performance. First, as expected, the longitudinal control variables provide most of the explanatory power. When compared to $\operatorname{M}_{1}$, the social network and communication metrics increase the performance by about one percentage point according to the OLS estimates. The infrastructure metrics subsequently increase the performance by about three percentage points according to the adjusted $R^{2}$ values. Second, the conditional median ($\tau=0.5$) and the conditional log-transformed mean (OLS) models indicate comparable behavior in terms of $\Delta$AIC. Third, the QR estimates indicate that the longer the delays, the bigger the performance gains from $\operatorname{M}_{2}$ and $\operatorname{M}_{3}$. Particularly when the tail is probed with $\tau=0.9$, the social network, communication, and infrastructure metrics reduce the AIC values substantially. In contrast, when short delays are proxied with $\tau=0.25$, these metrics do not bring performance gains. Last: at best, there are only small improvements after $\operatorname{M}_{3}$, and there are also cells with positive $\Delta$AIC increments.

However, all but one of the consecutively estimated nested models yield statistically significant rejections of the between-model parameter restrictions. In other words, 95% of the hypotheses that the smaller (restricted) models would be adequate compared to the larger models are rejected according to the bootstrapped computations. It may also be that information criteria measures such as AIC have problems in differentiating between models when the overall performance is modest [43]. The turn to statistical significance motivates to also take a look at the residuals from the full $\operatorname{M}_{6}$ in the form of Fig. 10. The residuals are rather randomly scattered across the $x$-axes only for the conditional median regression. For $\tau=0.25$, there is a $\cap$-shaped pattern. For $\tau=0.75$ and particularly for $\tau=0.9$, there exists a more visible linear heteroskedasticity pattern; the longer the delays estimated, the larger the residuals. It is beyond the scope of this paper to review and evaluate how the QR-based inference performs under the heteroskedasticity patterns observed. Nevertheless, it seems reasonable to prefer the conditional median regressions and proceed with caution particularly when interpreting the $\tau>0.5$ quantile regressions.

The regression estimates from the full unrestricted model are shown in Fig 11. The accompanying Fig. 12 shows the results from the between-quantile tests. As noted in the previous section, these tests as well as the statistical significance of the coefficients should be interpreted tentatively. While keeping this point in mind, the following enumeration summarizes the key observations.

• •

  The effects of the longitudinal control metrics are consistently strong. When compared to 2008, the increased delays particularly in 2011, 2012, and 2016 are visible especially with respect to the $\tau=0.9$ quantile regression. The same information is conveyed in Fig. 8. There exists also monthly variation in the delays. In addition to annual holidays, another reason may relate to security conferences and related events that tend to spike the public disclosure of new vulnerabilities [37]. In any case, it is simpler to interpret the positive effect of WEEKEND. When compared to other days of the week, the median delays have been about two weeks longer for requests made on weekends. The effect of WEEKEND also increases at the tails of the conditional delay distribution.

• •

  The social network and communication metrics exhibit strong effects. In particular, the assumption about “too many cooks” seem to hold well; the effect of $\log(\textmd{SOCDEG}+1)$ is large particularly for long coordination delays. Also the communication surrogates $\log(\textmd{MSGSLEN}+1)$ and $\log(\textmd{MSGSENT}+1)$ show large effects. The signs vary, however. Because the coefficients are positive for $\tau\leq 0.5$ and negative for $\tau\geq 0.75$, the prior theorization in Subsection 3.4.2 should not be as unequivocal as was presented.

• •

  From the infrastructure metrics, the coefficients for INFDEG, NVDREFS, and BUGS are statistically significant for each regression. The coefficient magnitudes are also notable. If a CVE request was accompanied with a hyperlink to a bug tracking system, the median delay was about five to six days shorter, for instance. As was expected (see Subsection 3.4.3), the signs are also negative for NVDREFS and BUGS but positive for INFDEG. It seems that hyperlinks can also increase the noise, which tends to increase the CVE coordination delays.

• •

  The CVSS and CWE metrics show diverging results. On one hand, only three of the coefficients for the CVSS metrics are statistically significant in the five $\operatorname{M}_{6}$ regressions. The effect sizes are also small, and mostly equal according to the between-quantile tests. On the other hand, many of the CWE metrics attain statistically significant coefficients with large magnitudes. All of the CWE metrics that are statistically significant have negative signs. As these observations apply also for the predominantly web-related weaknesses (CWE-79, CWE-89, and CWE-352; to some extent, also CWE-20), it seems that mundane low-profile vulnerabilities are generally coordinated faster. As was discussed in Subsection 3.4.4, interpretation is not easy, however. The difficulty of interpretation further increases because some of the CWE metrics likely proxy the effects of the CVSS metrics due to multicollinearity (see Fig. 9). All in all, it can be concluded that the coordination delays vary also in terms of the technical characteristics of the vulnerabilities coordinated. As will be shown in the next section, performance can be also slightly increased with additional weaknesses, although the signals from the CVSS and CWE metrics still remain somewhat weak.

Four computational checks are made to assess the robustness of the conclusions. The first check relates to the number of CWEs included; the model $\operatorname{M}_{6}$ includes the ten most frequent CWEs in the sample, but there are $55$ unique CWEs in total. The summary shown in Fig. 13 displays the performance when separate models are estimated by adding $5,10,15,\ldots,55$ weaknesses to the fifth model specification. The case with ten weaknesses thus equals $\operatorname{M}_{6}$. The OLS performance increases up to around $25$ weaknesses. For the median QR regression, all of the fifty-five CWEs seem to steadily reduce the AIC values. The negative $\Delta$AIC increments are also relatively large compared to those in Table 4. These improvements should not be exaggerated, however. The adjusted $R^{2}$ values from the OLS regressions still do not reach the $0.3$ threshold, for instance.

The second check relates to the classification of low-delay and high-delay groups (see Subsection 3.5). The split according to median results in a roughly balanced metric for classification: $2,904$ of the CVEs observed attain a delay less than or equal to the median delay and $2,876$ a delay higher than the median delay of $15$ days. The accuracy of classifying these two groups is shown in Table 5. The accuracy rates increase steadily up to the fourth model and decrease thereafter. These classification results agree with the regression results shown in Table 4.

The third check is about the annual variation. In contrast to subsetting the explained metric, segmenting a model matrix into subsets according to the conditioning explanatory metrics is a sensible regression modeling approach [46]. Thus, to examine whether also the model performance varies annually, all six models are estimated with OLS in annual subsets, omitting the eight yearly dummy variables present in the original model specifications. For instance: if $\operatorname{\tilde{M}}_{1}$ denotes the first subset-model without the annual dummy variables, there are $k=21-8=13$ parameters in the model and its regression coefficient vector. The results from these subset OLS regressions are summarized in Table 6. The table should be only read horizontally by comparing values in a given row across the columns; the different sample sizes in the annual subsets (see Fig. 8) do not allow to make sound comparisons across the years. Given this interpretation guide, the results are clear. For the majority of years, there are no notable performance gains after the model $\operatorname{\tilde{M}}_{3}$. Another way to look at the annual variation is to check how well the large effect sizes of the annual dummy variables (see Fig. 11) balance the effects of the other metrics. For this purpose, the median QR regression for the subset model $\operatorname{\tilde{M}}_{6}$ can be briefly examined. Even when keeping in mind that the coefficients are not directly comparable as some of the metrics have different scales, the summary shown in Fig. 14 roughly tells that the large effects are indeed pronounced during those years when the CVE coordination was delayed. Given that the majority of metrics show strong effects during these four years, the annual dummy variables seem to control relatively well the coefficient magnitudes reported in Fig. 11.

The fourth and final check is about (12). The four hundred QR-LASSO regressions estimated are summarized in Fig. 15 for $\operatorname{M}_{6}$. \textcolorblackThe upper plot indicates that even with a modest regularization such as $\lambda=20$, about a quarter of the coefficients are close to zero. Given that a week seems like a reasonably strong effect size with practical relevance, the lower plot indicates that the regularization seems to stabilize around $\lambda\leq 40$ for $\tau<0.9$. For the $\tau=0.9$ regressions, which generally gather the strongest effects (see Fig. 11), the applicable regularization seems to continue further. For the conditional median QR-LASSO regressions, only eight coefficients are larger than or equal to seven in absolute value at $\lambda=30$. These are: the annual dummy variables for 2011, 2012, 2013, and 2016, the monthly February dummy variable, WEEKEND, SOCDEG, and NVDREFS. These belong to the longitudinal control metric group, the group of social network and communication metrics, and the infrastructure metric group. None of the CVSS and CWE metrics pass this subjective threshold.

The main empirical findings can be summarized and theorized with the following five points.

1. 1.

   The first point is clear: even though nearly fifty explanatory metrics were considered, the statistical performance was relatively modest for explaining the CVE coordination delays. Less than one third of the total variation in the delays is explained by the OLS regression models estimated. The median quantile regression results largely agree. Analogous results were also obtained by classifying CVEs with a low-delay and high-delay split according to median. The adjusted coefficient of determination is not an ideal statistic to make comparisons (especially when OLS is not used), but it is still worth remarking that the bug tracking research frequently attains values around $0.5$ or even more [10, 26]. The modest performance should not be interpreted as poor performance, however. Given the typical effect sizes seen in empirical software engineering research [42], some of the regression coefficients and their standard errors indicate consistent, accurate, and large effects upon the CVE coordination delays.

2. 2.

   The second point is likewise clear: most of the explanatory power comes from metrics used to proxy longitudinal variation. This result is not surprising. In contrast, it would be surprising if a software engineering study would not reveal longitudinal effects in a period covering almost a decade. The result is also familiar from comparable studies about vulnerability coordination [88] and time series aspects of vulnerability archiving [37, 106]. If avoiding delays is important, the results can be also used to conjecture that “do not request CVEs during weekends”. The longitudinal variation implies a further important takeaway message. Given that the efficiency of software engineering coordination is always dependent also on the volume of items being coordinated, it seems that the oss-security case studied reflects the larger coordination issues that have gained publicity in recent years. By assumption, delays for CVE assignments are largely explained by the mere amount of these vulnerability identifiers requested. By implication, a good theoretical metric for prediction would simply be the amount of identifiers in an abstract CVE backlog.

3. 3.

   The third point is about noise that tends to lengthen the coordination delays. Both the social network (RQ₁) and the infrastructure (RQ₂) effects manifest themselves through different abstractions for such noise. In terms of the former, noise increases when there are “too many cooks” posting emails with high-entropy content. Both factors tend to increase also the CVE coordination delays. In the terms of the latter, noise increases with emails containing multiple hyperlinks to distinct tracking infrastructures within which the vulnerabilities have already been discussed or to which these have already been archived.

4. 4.

   The fourth point relates to prerequisite constraints. When satisfied, these constraints tend to balance the positive effect of noise upon the delays by shortening the coordination delays to some extent. For instance, traces to bug tracking systems show a small but visible negative effect. The same applies regarding traces about the information sources regularly polled by the parties involved in the CVE tracking. For developers requesting CVE identifiers for their projects, the practical takeaway message is conveyed by an advice such as: “write good bug reports to backup CVE requests”.

5. 5.

   The fifth point relates to the technical characteristics of the vulnerabilities coordinated. The social network, communication, and infrastructure metrics (RQ₁ and RQ₂) explain a few percentage points of the total variation in the delays. Given roughly comparable observations about the impact of social interactions on software quality [8], these effects seem reasonable. Although the results are somewhat mixed, the technical characteristics (RQ₃) do not generally explain the delays well. When backtracking to Fig. 2, the explanation may be simple: for participants who have assigned nearly thousand CVEs via oss-security alone, it may be irrelevant whether a request is about XSS or about buffer overflows. As there are still some signals about the statistical relevance of the CVSS and CWE metrics, another plausible theoretical explanation may be that it is mentally easier to handle vulnerabilities that fall explicitly into the scopes of some particular CWEs. Given the common problem of overloading a single vulnerability report with multiple distinct (security) issues [19], the final practical takeaway message could be that: “try to avoid ambiguities when requesting CVE identifiers”.

All in all, software vulnerability coordination can be concluded to exhibit typical characteristics of software engineering coordination in general. Dependencies, social relations, communication, and technical software elements are all present. These are also the factors that with varying degrees correlate with different coordination problems, including the CVE coordination delays observed.

Some limitations must be mentioned. According to a common taxonomy, the validity of the results reported are potentially exposed to threats to external validity, construct validity, and internal validity [114]. The discussion that follows is structured according to this taxonomy.

The limitations discussed prompt a couple of points about potential means for moving forward. First, it is sensible to recommend a closer marriage between the software vulnerability and bug tracking research domains. The intersection between the domains is large but seldom explicitly articulated. Such a marriage might also remedy the \textcolorblacksomewhat modest statistical performance reported. \textcolorblackIn this regard, a sensible conclusion is that the information used was too limited for capturing the truly relevant elements affecting the particular type of coordination observed. When looking at the bug tracking research domain, some of the frequently incorporated dimensions (such as the reputation of bug reporters and the text mining of bug reports) seem prolific for pursuing further also in the context of software vulnerabilities. These and related dimensions are relevant for studying also more contemporary questions such as those related to bug bounties.

Second, a marriage between two domains alone seems insufficient for making more practical advances. Mining of software repositories frequently provides valuable insights about software engineering coordination through case studies. Yet, as was briefly demonstrated, the practical problem is that there are hundreds (if not thousands) of relevant repositories to mine. For advances with practical relevance, affairs are required also with other computer science domains such as information retrieval and web crawling. If one is to believe the current “mega trends”, maybe some day, perhaps in a galaxy far away, also the coordination of abstract identifiers could be done by robots.