Citations or dollars? Early signals of a firm’s research success

We start by defining the metrics of economic and technological value at the patent and firm level. We consider two dimensions of patents’ value: their technological value and their economic value. To quantify the technological value of a patent, we measure its number of received citations [25]. A potential shortcoming of the citation count (even when restricted to a $10$-year temporal window [7]) is its strong bias by patent age (see Fig. 2A and Fig. S1 in SI). To eliminate this bias, we compare each patent’s citation count only against the citation counts of patents issued in the same year. Hence, we define the age-normalized citation value (NCV) of a patent as its ranking position by citation count among all patents issued in the same year (see Fig. 2B and Methods for details).

To quantify the economic value of a patent, we rely on a recent measure based on the firm stock price movement over a narrow time window after the patent is issued [23]. The core idea of this metric (denoted as $\xi$) is that the market’s reaction to a patent is a combination of the dollar value of the patent and the investors’ ex-ante probability assessment on the patent’s success [23] (see Methods for details). Differently from the time-varying citations of patents, a patent’s $\xi$ is determined shortly after the patent’s issuance and does not change over time. Similarly to citation count, the economic value metric $\xi$ also exhibits strong bias by patent age, as shown in Fig. 2C. Again, to prevent this bias from influencing our firm-level results, we define the age-normalized economic value (NEV) of a patent as its ranking position by $\xi$ among all patents issued in the same year (see Fig. 2D and Methods for details).

Technological and economic value do not always coincide. A patent may represent a major technical advance, but its announcement might fail to restrict competition or attract the attention of investors, thereby generating a modest impact on the company’s stock price. For example, patent US$3728480$ from Sanders Associates (see Table 1) reported the invention of the first video game that could be played on a home television. This can be considered as a substantial technological advance compared to computer games, and the patent was highly cited, resulting in a high technological value ($NCV=0.98$). At the same time, the patent failed to capture market interest shortly after its issuance, likely because of the recession in the cable TV industry at that time¹¹1http://www.pong-story.com/sanders.htm, which resulted in a low economic value ($NEV=0.17$). We refer to Tables 1 and S1 in SI for the $NCV$ and $NEV$ of a set of expert-selected historically significant patents [31], and to Tables S2–S3 for a list of top patents by NCV and NEV, respectively.

Overall, the Pearson correlation between patents’ technological and economic value is as low as $r(NCV,NEV)=0.09$, and the correlation between the two non-normalized variables is also low ($r(c,\xi)=0.09$, see Fig. S2 in SI). To explain the discrepancy of our finding and previous claims of high positive correlation between technological and economic value [32; 25; 22], we show that such correlation increases as patents are grouped into increasingly-large sets of patents with a similar citation value (see Fig. S2 in SI). Therefore, whereas previous works demonstrated that groups of patents with higher citation impact exhibit higher economic value [22], the low correlation reported here indicates that there is little predictability of economic value from citation value at the individual patent level.

To quantify the research success of a given firm, one could average or sum the value of all its patents. However, it is well-known that patents’ quality is highly heterogeneous [33], and prior works placed a greater emphasis on a firm’s most prominent innovation than on ordinary innovations [34; 35; 36]. For this reason, we focus on a firm’s patents with the highest technological and economic value, which we refer to as its technological and economic hit [34; 37], respectively. The two hits coincide for a minority of firms, which account for $2\%$ of the analyzed firms (see Fig. S3 in SI for the correlation details), and we show below that the value of early economic and technological hits have substantially different implications for the firms’ future research success.

Based on the hits, we define the two dimensions of the innovation value of a given firm $\alpha$: its technological value ($TV$) and economic value ($EV$). We define the $TV$ and $EV$ of a given firm as the technological value of the firm’s technological hit and the economic value of its economic hit, respectively. In formulas, $TV_{\alpha}=max_{i\in P_{\alpha}}\{NCV_{i}\}$ and $EV_{\alpha}=max_{i\in P_{\alpha}}\{NEV_{i}\}$, where $P_{\alpha}$ denotes the set of patents that were granted to firm $\alpha$. Note that to simplify exposition, in the following, we refer to a “firm’s value” as a shorthand for its innovation value, i.e. the value of its patents. This should not be confused with the firm’s stock price or other measures of firm’s performance, which are not considered here.

We divide firms into three groups according to their technological value and economic value. Specifically, we consider the top-$5\%$ firms as high-value firms, the bottom-$35\%$ as low-value firms, and the intermediate $60\%$ as medium-value firms. All our results do not strongly depend on the exact choice of these separation thresholds (see Figs. S14 and S15 in SI). These three groups of firms exhibit markedly different productivity (in terms of number of issued patents) and value dynamics (see SI Fig. S4). High-value firms exhibit a sustained advantage over medium and low-value firms in terms of both productivity and value. This gap is evident even in the very early stage. Motivated by this finding, in the following, we will test whether early activity data can be used to predict firms’ future value.

We start by examining whether firms’ early value predicts future research success. To this end, we split each firm’s research lifecycle into a $5$-year initial window of early activity and a later window composed of all its remaining years. Our results are qualitatively unchanged for different choices of the initial period’s duration and the later period’s duration (see SI S3.4). A firm’s early technological (economic) value is defined as the technological (economic) value of the early technological (economic) hit (i.e., the highest-value patent among the patents issued within the initial $5$-year window). We can define firms’ subsequent technological and economic value in a similar way.

We find a strong predictability: firms among the top-$5\%$ by early economic value are $21.9$ times more likely to be among the top-$5\%$ by subsequent economic value than the other firms; firms among the top-$5\%$ by early technological value are $5.1$ times more likely to be among the top-$5\%$ by subsequent technological value than the other firms (see Fig. S5 in SI). These initial findings motivate the question: Is early technological or economic value more predictive of firms’ subsequent research success? We answer this question by first quantifying the predictive power of both variables, and then mimicking an experiment by creating treatment and control groups of firms that differ by their early technological or economic value.

To quantify the predictive power of firms’ early technological and economic value, we study a set of classification problems where we use information on firms’ early patents to predict which firms will subsequently be among the top-$5\%$ by two dimensions of future success: the technological value of the future technological hit (i.e., the highest-value patent among the patents issued in the late window), and the economic value of the future economic hit. Based on the literature, we consider various metrics of firms’ early performance that might be predictive of future success: not only the firms’ early technological and economic value, but also their early productivity (in terms of the total number of early patents) [38; 39], total citations of early patents [40; 41], and other aggregate measures of early patent value (in terms of cumulative $\xi$, NCV, and NEV). For each of these early performance metrics, we measure various predictive accuracy metrics, including precision, recall, area under the precision-recall curve [42], for a Naïve Bayes Classifier that classifies a firm as successful if and only if it is among the top-$z\%$ by the metric, where $z$ is a parameter that can be tuned to achieve a desired value of recall (see SI S3.2).

We find that a firm’s early economic value is the strongest predictor of both high-economic value firms and, more surprisingly, high-technological value firms in the future (see Fig. 3 for results and Fig. 4 for examples). By considering classifiers with $z\%=5\%$, the precision of the classifier based on early economic value reaches $51.6\%$ and $31.9\%$ for the prediction of high economic and technological value firms in the future (10.3-fold and 6.4-fold increase compared with a random classifier, respectively), as opposed to the smaller precision of the classifier based on early technological value ($2.8$-fold and $3.4$-fold increase compared to a random classifier, respectively: see Figs. 3A and B; the results based on raw accuracy metrics are shown in Fig. S6 in SI). By summing over all possible values of $z\%$, the area under the precision-recall curve (AuPRC) of the classifier based on early economic value is $4.12$ times and $1.72$ times larger than that of the classifier based on early technological value in the prediction of future high economic value firms and high technological firms, respectively (see Figs. 3C and D).

The predictive power of firms’ early economic value is substantially stronger than that of other predictors from the literature (such as early productivity and total citations), and significantly larger than that of a random classifier (see the dashed black lines in Fig. 3). These conclusions are robust with respect to alternative choices of the prediction evaluation metric (see Figs. S6 in SI) and variations in the duration of the early window (see SI Fig. S7) and subsequent window (see SI Fig. S8). Combining all the early performance metrics via a binary logistic regression model can moderately improve the predictive accuracy only for the prediction of high-technological-value firms, at the cost of increasing model complexity (see Figs. S6 in SI). For this reason, in the main text, we only show the result of single performance metrics.

Importantly, the stronger predictive power of early economic value holds as well when restricting the analysis to individual industrial sectors: by considering 10 macrosectors based on the first two digit of firms’ Standard Industrial Classification (SIC) code²²2https://siccode.com/, we find that the early economic value is the strongest predictor of future success for all $10$ industries except for the Transportation & Public Utilities sector (see SI, Fig. S13 for details). This exception might occur because in this sector, the economic value of generated research is a weaker determinant of governments’ and agencies’ investment decisions.

Two underlying mechanisms could explain the observed predictive power of the economic value of a firm’s early patents, which we refer to as competitive advantage and fitness mechanism, respectively. According to the competitive advantage mechanism, in the long-term, a firm might succeed because of the economic value she derives from her early patents. According to this interpretation, early economic success might allow firms to invest more in research, produce more patents in the future, and as a consequence, have more attempts to produce higher-value hits. This mechanism would align with the Matthew Effect found in many other systems [43]. On the other hand, according to a fitness mechanism, the value of a firm’s early hits might be a manifestation of the firm’s ability to produce successful research, which could be interpreted as the firm’s fitness. This mechanism would align with recent theories on the success dynamics of scientists that assume that a researcher’s ability to produce high-impact papers is constant over time [7; 11].

Recent works on success predictability [44] and the science of science [28; 11] pointed out that disentangling the two mechanism in observational data is challenging. A definitive answer would indeed require a randomized controlled experiment [45; 11], which is impossible in our case. Nevertheless, to move an initial step toward disentangling the two mechanisms, we implement a widely-used technique to approximate randomized experiments in quantitative social sciences: matched pair analysis [28; 45].

Specifically, we use propensity-score matching [46] (see Methods) to split pairs of same-industry firms (according to the 10 SIC macro-sectors) that are similar in terms of their early productivity and technological value, but differ significantly in terms of their early economic value (see Fig. 5A), which leads to two groups of firms. We find that firms in the “treatment” group with a larger early economic value exhibit a significant advantage in terms of number of issued patents ($+195\%,P<0.01$), technological value ($+6\%,P<0.01$) and economic value ($+60\%,P<0.001$) in the late window (see Fig. 5A and Table S5). To verify whether a similar result would hold for early technological value, we perform a complementary experiment where the same-category firms’ pairs to be split into two groups exhibit similar early productivity and economic value, but significantly different early technological value. In this scenario where only early technological value differentiates the two groups of firms, the difference in late technological value is significant ($+7\%,P<0.01$), but that in late economic value is not ($+10\%,P>0.05$, see SI, Table S5).

These results might be naively interpreted as evidence in favor of the competitive advantage mechanism. But if firm’s late success is entirely due to increased late productivity, success differences among the treated and controlled firms would disappear if the late number of issued patents is added to the covariates in the matching procedure. However, we find that this is not the case: When adding the late productivity to the covariates, treated firms still exhibit a significant advantage over controlled firms in terms of technological value ($+3\%,P<0.01$) and economic value ($+35\%,P<0.001$) in the subsequent years (see Fig. 5B and Table S6 in SI). This finding rules out the possibility that firms with early high economic value succeed in the future merely because of increased late productivity. Taken together, these findings indicate that the competitive advantage mechanism alone cannot explain the observed predictive power, and the fitness mechanism may play a main role in the research success of firms.

The observed predictability relies on the hypothesis that early top-firms are more likely to be among the high-value ones in the future. We refer to firms with early high economic value that maintain high economic value in the subsequent years as predictable firms. Despite the high precision of the resulting classifiers, there exist $2.3\%$ of firms that are not initially among the top-performing ones (i.e., top-$5\%$ by early economic value) and later end up among the top-$5\%$ (see Fig. 6A). These “low-to-high value” firms, which we refer to as hidden gems, are reminiscent of sleeping beauty papers in science [47]: They are only able to be granted an economic hit after a relatively long time after their first patent issuance. Here, we aim to quantify the early detectability of the set of hidden gem firms that transition from medium or low value to high value.

Both predicable firms and hidden gems exhibit high late economic value. Besides, we refer to $92.7\%$ of firms that never reach high economic value as non-top firms; to $2.3\%$ of firms that start from high economic value and descend to a lower value level as declining firms (see Table 2 for the classification of firms). We show the heterogeneous economic-value trajectories of 14 well-known firms in Fig. 6A. Among them, Microsoft, General Electric, AT&T, eBay and Apple maintained a high value (predictable firms according to our definition), while Amazon fell from high to medium value (declining firm). By contrast, Intel, IBM, and HP went up from medium to high value, and Applied Materials rose from low to high value; these four firms are hidden gems according to our definition.

Applied Materials is an outstanding example of hidden gems. The firm was unable to produce high economic-value patents within its earliest $5$ years of patenting activity, although it was granted high-technological value patents in the early stage. After 1982, its economic value exhibited a steady growth, and subsequently, the firm became able to produce high economic-value patents (see Fig. 6B and Fig. S9 in SI for more examples). This transition is reflected in the company’s history. Applied Materials went public in 1972. In the subsequent few years, the company followed a diversified business strategy. During this period, its technological value was high, while its economic value was low. In 1976, it changed CEO and refocused to its core business of semiconductor manufacturing equipment³³3https://en.wikipedia.org/wiki/Applied_Materials. After that, its economic value rapidly increased, whereas its technological value stayed at a high level. At the time of writing, the company is a global leader in its core industry.

The existence of hidden gems raises the question: Are they predictable? The Applied Materials example suggests that early high technological value might predict transitions from low or medium early economic value to high late economic value. We confirm this conjecture in two ways. First, we compare the early technological value for four groups of firms with distinct economic value dynamics (see SI, Fig. S10). We find the average $TV(5)$ for hidden gem firms is 0.957 (s.e.m. 0.009), which is markedly larger than that for declining firms (0.939 (s.e.m. 0.022)), non-top firms (0.876 (s.e.m. 0.003)), and even slightly larger than predictable firms (0.954 (s.e.m. 0.016)). Subsequently, we perform a matched pair analysis in which we only consider firms with non-top early economic value, and the early technological value is used to split pairs of firms among a treatment and control group. Among same-industry firms with similar non-top early economic value and early productivity, those with high early technological value exhibit $10\%$–higher late economic and technological value than those do not (see Fig. 6C and SI, Table S7). These findings indicate that among firms with non-top early economic value, an early advantage in early technological value translates into a late advantage in terms of economic value.

We further evaluate our ability to early detect the hidden gems via their early economic and technological value. To this end, we measure firms’ economic and technological value within the earliest $5$ years, and we evaluate the predictive performance of a Naïve Bayes classifier that classifies a firm as a hidden gem if and only if it is among the top-$z\%$ by a given metric, where $z$ is a parameter that can be tuned to achieve a desired value of recall. We consider various performance metrics, including early productivity, $N(5)$, early economic value, $EV(5)$, early technological value, $TV(5)$, and the sum of early economic and technological value, $EV(5)+TV(5)$.

We find that the $EV(5)$ alone achieves a $5.1$ fold increase in AuPRC compared to a random classifier (see Fig. 6D). This signals that, unsurprisingly, firms that are nearer the top threshold in early stage are more likely to transition to high value. More interestingly, the $TV(5)$ alone achieves a $2.5$ fold increase in AuPRC compared to a random classifier, and a combination of the $EV(5)$ and $TV(5)$ achieves the most accurate predictions, leading to a $5.8$ fold increase in AuPRC compared to a random classifier (see Fig. 6D and Fig. S10 in SI for shortening the duration of early window), which confirms the key role of early technological value in the transition to high economic value.

The observed predictability of firms’ future hits from early patents motivates us to study the temporal dynamics of firms’ patent value. Do firms tend to be granted their hits at the beginning of their research lifecycles? Or are firms’ highest-value patents randomly distributed along a firm’s lifecycle, similarly to the highest-impact works for scientists, artists, and musicians [7; 8; 30]? How do these patterns differ for predictable and hidden gem firms? We find that differently from results for individuals’ creative works [7; 8; 30], the temporal position of a firm’s hits is markedly non-random.

Specifically, we study the distributions $P(N_{C}^{*}/N)$ and $P(N_{E}^{*}/N)$ of the relative position of a firm’s technological hit ($N^{*}_{C}$) and economic hit ($N^{*}_{E}$), respectively, compared to the firm’s total number of issued patents, $N$ [7; 30]. Both types of hits are significantly more likely to occur among earliest patents than expected by chance, which is demonstrated by the left peaks of the two distributions (Figs. 7A and B). The observed peaks cannot be explained by randomized patenting histories where for each firm, patents’ value scores are randomized, while the total number of patents is preserved (see the shadowed area in Figs. 7A and B) [7]. At the same time, whereas the probability to achieve the technological hit steadily decreases as a firm is granted more patents (Fig. 7A), the probability to achieve the economic hit exhibits a second peak around the end of the lifecycle (at $N_{E}^{*}/N\sim 1$, see Fig. 7B). These results hold not only when considering all firms together, but also when considering separately high-value (top 5% by their hits’ value), medium-value (middle 60 % by their hits’ value), and low-value firms (bottom 35% by their hits’ value), and when considering firms from different industries, see Figs. S11 and S12 in SI.

The heterogeneity of firms’ hit position is well-illustrated by a few key case studies in Fig. 7C (see Table S4 in SI for details). IBM achieved its economic hit (about an integrated circuit with dielectric insulation) in 1976, whereas it achieved its technological hit significantly later (in 2002) with a patent on controlling access to shared storage devices. On the other hand, the Apple’s technological hit appeared in 1992 (on a powered manager for a portable laptop computer), whereas its economic hit was issued substantially later (in 2006, on an improved method for generating multimedia non-linear effects).

The different behavior of $P(N_{C}^{*}/N)$ and $P(N_{E}^{*}/N)$ is a reflection of firms’ heterogeneous value dynamics, which is linked to the predictive problem studied above. To demonstrate this point, we consider the previously-defined four groups of firms: predictable, hidden gem, non-top, and declining firms. For the four groups of firms, we find that the average technological value of their patents tends to steadily decrease over time (Fig. 7D), which matches the higher probability of early appearance of technological hits. The only exception is the group of hidden gem firms, which exhibits a stabler trend. This suggests that the hidden gems’ innovation ability does not diminish as they mature, which could be the key for their later transition. By contrast, the dynamics of average economic value exhibits heterogeneous patterns. Whereas the average economic value of predictable and declining firms’ patents tends to remain stable or decrease over the firms’ lifecycles, the economic value of hidden gem and non-top firms sharply increases over time (Fig. 7E). This different behavior is reflected in the behavior of $P(N_{E}^{*}/N)$: predictable and declining firms only contribute to the early peak, whereas hidden gem and non-top firms only contribute to the late peak (Fig. 7F).

The observed early peak of $P(N_{C}^{*}/N)$ supports previous studies which claimed that newcomer firms are more likely to produce innovations of high technical quality [48; 49]. This is because as firm age, they might gradually refine their innovation competence and organizational routines [50]; in this phase, benefits from new technological advances might reduce [49]. Hence, inventions by experienced firms are more likely to be the extension and improvement of their established innovative domains and technologies [50]. Based on our previous results, we conjecture that the second economic peak of hidden gem firms might be due to the increasing ability of technologically-competitive firms to attract interest from the market. In some cases, like Applied Materials, this might be due to organizational transformations. In other cases, the late peak might be due to factors that have been associated with late success in innovation research, including time-consuming knowledge acquisition [51], experience [52], and reputation [53].

Taken together, these findings indicate that the firms’ hits are not uniformly distributed along the firms’ research lifecycles, which markedly differs from previous findings on the timing of success for scientists [7], artists [30], and musicians [30]. This discrepancy indicates that previously-identified mechanisms (such as the $Q$-model [7]) are unable to explain the value dynamics in firms’ research lifecycles, which calls for new modeling approaches.

We analyze the $2,458,402$ patents granted to $7,440$ firms by the United States Patent and Trademark Office from $1926-2017$ [24]. The average number of patents per firm is $330.6$, and the largest number is $123,220$ (granted to IBM). For each patent, the dataset includes an ID, date of filling and grant date (we employ the latter date), IDs of the applicant firms, and the list of its cited patents. Note that this dataset only includes patents whose assignee has been matched to a firm in CRSP (Center for Research in Security Prices), so that each patent’s applicant is a firm listed in the US stock market. All other patents are not included in the original data.

A potential issue when measuring the number of citations received by a patent is that it might be unreliable for patents issued near the end of the data. To prevent this issue, we make a conservative choice and limit our analysis to patents issued up to $2006$ (and their applicant firms). In this way, patents’ citation counts are measured over a time-window of at least 10 years, thereby avoiding short-term fluctuations (results stay qualitatively the same if we consider patents issued up to $2011$). At the same time, we are interested in firms with a sufficiently productive research activity. For this reason, in the main text, we limit the firm-level analysis to firms that have at least $15$ patents, which includes $2,819$ firms. In the SI, we show that our main results are qualitatively the same when filtering the firms based on their number of years of activity, see Fig. S16 in SI.

Matched pair analysis is a form of analysis in which each of the subjects in a treatment group is paired with each of those in a control group on the basis of matching covariates. This technique is widely used in medical and social research to evaluate the effect of a treatment, with the ease of implementation and comprehension. We obtain such pairs via Propensity Score Matching [46], where the propensity score is defined as the probability of treatment assignment conditional on baseline covariates. We implement the matching by a Python package available at http://www.kellieottoboni.com/pscore_match/ (with minor changes to support one-to-one matching).

Take Fig. 5A as an example to explain the matching process. Firstly, we calculate propensity scores for each analyzed firm by applying a logistic regression where the covariates are early productivity and technological value and the dependent variable is whether the firm has top-5% economic value in the early stage (1 if yes, 0 otherwise). Then each firm with top-5% early economic value will be tried to match with one firm with non-top early economic value according to their propensity score, at the same time, we require the matching pairs to have identical SIC industry (the major 10 sectors). Note that we use one-to-one match so that each firm with top early economic value will be matched with at most one firm with non-top early economic value. If the match succeeds (i.e. the two firms have close enough propensity score), the firm with top-5% early economic value will be assigned to the treatment group, the other firm will be assigned to the control group. By going through all firms that have top-5% economic value in the early time, we construct the treatment group and control group, and compare the subsequent performance of firms in the two groups.