The Risk-Taking Software Engineer:
A Framed Portrait

It has long been known that how choices are presented to individuals greatly influences the decisions they make. A particularly influential paradigm in this regard has been developed in the so-called “heuristics and biases” literature. Specifically, Tversky and Kahneman introduced the idea that the “framing” of a choice, i.e., whether it is worded in terms of potential gains or losses (while remaining logically the exact same choice), has a profound implication on respondents’ level of risk-taking [5]. They studied different wordings (with the same expected outcome) and found that choices described as losses induce higher risk-taking than choices described as gains. These results have since been replicated in various studies [6, 7, 8, 9, 10].

The software engineering literature includes substantial work on heuristics and biases in general. Researchers have, for instance, found that developers are susceptible to temporal discounting [11, 12]. Other scholars found proof that developers can be substantially biased by anchoring effects [13] and that selection bias leads to project overruns [14].

Yet, there is little framing-specific research. A recent mapping study on biases in software engineering identified only three studies on framing [15]. However, they either only cursorily treat the subject or they take a much looser definition of framing, allowing for substantive differences in task descriptions (e.g., labeling desired system properties as either “requirements” or “ideas”). Further, a recent qualitative field-study on biases in software development in general mentioned framing. However, it lumped the specific effect of framing into a larger category of biases caused by superficial thinking, neglecting the fact that framing effects tend to persist even in situations where individuals fully reason through their choices [16]. In another recent qualitative study of biases and architectural technical debt, which is closely related to risk-taking, framing was mentioned as a potential influence factor, although it was the least frequently mentioned one [17].

The most closely related work to ours is probably a study of student decision-makers who had to make requirement selection decisions. They were susceptible to a framing effect and became more risk-seeking when choosing between requirements formulated in terms of cost, compared to when choosing between requirements formulated in terms of revenue [18].

Although there are many different personality models in the psychology literature, the currently dominant one is arguably the five-factor model [19, 20]. As the name suggests, it comprises five personality traits, frequently also referred to as the “Big Five”. These are openness to experience, conscientiousness, extraversion, agreeableness, and emotional stability (sometimes also referred to as its inverse, neuroticism).

Psychologists have repeatedly linked these personality traits to risk-taking, although with partially inconsistent findings. Some scholars, for instance, found that high extraversion and openness, combined with low neuroticism, agreeableness, and conscientiousness, is particularly predictive of risk-taking [21]. Other researchers found extraversion and agreeableness to be the key predictors of risk-taking [22].

Empirical software engineering also already has a rich tradition of studying the personality of people involved in software engineering [20, 23]. Scholars have, for example, used the Big Five personality framework to study the effect of developers’ personality on the likelihood of pull-request acceptance [24]. Similarly, other studies found that committers’ personality is linked to their behavior in FLOSS projects [25]. In addition, research has found that developers higher in openness to experience make more contributions to open source software projects [26]. Finally, there is extant research linking personality to programming styles [27].

At the same time, there is no research that we are aware of that explicitly attempts to link personality and risk-taking in a software engineering context.

We took inspiration from Tversky and Kahneman’s original so-called “Asian disease” problem [5], an implementation of a framing study that has been frequently used in subsequent research. To create ecological validity for our context, we adjusted the stimulus material’s wording to relate it to a common software engineering problem, i.e., project delays.

Participants were randomly assigned to one of two conditions. In both conditions, participants had to make a choice between two options. The two options were substantively the same across conditions. The conditions only differed in how these options were described, or “framed”. In the first condition, the options were framed as “gains”, i.e., participants read about their chance of recovering time. Participants in this gain condition read the following text:

Imagine that you are working on a software project with a deadline. You just realized that some requirements were implemented incorrectly, and you estimate that this will make you miss the deadline by 6 weeks. You think about potential remedies, and you come up with two options. You can only choose one.

(A) If you reduce non-essential features, you will recover 2 weeks.

(B) If you simplify the software architecture, there is a 1/3 chance that you will recover the full 6 weeks, and there is a 2/3 chance that the simplified architecture will lead to performance problems and you will not recover any time at all.

Which option do you choose?

In the second condition, the options were described in terms of “losses”, i.e., participants read about the delay with which they would finish the project. In this loss condition, the participants were given the following options:

(A) If you reduce non-essential features, you will finish with a delay of 4 weeks.

(B) If you simplify the software architecture, there is a 1/3 chance that you will finish the project with no delay at all, and there is a 2/3 chance that the simplified architecture will lead to performance problems and you will finish with a delay of 6 weeks.

After participants made their choice, they were forwarded to further screens on which they were asked for demographic and personality information. We captured programming experience by asking for respondents’ number of years of experience [28]. We employed the widely used Ten-Item Personality Measure (TIPI) to capture respondents’ personality [29]. Since this measure has been used extensively across different populations, we have no reason to doubt its suitability to assess the personality of software engineers.

We performed a power analysis using G*Power 3 [30] to avoid false positives and false negatives in the analysis of framing (RQ1) due to a potentially underpowered study. Specifically, we performed a power analysis for a $z$-test for proportions. We assume the relevant proportions of respondents choosing the risk-taking option to be 0.1 in the gain condition and 0.3 in the loss condition based on introspection and the stereotype that software developers overall might be fairly risk-averse, as well as a presumed limited strength of our stimulus material. This translates into a medium effect size of $h$ = .52 [31]. Conservatively specifying a two-tailed test, and setting desired alpha to 0.05 and desired power to 0.80, we obtain a critical $z$-value of -1.96. Further assuming an even split of participants between conditions, this implies that a sample of 124 participants is needed. Given the number of assumptions needed for a probit (or logit) power analysis, which would be needed for our analysis of personality (RQ2), and the limited empirical grounds we have to make them, we opted not to perform one.

To recruit participants, we obtained the contact information of all developers who made at least one commit to one of the 29 Apache open source projects that are part of the “Technical Debt Dataset” in version 2 [32]. We then identified all individuals listed as “authors” in the resulting data, and manually cleaned the data to remove duplicates and merge records for individuals who used different names (but the same email address) or different email addresses (but the same or an extremely similar name) for different commits. This required occasional judgment, and decisions about the identity of authors were made as conservatively as possible. In the end, we had a list of 1,555 unique individuals and one or more corresponding email addresses. To avoid excessive spam, we selected only one email address per person, preferring personal email addresses over professional email addresses to maximize the chance that the email address was still valid despite the person’s contribution(s) to the projects being potentially already several years old. We invited all 1,555 developers to participate in our survey experiment (which was part of a larger data collection effort for multiple studies).

We assured the developers that their data would be treated confidentially and not be shared with third parties, and we pledged to donate US$ 2 per completed response to the United Nations World Food Programme [33]. We sent two reminders to reach developers that were busy at the time of the initial mailing or who had started but not completed the survey [33], including a link to an official university page confirming the authenticity of the survey because some developers responded to the initial invitation, voicing concerns about it being a scam.

In total, 165 emails bounced, allowing us to reach 1,390 developers (89.4% deliverable emails). Of this group, 194 developers started the survey, and 124 completed it. Our response rate was thus 8.9%, which is in line with that of prior studies surveying developers on GitHub. Graziotin et al., for example, reported a 7% response rate and a share of 96.6% of deliverable emails [34].

Given that our ultimate number of participants surprisingly corresponds exactly our calculated sample size and the randomized assignment of participants to conditions lets us expect an approximately even distribution between them, we conclude that our experimental study is sufficiently powered.

There are several implications for research. First, we showed a framing effect for a scenario related to project delays. This raises the question for which other types of decisions or risks framing effects might exist in software engineering, and for which there might be no such effects. Similarly, our findings also raise the question if such effects are stronger or weaker for different types of roles in software development teams. In fact, one might suspect, for instance, that there could be interactive effects between task type and decision-maker role.

Second, one might wonder how to attenuate framing effects. Since the influence of framing can be considered a bias, future researchers might wish to study the effectiveness of so-called debiasing interventions in software engineers. Given that biases are mutual properties of people and tasks [40], there are two avenues for debiasing. On the one hand, one might attempt to debias individuals themselves, as has for instance been proven effective with software engineers regarding the anchoring bias [13]. On the other hand, one might study external influences as debiasing interventions. Prior research in other disciplines has, for example, found that strong warning messages may attenuate framing effects [41].

As one may challenge the accuracy of our measurements, we highlight that all of our measures are established and validated scales. At the same time, we recognize that the nature of short scales like the TIPI potentially introduces substantial noise into our measurement, which might also explain why our results are not significant with regard to personality.¹¹1Note that the TIPI is designed to capture all facets of the Big Five with content and criterion validity with one item each, making reliability measures like Cronbach’s $\alpha$ uninformative [42]. We thus do not report any.

While we contend that our experimental study has high internal validity, our analysis on personality may suffer from deficiencies. Critically, personality is of course not randomly assigned to participants, making it possible that we missed relevant control variables that would confound our results.

In addition, the number of participants is somewhat low for a regression analysis with as many predictors as we include. Our conclusions of no personality effects might thus also be driven by low sample size and therefore be overly conservative, even though others have also reported null findings regarding developer personality [26].

There is a risk that our findings may not generalize to other contexts. We studied developers involved in a limited number of large open source Java projects, with a limited response rate to our survey. Our sample is thus likely not representative of all software engineers [33]. However, we also highlight that this is possibly only a minor issue for our experimental research design, which pits one group of randomly assigned developers against another group. While these findings may thus not strictly generalize to all developers, we are nevertheless able to provide internally valid results from a sample of experienced programmers [33]. This is of course not the case for our analysis of personality, where external validity is more substantially limited.

Additionally, one might challenge whether our experiment task has external validity. For one, although this is not typically considered very problematic [43], the decision is of course hypothetical. For another, some have argued that developers make many kinds of decisions, but rarely specifically decide between two options, as they had to do in our study [44].

Since we provide the stimulus material and there is no human judgment involved in data analysis, our research should be highly replicable. All data to repeat the analyses of the experiment is provided in this article. As we explicitly promised all participants that their data would not be shared with third parties, we can unfortunately not release the personality data.