A Qualitative Study on the Implementation Design Decisions of Developers

We recruited participants with different levels of software engineering experience, technology expertise, job titles, and engineering team sizes. The authors then released a survey (Section III-B) on their personal social media accounts for recruitment. Social media posts displayed brief descriptions about implementation design decisions, the estimated time of completion, and a survey link.

We increased sample coverage by also recruiting on Reddit. The first author advertised the survey in a text post on 10 popular developer-centric Reddit communities. As of January 2022, the subreddits had between 6,681 members to 291,170 members. Reddit communities were selected by technology (e.g., r/reactjs, r/php), geographic location (e.g., r/developersIndia), or role (e.g., r/dataengineering). Posts were only published after receiving permission from the community’s moderators, as stipulated by the institutional review board. The Reddit recruitment posts introduced implementation design decisions, provided an estimated time of completion, explained why the subreddit was a good fit, and linked the study. We also relied on snowball sampling by encouraging interview participants to share the study to others. In the survey, participants indicated if they wanted to participate in a follow-up interview.

In total, 60 developers agreed to participate in the study, with 53 who met the inclusion criteria and participated in the study. 46 participants completed the survey and 14 participants completed an interview.

We report interviewee demographics in Table I. In our study, participants represented diverse geographic locations, including the Americas ($n=27$), Africa ($n=1$), Asia ($n=7$), Europe ($n=14$), and Oceania ($n=3$). Multiple genders were also represented, such as man ($n=44$), woman ($n=7$), and non-binary ($n=1$). Participants had job titles such as junior, senior, or principal software engineer; Chief Technology Officer; software architect; machine learning engineer; and research engineer. Participants reported contributing between 1 to over 1,000 projects, with a median of 25. Participants worked in companies of varying sizes, whose engineering organization sizes ranged from 1 to 36,000, with a median of 18. Participants reported using a variety of technologies (e.g., Java, MongoDB, Angular, R, Verilog, Jupyter notebooks) and working on diverse problem domains (e.g., deployment infrastructure, IoT, static analysis tools, financial technology, healthcare, online exams).

In the survey, we first presented participants with the definition and three examples of implementation design decisions. We then asked participants to provide examples of implementation design decisions and considerations, limiting examples to the past five days in order to reduce memory bias. These questions had a response length minimum of 30 characters to encourage participants to record sufficiently detailed answers. Following best practices, we used the HCI Guidelines for Gender Equity and Inclusivity [44] to collect gender information. We allowed participants to select multiple responses for questions on gender. The full survey instrument is available in our supplemental materials [45].

Following best practices for experiments with human subjects in software engineering [46], we conducted pilots of the survey to identify and reduce confounding factors. We piloted drafts of the survey with four software developers to clarify wording and updated the survey after each round of feedback. To ensure data quality, the survey was deployed publicly, piloted, and updated on the first 15 survey responses.

Topics in the interview included implementation design decisions and considerations participants made in the past five days, as well as written explicit programming strategies on how participants made their decisions. We collected written explicit programming strategies as a structured format to capture developers’ processes and knowledge. An example of an explicit programming strategy from our study is in Figure 2; we report all collected strategies in our supplemental materials [45].

Explicit programming strategies are “human-executable procedure[s] for accomplishing a programming task” [47]. In this study, explicit programming strategies represent the process software developers used to make implementation design decisions. We collected them since developers can follow systematic processes to make some design decisions, such as selecting between multiple software architecture alternatives [27, 26]. Additionally, developer expertise and processes can also be externalized by developers via explicit programming strategies [48], which allowed us to elicit software developers’ decision-making processes in interviews.

Two authors conducted the interviews: one to execute the protocol and one to record the participant’s explicit programming strategy. Having a interviewer experienced in strategy authoring to write strategies ensured strategy writing quality, as authored strategies could be ambiguous or struggle to generalize [48]. During the interview, we reminded participants of the definition of implementation design decisions in a Google Slides presentation, using similar wording and examples from the survey. Next, we collected implementation design decisions and considerations. Finally, for as many design decisions that time allowed for, we extracted an explicit programming strategy the participant used to make their decision. The interview protocol is available in our supplemental materials [45].

To extract programming strategies, the participant explained their decision-making process step-by-step. Interviewers translated this to an explicit programming strategy in a shared Google Document to reduce the participant’s cognitive load while recalling their process, as strategy authoring is cognitively demanding [48]. The participant then reviewed the strategy and provided corrections or feedback. Because authored strategies may omit details that prevents the strategy from being usable  [48], the participants elaborated on edge cases and updated their strategy accordingly to increase its robustness. Since authored strategies’ scope may be too narrow [48], interviewees updated the strategy to be general enough for a similar problem. After reviewing the strategy, the participant edited wording and clarity so the strategy was at a quality that could be released publicly. Following prior work [49], we documented a brief description of the strategy; tools, technologies, and prior knowledge necessary to use the strategy; and the steps of the strategy. The authors updated the strategy for consistency, but participants could edit the strategy upon request. Participants also could add additional data (e.g., comments, visual media), to explain their process.

Following best practices in experiments with human subjects in software engineering [46], we piloted the interview to identify and reduce confounding factors. The first author ran the interview protocol on one author and three developers and updated the protocol based on their feedback. The purpose of the pilots was two-fold: 1) to improve the clarity of the interview protocol and 2) validate the hypothesis that developers had systematic processes to make implementation design decisions and further, could articulate their processes. We found that all pilot participants could recall and articulate strategies for making implementation design decisions.

For open coding, we followed best practices by Hammer and Berland [52], which outlines procedures on interpreting coding results and reporting coding disagreements. We treated generated codes as tabulated claims about the data that could be investigated in future work. We checked the reliability of the coding by resolving disagreements, first by discussing any disagreements and then coming to an agreement as a group. Finally, we interpreted coding disagreements as coding variance and reported the content of the disagreements.

We followed best open coding practices [53], preparing separate documents for each coder for qualitative analysis, taking care to remove the prompts from the responses. In these documents, survey responses and interview responses were stored separately and analyzed independently, as the data was collected in different contexts. We also shuffled all the rows in the data to remove any ordering effects. Finally, we removed data collected from piloting.

Open coding occurred in multiple phases. In the first phase, three authors separately reviewed the responses and inductively generated codes for each dataset. Each response was labeled with zero or more codes. Each code was given a unique identifier and a brief description. To aggregate the codes, the authors compared their separately generated codes and identified codes with similar concepts. These codes were merged under a single code and copied to a shared codebook. For the remaining codes, the authors discussed instances of disagreement and resolved them by unanimously agreeing to add or remove the code in the shared codebook. Disagreements were most frequently the result of differing scopes of codes, rather than the meaning of the participants’ statements. Some disagreements arose due to an author not coding a part of the response another author did. In the second round, the authors applied the shared codebook to the original data. If there were multiple datasets to analyze for the same research question, each dataset was coded using the aforementioned process. Then, the resulting codebooks were merged by the authors. The authors identified codes with similar definitions and added them to a final codebook with a unique identifier. The remaining unmerged codes were automatically added to the final codebook. The authors performed a third round of coding with the final merged codebook. For RQ3, the authors then applied pattern coding to the final codes to group the codes into broader categories [54]. To do this, the authors placed each code into an initial category by unanimously agreeing to put it in a category or create a new one. Then, they reviewed each category and unanimously finalized its scope and, if necessary, moved the codes between categories to reflect the new scope.

For closed coding, the first author identified a codebook to code the dataset with. For RQ4, we used Table 3 and Table 5 from Li et al. [1], which respectively contains codes on attributes of expert developers’ decision-making processes and their software and designs. The three authors deductively applied the codes to the dataset. Each instance was labeled with zero or more codes. Next, they reconvened to discuss instances of disagreement and resolved them by unanimously agreeing on which codes were to be applied. In this step, disagreements arose due to the scope of the code rather than the meaning of the statements.

For RQ1, we analyzed 82 examples of implementation design decisions from the survey and interviews. For RQ2, we analyzed 113 examples of considerations from survey and interview data as well as implementation design decisions from interviews, since participants mentioned considerations in context of their decisions. For RQ3 and RQ4, we analyzed a dataset with 99 steps from the 16 collected strategies.

In addition to extracting action codes for RQ3, we analyzed the programming strategies on the decision-making process as a sequence of action codes and categories. We did this by replacing each step of the strategy with its respective code or category. If a step had multiple codes or categories, we represented the step as a sequence of codes or categories in the order they were mentioned. If a code or category occurred consecutively, they were reduced to a single occurrence. We include these representations of strategies in the supplemental materials [45].

Participants described 15 types of actions in their strategies. We grouped these into 7 categories: defining the problem space; ideating, evaluating, prototyping, implementing, and verifying potential solutions; and updating knowledge. Many of these actions overlapped with the traditional software development process, similar to prior work [43]. Furthermore, participants’ actions were often a dialogue between the implementation and requirements. The full list of the actions in implementation design decisions are in Table IV. All codes occur in the data at least twice.

Just as how software designers follow individualized processes [43], we found that strategies about developers’ decision-making processes were unique: there were no repeated sequences of action codes or categories in the strategies. One source of dissimilarity were action codes and categories that were repeated in strategies. Table IV shows the occurrences of repeated action categories in the strategies. All action categories were repeated except ideating actions. Implementing and defining problem space actions were most repeated in participants’ strategies. Yet, there were commonalities in the structure, namely when certain types of actions occur. This is shown in Table V, which reports the median position of each action category across all action sequences.