Software Engineering User Study Recruitment on Prolific:
An Experience Report

Prolific is an online recruitment platform with over 150,000 active participants as of December 2021 (pro, 2021b). Members are encouraged to answer demographic questions, which can then be used by researchers to narrow the pool of eligible participants for their studies. However, the existing prescreening features are insufficient; questions are limited and rely on self-reporting. This self-reporting can be an issue, as Software Engineering user studies need to recruit programmers, but non-programmer participants often identify as programmers to be admitted into more studies.

We recruited 10 participants for a user study involving the completion of simple programming tasks in Node.js using a code search tool (NCQ) (Reid et al., 2021). While existing work has evaluated the accuracy of self-reported programming skill, establishing its lack of reliability (Tahaei and Vaniea, 2022) (Danilova et al., 2021), this study documents our experiences recruiting participants for user studies on Prolific. As these types of studies often require researchers to supervise sessions, non-programmer participants can have a significant negative effect, taking up valuable research time.

While a generic prescreening option for programming skills exists, to recruit programmers of a specific language, Prolific recommends (pro, 2021a) first running a custom screening survey to identify the desired demographic, then inviting them to a second study. To recruit Node.js programmers we surveyed 680 participants, using a set of programming knowledge questions derived from Danilova et al. (Danilova et al., 2021). We also used attention checks and validated the answers to prescreening questions and found that 33% of participants who completed our survey answered inconsistently, providing different answers in our survey than in their prescreening questions.

We identified 55 Node.js programmers from a pool of 206 self-reported Node.js programmers, for a total cost of £193. Our findings suggest that researchers should not solely rely on self-reported programming skill and instead take measures to verify this information.

We designed a two-part study consisting of a screening survey and video call study. All participants who complete Part 1 were paid £0.35, and £7.50 for Part 2. We ran Part 1 in multiple waves with different study sizes, first to pilot the study and then until we reached 10 successful user study sessions. Figure 1 illustrates the different steps of our participant recruitment process.

All studies on Prolific require a description and link to a survey. Researchers cannot collect any personal details, instead they must collect participant’s Prolific IDs. A completion code must be given at the end. We used Google Forms to host our surveys; the full surveys are available on GitHub.¹¹1https://github.com/Brittany-Reid/SE-Prolific-User-Study-Recruitment

We received 680 responses to our screening survey, however, there were an additional 664 participants who attempted our study and did not submit before returning their spot or timing out. Participants who returned their spot or timed out did not receive any payment. Figure 1 shows the breakdown of participants at each stage of our recruitment process. We found that 33% of participants gave inconsistent responses to prescreening validation (P1 and P2), justifying the importance of these questions. Another 24 failed attention checks and 5 were filtered for insufficient effort (nonsensical or random responses), leaving us with 428 eligible participants. 206 participants self-reported as Node.js programmers, and only 55 passed all questions. While only 24 of these participants responded to the invitation to Part 2, this may not reflect total interest; once all places filled, no more participants could respond. 10 participants returned their place without booking, while 14 booked a session. Of these 14, 10 video calls were successful, while 4 withdrew during the session. Reasons for withdrawal included lack of confidence in programming ability and anxiety programming under observation.

Of the programmer subset, 47% reported working in a software related industry, compared to just 18% in the eligible set, indicating that the programming knowledge questions did indeed identify programmers. 85% of programmers were male, in line with other studies of software engineers (Tahaei and Vaniea, 2022). Figure 2 shows the percentage of non-programmers who got each question correct; the most effective questions were K6, K7 and K5.

The results of our screening survey and user study leave us with a set of recommendations for other researchers:

• •

  Researchers should not rely on self-reported programming skill when conducting studies. We identified that only 12% of eligible participants, who self-reported having programming skills using Prolific’s prescreening questions, and only 27% of participants who self-reported as programming in Node.js, met our criteria. We recommend that researchers use other measures such as the programming questions described in our paper.

• •

  Prescreener validation is important; 33% of participants gave inconsistent responses.

• •

  Pilot studies are always important; in this case it is especially useful to gather data on the number of non-programmers to account for.

• •

  Be clear about the requirements of each study, such as the required programming skill. Participants on Prolific cannot see enabled prescreeners, so we included the need for programming skills in the description for Part 1. In an effort to reduce withdrawal rate, we also included an example task when booking Part 2, so that participants would be confident in meeting our requirements.

In summary, finding programmers on Prolific is possible with the right methods, however, the built-in mechanisms are not enough. The method presented in this paper requires a large amount of effort from both researchers and participants, and thus there is still room to improve accurate measures of programming skills on recruitment platforms.