Table 1 presents the open and close-ended questions used to collect data from the 359 students who took one of the open source courses taught at one of the three institutions considered in this study.

First, all students from all courses were asked to answer an open-ended question about the challenges that they anticipated facing before they began their team projects. The same question was asked after their contribution attempt, to provide an account of the challenges they actually encountered.

Using the same before-after design, we administered a self-efficacy questionnaire with a five-point Likert-scale (Strongly Disagree, Disagree, Neutral, Agree, Strongly Agree) to measure the impact of the course. Self-efficacy is a measure of the confidence in the participants’ perceived ability to perform a task, which can impact one’s actual ability to complete a task (Bandura, 1986). The questionnaire applied was borrowed from Steinmacher et al. (Steinmacher et al., 2016). The items had been further classified into Social, Process, and Technical categories (Steinmacher et al., 2018), to enable a better understanding of these dimensions in the perceived experience.

To analyze the impact of the OSS course on the students’ self-efficacy, we first mapped the Likert scale answers to an ordinal (numeric) scale, from 1 to 5 (with 1 representing Strongly Disagree and 5, Strongly Agree). We kept only the entries from those students who provided answers both before and after the course (n=359).

We, then, mapped a mixed-effects logistic regression model. We used the answer provided per question as our dependent variable and the item type (social, process, or technical) and when the answer was provided (before or after the course) as fixed effects. We also modeled the participant and the item itself as random effects.

We used ANOVA to evaluate the differences between before and after answers per type. We used the estimated marginal means (EMMs) to compare among groups after fitting the model and reporting the effect size per item type.

We analyzed the open-ended answers using the Thematic analysis (Braun and Clarke, 2012) approach. Thematic analysis is a method of analyzing qualitative data that involves breaking down and coding the text into meaningful themes. It is useful for understanding how different pieces of information are related, identifying patterns in responses, and uncovering underlying meanings within a dataset. Thematic analysis provides researchers with an effective way to explore their data more deeply by allowing them to identify key ideas or concepts that can be used as the basis for further research or decision-making.

The analysis consisted of two phases, one phase to analyze answers to the question related to challenges the students anticipated before contributing, and another phase for the question about the challenges that students faced after contributing to an OSS project. Two of the authors worked individually on thematically analyzing the challenges reported by the students. The researchers analyzed each answer and derived themes according to the content of each student’s response. For simplicity’s sake, we decided to adopt the same themes and categories nomenclature from two similar studies that one of the authors conducted in the past  (Steinmacher et al., 2019; Balali et al., 2018); some of the categories did not have a correspondent, so for those we added new terms to the original nomenclature. The outcome of the analysis was a list of themes that were placed into five categories: Newcomers’ orientation, Newcomers’ characteristics, Communication, Documentation problems, and Technical hurdles.

In total, 265 students responded the before questionnaire, and 191 students answered the after questionnaire. We analyzed the data combining the three courses. The themes were generated according to the content of the students’ answers. The researchers read each response carefully, derived a list of challenges reported in each answer, and then generated themes by merging challenges. For example, one participant mentioned $-$ “Learning the syntax and language that the project uses might take awhile depending on the language and application towards that project”. From that chunk of text, we identified that the need to learn a new programming language might be a challenge when contributing to an OSS project for the first time, so we added “Learn a new programming language” to the list of potential themes. In general, the two researchers had no difficulty in reaching a consensus, as the themes each researcher identified were similar.

Analyzing the data, we identified a number of challenges students expected to find before contributing to an open source project. In total, 24% mentioned that the orientation and support they received from the community was a key factor.

Out of the five subcategories, ”Making a meaningful contribution” and ”Finding a task to start with” were the most prominent, with 12% of students indicating that not being able to make a meaningful contribution to the project could be a challenge. One of the participants said $-$ ”I think the big challenge is when I have to create a new useful feature for the project. I need to define goals and objectives”. Finding a task to start with is also an aspect that students mentioned, 8% stated that the process of picking an issue to work on could be challenging, especially because of their skill level.

After contributing to an open source project, 29% of students encountered challenges related to the orientation they received. The percentage slightly increased, but the most noticeable change is in the subcategories. Only 2% of the students stated that making meaningful contributions was challenging, but 16% mentioned that finding a task to start with was difficult. The main reason why is that they could not predict the task difficulty level. One of the participants mentioned mentioned how their team struggled to pick the right issue to work on $-$ ”My team had trouble choosing an issue because we have no idea how difficult one issue is”, another student mentioned $-$ ”Identifying the ”good first issue” for our team to take up. There were numerous issues on the [project] GitHub repository. We wanted to make sure we do not pickup [sic] too hard or too easy to solve issue as the contribution should be significant enough”.

Before making a contribution to an OSS project, 46% of the students expected that their characteristic traits could potentially be a challenge. In terms of their previous knowledge, 34% of the students believe that their lack of technical background could be an issue, especially when it comes to their lack of coding skills and knowledge of technologies and tools used in the project.

The lack of technical background proved to be a significant hurdle after their contribution. Approximately 22% of the students faced challenges regarding the technologies and tools used in the projects. The need to learn new programming languages and use tools they had never used before was the main aspect reported by students $-$ ”Before I worked on this project, I knew nothing about JavaFX, and I have not used Gradle once. It was hard for me to learn a new thing from scratch, especially JavaFX since it is a minority framework that not too many people are using it” said the student. On the other hand, the fear of students regarding their lack of coding skills did not become a reality, as only 3% of the students reported facing challenges related to this aspect.

Regarding communication barriers, only 16% of the participants expected to face any problems in this sense; most participants were expecting to face reception issues, which included receiving delayed answers and not receiving an answer from the community. After contributing, 27% of the participants encountered communication challenges; 12% of participants mentioned that they received delayed answers and 2% did not receive any answer or feedback $-$ ”Our final challenge was getting a response from the Project owner. Although we made a PR, we never got the feedback”.

The participants’ English level was also reported as an issue, as the majority of students are from non-English speaking countries $-$ ”Since most of the open Source projects are English, the language is an important question. Sometimes I can’t correctly get the idea about what should I do. I’m confused about my goal. It troubles me deeply”.

As the students were having contact with the projects for the first time, the documentation to understand the project and the code was crucial. Before contributing, only 4% of the participants were expecting to find a lack of documentation about the projects they would be working on. However, this scenario changed after contributing, when 12% of the participants mentioned that one of the challenges they faced was the lack of documentation about the project. The lack of documentation in general, comments in the code, testing guides, and setting up the environment documentation were aspects they indicated as challenging after contributing.

The category of technical hurdles emerged as the most frequently cited by the students both before and after contributing. Before contributing, 77% of the students were expecting to face technical challenges, such as challenges related to the understanding of the code structure and architecture of the project.

Approximately 30% of students expected challenges when understanding the architecture of the project and code structure $-$ ”Some source code can be very hard to read depending on the structure. The code might be separated into different files with dependencies from another file. Understand what the code is already doing can take some time before making any contribution to the code”; another student said $-$ ”Understanding the code structure could take quite a while due to either an unusual code style or the size of the project”.

Besides the architecture and code structure, 10% of the students were expecting to face challenges in understanding the code itself, especially in terms of understanding different coding styles $-$ ”The main challenge I would encounter is the steep learning curve that comes with understanding code written by other people. I find the starting point to be the most difficult in every project”. One student also pointed out how the variety of coding styles can affect the code readability $-$ ”As the community grows and a lot of developers participate, it will be challenging to establish coding standards, as each individual has their own style in coding. This might impact the readability of the code”.

The lack of knowledge about procedures and conventions was also a concern for 18% of the students before contributing. Students believe that not meeting the projects’ code standards could prevent them from having their contribution accepted $-$ ”It’s possible to receive some negative feedback from the community if my coding practice does not meet the standard”; another student also shared the same perception $-$ ”Another challenge will be adhering to the requirements of the submission such as some projects might have a certain code coverage that needs to be added or to ensure that all existing test cases pass”.

The concerns students had regarding technical hurdles became a reality, as 83% of the students faced technical challenges while making contributions to the projects. Understanding the code structure and architecture was the major challenge, being mentioned by 33% of the students $-$ ”As expected, understanding the project structure is hard. I was at a loss for how to start at the beginning”; another student reported $-$ ”Jumping on board and knowing nothing about the source code or the software architecture was quite challenging”.

Understanding the code was also a challenge for 11% of the students, similarly to before contributing. The main issue was related to the different coding styles $-$ ”The coding style was inconsistent across the files of the project. As a result, our team had to take more time trying to figure out which coding style was the most common”. The lack of knowledge about procedures and conventions also happened to be an issue for 11% of the students. According to one of them, the project demanded a strict standard, but it was not offering any information or instructions $-$ ”Abiding by coding standards/style of the open source project. We had a few issues where our pull request was not accepted due to the way we did the task”.

A single contribution experience is not sufficient. Our study revealed that a single OSS course might not be enough to improve students’ self-efficacy and ability to overcome barriers. Our findings showed that some of the challenges students anticipated turned out to be true, while others were even more difficult to overcome than expected. For instance, we observed an increase in performance anxiety, from 3% expected to 10% encountered. To address this, we recommend educators consider incorporating OSS contributions into multiple courses and/or encouraging participation in programs such as Google Summer of Code, despite scheduling constraints.

Tools and technologies are crucial for success. Our study found that students were initially worried about their lack of coding skills (13%) and knowledge of tools and technologies (12%) before starting their OSS contribution journey. However, in hindsight, they realized that knowledge of tools and technologies (22%) was a much greater issue than coding skills (3%). While coding skills are necessary, we recommend educators to incorporate tools and technologies into their curriculum. Real-world software projects, both in industry and open source, rely heavily on tools and technologies, and students’ proficiency in using them is crucial for their future.

Documentation issues are more prevalent than anticipated. One of the most striking differences in our “before” and “after” survey responses was related to documentation problems. While only 4% of the students expected such issues, they were actually encountered by 13%. The problem of inadequate or outdated documentation is a well-established issue in Software Engineering literature. Therefore, we urge educators to prepare students for the realities of software documentation, including teaching them how to write clear and comprehensive documentation and how to navigate code bases where documentation may be lacking.

Non-native speakers face difficulties with conventions, communication, and documentation. Large-scale software development is equally about communication and collaboration as it is about programming. English is the primary language of communication in most projects, and students for whom it is not their first language can struggle with understanding and adapting to common conventions and styles. Educators can support these students by providing them with templates for effective first messages to a project or pull request titles and descriptions. Encouraging them to study and learn from successful contributions by other open source contributors can serve as a guide for them to communicate effectively with contributors from diverse backgrounds.

Understanding code and code structure requires significant effort. This includes becoming familiar with coding styles, conventions, and best practices used in the project. However, the strict time frame of a university course does not align well with the flexible nature of OSS contributions. OSS projects are ongoing and can take a long time to fully understand, whereas university courses are often limited to a specific semester or term, creating a mismatch between the two environments. Educators should be aware of this mismatch and provide students with adequate time and resources to become proficient in navigating and contributing to OSS projects before they are ready to work on their first contribution.

The conclusions we make are based on data from 359 students from four instances of three courses at three universities in three countries. While we consider this to be a large sample size, we note that our findings may not necessarily generalize to other courses or student populations. Additionally, the scope of the projects that the students contributed to is relatively limited, and primarily comprises projects that the lecturers were familiar with. This may not accurately reflect the experience of students who work on projects that are not specifically advised to expect student contributions. Furthermore, the qualitative analysis component of our research introduces an element of subjectivity. To address this concern, we employed a nomenclature in line with similar studies in the field.

The survey responses may be influenced by social desirability bias, where participants provide responses that they believe are socially acceptable rather than accurate. To mitigate this threat, we emphasized to students that their answers would not affect their grades. Additionally, the measures used to assess self-efficacy may not be completely reliable or valid. We followed established practices in phrasing the self-efficacy questions in the survey.