Student-AI Interaction: A Case Study of CS1 students

The recent widespread deployment of generative AI programming assistants has motivated much research on the use of these tools. Researchers have conducted empirical studies (Leinonen et al., 2023; Hellas et al., 2023; Sarsa et al., 2022) to evaluate the quality of code and explanations that Generative AI programming assistants can generate. Others have explored the usefulness of Large Language Model (LLM)-based programming tools through user studies (Ziegler et al., 2022b; Vaithilingam et al., 2022; Barke et al., 2023; Kazemitabaar et al., 2023b; Xu et al., 2022; Mozannar et al., 2022; Ross et al., 2023; Srinivasa Ragavan and Alipour, 2024). Notably, Vaithilingam et al. (Vaithilingam et al., 2022) compared the user experience of GitHub Copilot with traditional auto-complete and found that programmers faced more frequent difficulties in completing tasks with Copilot, although there was no significant impact on task completion time. Barke et al. (Barke et al., 2023) took a step further to understand how programmers interact with code-generating models using a grounded theory analysis, identifying two primary modes of interaction: acceleration, where Generative AI is used to speed up code authoring in small logical units, and exploration, where Generative AI serves as a planning assistant by suggesting structure or API calls. While these studies have contributed to our understanding of how developers use Generative AI developer tools, they mainly focused on evaluating the effectiveness of such tools and did not directly address developers’ perceptions of them. Some studies, such as those by Liang et al. (Liang et al., 2023) and Ziegler et al. (Ziegler et al., 2022b), have also focused on how developers perceive these tools. For instance, Liang et al. (Liang et al., 2023) conducted a survey involving 410 developers to investigate the usability challenges associated with Generative AI programming assistants, finding that developers appreciate their autocomplete capabilities, but also reported challenges ranging from the quality of code generation to potentially infringing on intellectual property. Although these studies provide valuable insights on how users perceive the Generative AI tools, the insights are primarily applicable to professional programmers; our study focuses on the use of such tools by CS students, and more studies are needed in this area.

Recent studies have started to explore the applications of Generative AI tools in educational contexts. For example, Denny et al. (Denny et al., 2024b) have identified desirable characteristics for Generative AI teaching assistants in programming education. They emphasize the importance of features that provide immediate, engaging support while allowing students to maintain autonomy in their learning process. Liu et al. (Liu et al., 2024) investigated the integration of Generative AI tools in the CS50 course at Harvard University, demonstrating how they can significantly enhance the learning experience for students in introductory programming classes. Their approach involved developing and deploying a suite of Generative AI-based tools designed to emulate a 1:1 teacher-to-student ratio, thereby providing personalized, real-time support to students. These tools, including the CS50 Duck, a virtual rubber-duck debugging assistant, were well received by students, who reported that the Generative AI tools made them feel as if they had a personal tutor available at all times. Prather et al. (Prather et al., 2023b) have recently published a series of studies that examine the usability and interaction challenges faced by novice programmers using Generative AI tools, e.g., GitHub Copilot. Their research highlights the dual nature of such tools: while they can significantly accelerate the coding process and aid in overcoming programming blocks, they also introduce unique cognitive and metacognitive difficulties. Students often found it ”weird” how accurately Copilot predicted their needs, which led to a mix of fascination and dependency concerns. The study also identified new interaction patterns such as ”shepherding” and ”drifting,” reflecting the nuanced ways novices attempt to guide Generative AI or are misled by it. All these prior works together lend motivation to our research, highlighting the potential usefulness of Generative AI tools, while underscoring the need to balance learning support with independent problem-solving skills and over-reliance on Generative AI.

To develop Generative AI systems that balance fostering independent thinking with learning support and avoiding overreliance, it is necessary to understand how students use LLMs in practice. In the past, Kazemitabaar et al. (Kazemitabaar et al., 2023c) conducted a thematic analysis of novice learners using an LLM-based code generator in a self-paced Python course. They identified distinct coding approaches among the learners, noting that the Hybrid approach, which combined manual coding with LLM assistance, produced the best outcomes. However, the study also highlighted signs of over-reliance on LLMs, such as copying LLM output without changes, and positive self-regulation behaviors, like adding code to verify LLM output. In another study, Nguyen et al. (Nguyen et al., 2024) conducted a large-scale multi-institutional study that explored the challenges faced by near-novices when interacting with Code LLMs. Their study identified barriers such as difficulties in expressing problem understanding and using appropriate coding terminology. Prather et al. (Prather et al., 2024b) further investigated the impact of generative AI tools on novice programmers’ metacognitive awareness and problem-solving strategies, highlighting the potential widening gap between well-prepared and under-prepared students in the era of generative AI. However, there is a lack of studies elucidating when and how students use LLMs, to what effectiveness, and to what effect on their self-efficacy; this is what we cover in the present study.

Self-efficacy, or the belief in one’s capabilities to achieve a goal or an outcome, is a crucial factor in students’ learning processes. Recent studies have examined how interactions with GenAI tools influence students’ self-efficacy in programming. Tankelevitch et al. (Tankelevitch et al., 2024) discussed the metacognitive demands imposed by GenAI systems, highlighting the need for tools that support students in monitoring and controlling their learning processes. Xue et al. (Xue et al., 2024) investigated the impact of ChatGPT on introductory programming students, finding that the AI tool can enhance students’ self-efficacy by providing immediate feedback and support, thereby reducing the anxiety associated with complex problem-solving tasks. Furthermore, Denny et al. (Denny et al., 2024b) emphasized the importance of designing AI teaching assistants that not only assist with immediate problem-solving but also help students develop long-term self-efficacy by encouraging independent learning and critical thinking. This study serves to add further evidence on the effect of Generative AI use on student self-efficacy, albeit using a different approach (namely, pre- and post-task comparisons).

We recruited students from a CS1 course in a large US public university. We sent an email to the course email list, via the instructor (also an author of this paper), inviting volunteers to participate in a research study involving programming tasks with AI. Each participant was offered five bonus points in their course for their participation. We considered offering bonus points so as to not coerce students into participating. Participants could simply fill in an available time slot for the study if they were willing to participate. The course had 110 students, of which 15 students volunteered to participate.

Table 1 summarizes the demographics of the participants. Our study was gender balanced (8 females, 7 males). Two of our participants were first-generation students (i.e., no parent or guardian possessed a four-year college degree); the rest were continuing-generation students and had at least one parent or guardian with a four-year college degree. Most of our participants (13 out of 15) were freshmen; the other two were junior and post-baccalaureate respectively. The participants reported diverse racial/ethnic backgrounds, and everyone used English as their first or second language fluently.

The study was conducted in an office within the computer science department during the spring semester of 2024, with one participant per study session. The moderator conducting the study was not part of the CS1 course in any capacity (instructor, tutor, or TA).

Each study session began with a study debriefing, and the participant signing the informed consent form. The participant then filled out a pre-study questionnaire, which contained questions about: 1) demographics, 2) programming background, and 3) self-efficacy questionnaire(from (Pintrich et al., 1991)).

Then, the participants were introduced to the study environment. We chose the Python programming language and VSCode environment for the study, since it was already used in the course. Participants were also introduced to the plug-in, and were briefed on how to ensert prompts into it, should they choose to use Generative AI for their tasks. We also emphasized to participants that other Generative AI or aids (e.g., web searches) were not allowed during this time. We also clarified that their performance would not be judged by others, or affect their course grades, to ensure that the participation was non-coercive.

Participants were introduced to the programming tasks on a sheet of paper, and were given one hour to complete the tasks. On completion of the tasks, the participants completed a post-study survey containing the same self-efficacy questionnaire administered as part of the pre-study survey. The study sessions lasted between 30 and 70 minutes each.

Participants were asked to complete three programming tasks, each of which is tailored to the complexity appropriate for a CS1 level and covered topics taught by the instructor throughout the semester preceding the study.

Participants were provided with input and output examples for each task. Collectively, these three tasks addressed core programming concepts such as conditionals, loops, functions, input/output operations, and simple algorithm design, providing participants with a comprehensive understanding of essential programming principles and problem-solving approaches.

As mentioned earlier, participants had a choice of using Generative AI or not to complete these tasks. The use of Generative AI had to be via the custom plug-in we developed, and the use of any other resource (e.g., web search) was disallowed.

The goal of this study is to understand when, how, and why CS1 students interact with Generative AI tools and how effective these interactions are. Therefore, we decided to allow only the use of Generative AI for students’ help seeking to complete their tasks and disallowed all other sources.

In deciding what Generative AI tool to use for the study, we had several choices: OpenAI ChatGPT, Google Gemini, and Github Copilot, to name a few. We chose to use OpenAI ChatGPT for two reasons: 1) it provided general help beyond programming, and 2) from a prior study, it seemed to be the most familiar Generative AI among students (Amoozadeh et al., 2024).

However, in a recent study, Tankelevitch et al. (Tankelevitch et al., 2024) highlighted the metacognitive demands placed on users when interacting with Generative AI systems (e.g., the need to remember sub-tasks and which one they are on), and suggested the need to minimize switching between the task and the Generative AI help-seeking contexts. Therefore, we used a GPT-3.5 plug-in within the VSCode¹¹1https://code.visualstudio.com/ (Nam et al., 2024) programming environment, to reduce the cognitive load associated with switching between different environments, thereby allowing participants to remain within a single, cohesive coding environment.

Figure 1 shows the VSCode plugin  (Nam et al., 2024) we used for this study. As the figure shows, the plugin had a simple interface comprising of a prompt-writing box (See ”Ask a question…” in the image), allowing users to access ChatGPT within the VSCode IDE. This prompt box allows students to freely input prompts without any restrictions on the number or type of questions, or prompt lengths. The Generative AI responses to the prompts are also provided in the same pane, as the figure shows. The fact that users can access Generative AI and see its responses together with their code in the same window allows for a focused environment where users do not need to switch context to another window, and are less tempted to use other AIs.

Our study employed a multi-faceted approach to data collection, designed to capture a comprehensive view of participants’ interactions with Generative AI (here, ChatGPT) during programming tasks. We utilized a custom VSCode plugin to log all text editor interactions (including any text selections) and ChatGPT prompts, and responses. The audio and video recordings of the participants, along with recordings of the participants’ screens throughout the session provided qualitative data on problem-solving behaviors and reactions to AI assistance. Pre-study and post-study surveys collected demographic information and self-efficacy data using Likert scales. In addition, we collected detailed logs of ChatGPT conversations, including the full text of prompts and responses. This diverse dataset allowed us to analyze the frequency and timing of Generative AI use (RQ1), students’ activities and their patterns (RQ2), their Generative AI use strategies (RQ3), and the effects of Generative AI use on students’ self-efficacy (RQ4). By combining automated logging, video analysis, and self-reported measures, we ensured a foundation for both quantitative and qualitative analysis of novice programmers’ engagement with AI-assisted coding.

To answer our research questions, we conducted both qualitative and quantitative analyses of the data.

We used the logs from the plugin to analyze the frequency of Generative AI use. Of the 40 completed participant submissions, 29 solutions were created with assistance from the Generative AI plug-in provided, and 11 without Generative AI. requests, as a measure of the frequency of participants’ Generative AI use during each task. Table 2 displays this frequency of participants’ plugin-aided Generative AI use, illustrating the different levels of dependence on Generative AI assistance among participants.

However, these use frequencies did not always translate into task completion successes. Overall, among the 29 Generative AI assisted solutions created by participants, only 19 solutions ($65\%$) were correct and the rest (35%) were incorrect. Even among two participants (P2, P5) used Generative AI extensively for a task, only one instance (P5, Q3) was successful; P2 did not manage to complete Q1 correctly, even after 9 Generative AI queries. Moreover, some participants, such as P3 and P6, achieved correct answers (’C’) for one or all tasks even without relying on Generative AI, indicating that some participants may not need Generative AI to perform well.

Thus, participants’ frequency of Generative AI use alone did not lead to successes; instead, participants’ strategies in using Generative AI and their inherent problem-solving abilities both played crucial roles in their success.

To understand how and why some participants were able to use Generative AI to correctly complete tasks, and others were unsuccessful in doing so, we delved into various qualitative aspects of participants Generative AI use–when and how participants interacted with Generative AI, and how these translated into task completion successes.

With such diversity in Generative AI usage among participants–in terms of frequencies, usage patterns and strategies, and success rates– we went beyond simple task completion to metacognition, specifically to evaluate the impact of Generative AI tools on students’ self-efficacy.

For this, we analyzed the self-efficacy questionnaires we administered as part of the pre-study and post-study surveys. Unfortunately, for the first five participants (P1 to P5), we did not capture the pre-study self-efficacy data; thus, we had data from 10 participants.

Figure 3 presents the participants’ perception of self-efficacy before and after using Generative AI. Figure 3(a) shows the distribution of perception of self-efficacy before and after the study. Given the small number of observations, we cannot draw any statistically-sound conclusion about any difference between the distributions.

Figure 3(b) compares the self-efficacy of the participants before and after the experiment. It shows that P6 experienced a noticeable increase in self-efficacy, from a self-efficacy score of $4.00$ in pre-study to $4.75$ after the study, indicating a potential positive impact of using Generative AI on self-efficacy. P6 completed all three programming tasks successfully and demonstrated a mix-model behavior by independently addressing two tasks and using a hybrid approach with Generative AI for one task. The increase in self-efficacy scores suggests that employing Generative AI was effective in improving her perception of self-efficacy.

In contrast, self-efficacy in P9 decreased from $2.58$, before the study, to $1.75$ after the study. Although P9 completed all programming tasks successfully, she copied the full description to the plug-in, asked ChatGPT for the answer, and pasted the results as the submission. Similarly, this occurred for P13, who began with a pre-study self-efficacy score of $3.92$, however, the post-study mean score dropped to 3.00. He submitted an incorrect solution for one task and did not finish the other two tasks, she used Generative AI in a step-by-step approach.

In this study, our objective was to investigate how CS1 students utilize Generative AI for programming questions to assess whether Generative AI serves as a help-seeking tool for beginners in programming. As observed in other recent studies  (Kazemitabaar et al., 2023c; Prather et al., 2024b), our findings indicate that a large number of Generative AI users rely on providing full descriptions of programming questions to find solutions without making sufficient effort on their own, even under supervision. This trend aligns with the patterns of over-reliance on LLMs identified by Kazemitabaar et al. (Kazemitabaar et al., 2023c), particularly among novices using the ’AI single prompt’ approach, which resulted in lower performance on subsequent tasks. The observed behavior raises concerns about the potential overreliance on Generative AI in educational settings, where students might increasingly rely on Generative AI to provide all solutions to the detriment of their learning. This echoes the observations of Fernandez and Cornell  (Fernandez and Cornell, 2024), who emphasized the need for careful integration of AI-driven code generation tools to avoid such overreliance. Help-seeking is crucial for students to grasp new concepts, acquire skills, and tackle challenges in their computing courses (Hou et al., 2024b). However, when participants use full descriptions of programming tasks as prompts and accept complete Generative AI-generated responses, Generative AI may not effectively fulfill its role as a help-seeking tool that constructively aids struggling students, but an oracle that does the learners’ job for them. This concern was also raised by Jošt et al.(Jošt et al., 2024), highlighting the need for instructional strategies that emphasize breaking down problems and leveraging Generative AI for incremental learning. Our findings, alongside those of Prather et al. (Prather et al., 2024b), suggest that some students may struggle with new metacognitive difficulties when using generative AI tools, including being conceptually behind in course material but unaware of it due to a false sense of confidence. Further exploration is needed to understand the underlying reasons for this undesired behavior and to encourage a more constructive use of Generative AI that promotes deep understanding and problem-solving skills in computing education.

Our observations reveal two main types of behavior in problem solving: iterative and linear. Students who employed an iterative approach refined their prompts to achieve correct answers, while students who used a linear approach used the full description of the problems to find answers directly. This behavioral split reflects the findings of Vadaparty et al. (Vadaparty et al., 2024), who noted similar patterns in student interactions with Generative AI in a CS1 course. The iterative approach can improve learning by encouraging deeper engagement with problem-solving processes, whereas the linear approach may indicate a tendency to seek quick fixes.

Some prompts were related to coding concepts, indicating that novices struggle the most to understand programming concepts. This is consistent with the findings of Liu et al. (Liu et al., 2024), who found that students often used Generative AI tools to clarify the concepts of coding and the logic of the program. Our study further suggests that while Generative AI can assist with the coding concepts, additional instructional support is needed for the logic and debugging of the program.

Our results show that in 30% of prompts, users accepted the Generative AI response, while in 70%, AI responses were used to learn concepts. This dual role of Generative AI as both an answer provider and a personal tutor aligns with the observations by Prather et al. (Prather et al., 2024b) regarding the mixed impact of Generative AI on student learning, where Generative AI facilitated both understanding and dependency. Effective Generative AI integration should balance assistance with promoting independent problem-solving skills.

We can categorize the nature of interactions with Generative AI based on the reliance of a prompt on prior prompts into two groups: (1) stateless and (2) stateful. In the stateless interactions, individual prompts are independent of the prior prompts, hence prompts can be interpreted and answered independently. However, in stateful interactions, the prompt assumes that Generative AI uses the history of the user’s interaction. For instance, P7 used stateful interaction, where he asked “So, what was wrong with my initial code?”.

We observed that seven participants P4, P7, P9, P10, P13, P14, and P15 employed the same approach on all tasks. We call their strategy for problem-solving, single-approach strategy. In contrast, P1, P2, P3, P5, P6, P8, P11, and P12 used different approaches for different tasks that we call mixed-approach. As generative AI becomes more commonplace and more learners can access them for diverse sets of topics, a more granular investigation of these approaches becomes more important to guide pedagogy.

Our observations suggest that users who used full descriptions of the questions may prefer smooth interactions without challenges. This behavior correlates with lower self-efficacy scores, similar to the findings of Xue et al. (Xue et al., 2024) and Prather et al. (Prather et al., 2024b), who observed that students with lower self-efficacy tend to rely more heavily on Generative AI tools. Overall, some users’ self-efficacy levels increased after programming with Generative AI. This suggests the varied impacts of integrating Generative AI in educational contexts, shaping self-efficacy outcomes according to individual learning strategies and initial confidence levels. Enhancing self-efficacy through scaffolded Generative AI interactions could support more confident and independent problem-solving.

The current advancement in prompt engineering focuses on productivity, aiming to help developers find final solutions quickly. However, in an educational context, the goal is not merely to reach a solution, but to ensure that students achieve the learning objectives. This goal is inherently different from prompt engineering for productivity improvement, which aims to minimize and eradicate failures, a proper learning strategy may require expecting or even encouraging a significant amount of purposeful failures along the way. Thus, prompt engineering research prioritizing individual learning, as seen in works like Jin et al. (Jin et al., 2024), should be further investigated.

Our results suggest that, even in the physical presence of a researcher and with knowledge of being recorded, in a considerable number of cases, the participants directly resorted to Generative AI to provide solutions to the tasks without making any attempts on their own. This undesirable way of help-seeking, if used as the default approach in solving homework problems, can negatively impact students’ learning. This suggests the need for students to improve their self-regulation skills, e.g. self-monitoring (Schunk and Zimmerman, 2012) to monitor and reflect the intensity and frequency of their use of Generative AI. Similarly, Generative AI educational tool builders should consider supporting such strategies that enhance students’ self-regulation and help-seeking behavior when they use Generative AI.

The findings of Margulieux et al. (Margulieux et al., 2024) are particularly relevant here. In their study, they found that some students used Generative AI to support and not replace their critical thinking and problem solving. However, they also noticed that the weakest students tended to use Generative AI earlier in the problem solving process. Their findings mirror those in our study. However, Margulieux et al.were somewhat optimistic that their findings meant that users who needed help and support could receive it. Our findings, as well as those by Prather et al. (Prather et al., 2024b), show that lower-performing students using Generative AI earlier in the problem solving process is likely an indication of overreliance and a failure to properly self-regulate with these tools.

One can imagine that a possible remedy for this problem is to devise novel homework assignments that can be difficult for Generative AI to solve. However, as Generative AI tools become more sophisticated and powerful, the arms race between educators and Generative AI seems to be a losing proposition, especially in introductory courses such as CS1 that only include basic algorithmic thinking that there is an upper limit to the appropriate complexity of problems  (Prather et al., 2023a; Denny et al., 2024c). To discourage students from using prompts to find complete solutions to homework problems, we should investigate novel types of problems that are compatible with the era of Generative AI. Recent work such as Prompt Problems (Denny et al., 2024a) that uses the graphical representation for problem description instead of a textual description is a step in this direction. Even though the newest models with visual modality have been able to solve Prompt Problems (Hou et al., 2024a), they remain useful as a way to scaffold students usage of Generative AI by helping them learn problem decomposition, iterative problem solving, describing a problem, and prompt engineering (Prather et al., 2024a).

An intriguing observation from our study was that most participants did not frequently execute their programs during development to verify the correctness or identify syntax errors. This lack of regular testing suggests a gap in their understanding of the importance of iterative debugging in the programming process. Consequently, students may miss out on early detection of mistakes, which can lead to more complex issues and frustration later in the development cycle. Addressing this behavior through new pedagogical tools such as (Ferdowsi et al., 2024) that visualizes the value of variables as students develop their solutions could improve students’ coding practices and overall comprehension of programming concepts.