dcc --help: Transforming the Role of the Compiler by Generating Context-Aware Error Explanations with Large Language Models

Compilers are a primary interface between a novice programming student and learning a language; however, issues interpreting and acting on compiler error messages have been well documented (Barik et al., 2014; Becker et al., 2016; Karvelas et al., 2020; Prather et al., 2017; Traver, 2010). Students have even been known to change their majors, citing cryptic and hard-to-learn compiler error messages as one of the reasons (Denny et al., 2021). Initial work has been done to enhance the readability of compiler error messages and explore the use of enhanced error messages (Becker et al., 2019b, 2016, 2018; Carvalho et al., 2021; Denny et al., 2014; Pettit et al., 2017). However, the results of this work are inconclusive, with no clear evidence in favour of enhanced compiler error messages (Becker et al., 2019a). Although there is evidence that suggests students are reading compiler error messages, it is not directly clear how many students successfully understand and act on them (Becker et al., 2016; Denny et al., 2014). One of the difficulties when enhancing error messages is the manual process involved in identifying and integrating the enhanced messages into the compiler. The hand-crafted nature of these explanations means that they fail to cover the breadth of possibilities of potential student errors.

The Debugging C Compiler (DCC) is a C/C++ compiler designed for novice programming students (Taylor et al., 2023). The tool has been used millions of times, by thousands of students, to enable a fundamentals-first introductory programming course (Taylor et al., 2023). DCC supports students by providing enhanced compiler error messages, which detect common errors that standard C implementations (such as GCC and Clang) miss. DCC achieves its goals via the following features, all incorporated into a single easy-to-use package (Taylor et al., 2023):

• •

  Additional compile- and run-time error detection: DCC embeds run-time error detection tools, such as Valgrind (Nicholas Nethercote and Julian Seward, 2007), AddressSanitizer and GDB into the generated executable to provide additional information to the DCC error explanation system including call stack printout and memory leak detection. Clang and GCC static analysis options are used to provide additional compile-time checks.

• •

  Enhanced error messages: The DCC explainer, both at compile-time and run-time, interprets and explains the most common novice error messages using simple, hand-crafted explanations for a range of common error types.

• •

  Additional context: DCC embeds source code into the executable to illustrate the location of run-time errors. This provides additional information to the student and allows more efficient bug identification and resolution.

In 1, we compile and execute a program which attempts to access an uninitialised variable. GCC does not detect the error; however, DCC flags the bug, and the location of the error.

DCC’s access to additional context at run-time including original program source, error location (line number), and the GDB call stack at the moment a run-time error occurs is critically important to generate actionable, contextual explanations at run-time.

Large Language Models, such as GPT-3 and later Codex models (Chen et al., 2021), display significant general-purpose and cross-domain capabilities in natural language processing. These transformer-based models are trained over large quantities of text scraped from the internet, and in Codex’s case, with code mined from millions of open-source repositories. Codex demonstrated state-of-the-art capabilities in code authorship via code-writing benchmark tests (such as HumanEval (Chen et al., 2021)), later underpinned the commercial GitHub Copilot.

A recent training methodology, Reinforcement Learning with Human Feedback (RLHF), can be applied to LLMs to produce models capable of following user intents (Ouyang et al., 2022). This is beneficial to override the default tendency of LLMs to simply act as a ‘smart autocomplete’, and makes it easier to have the models actually ‘follow instructions’. The premier LLM in this space is OpenAI’s ChatGPT, which was fine-tuned from GPT-3 and Codex to provide conversational-style instruction following completions. In software engineering, ChatGPT can thus be used to help translate and debug code and provide code explanations using natural language.

The adoption of ChatGPT-style LLMs within education is currently mixed, with some individuals, schools, and systems forbidding generative AIs (e.g. in Australian primary and secondary schools (Linton et al., 2023)) and others moving towards targeted and mass utilisation. The primary concern stems from the potential for wide-scale academic misconduct, but proponents of the technology argue that careful usage will unlock new pedagogical tools and strategies. Kasneci et al. provide a comprehensive survey in this area (Kasneci et al., 2023), finding that, for example, ChatGPT is already being used for educational methods such as generating tests, quizzes, and flashcards.

Research into the benefits and pitfalls of using LLMs to support novice learners in CS1 is still in its infancy (MacNeil et al., 2022; MacNeil2022AutomaticallyModels; Denny et al., 2023; Jalil et al., 2023; Becker et al., 2023; Finnie-Ansley et al., 2022). The majority of the work has focused on testing how well these tools can solve a public repository of CS1 programming problems (usually in languages other than C); and to what extent natural language modifications or prompts can lead to the generation of successful solutions. For instance, one study showed that Codex can perform better than most novice students on code writing questions in CS1 courses, scoring in the 75th percentile, and generating multiple solutions to problems (Finnie-Ansley et al., 2022). Another study found, using 31 questions from a popular software testing textbook, that ChatGPT was able to respond to 77.5% of questions and provide a correct answer in 55.6% of cases (further prompting of the tool led to a slightly higher rate of correct answers and explanations) (Jalil et al., 2023). Denny et al. found GitHub Copilot to be effective in solving standard introductory programming problems, successfully solving about half of 166 problem sets the on first attempt, and a further 60% of the remaining problems with some natural language changes to the problem specification (Denny et al., 2023). Concurrently with this work, Harvard’s CS50 has released an ‘AI chatbot’ (Dreibelbis, 2023) which aims to help students find bugs in their programs and perform Q&A over unfamiliarities or error messages. This tool is external to the compiler and implemented in their online platform, utilising a bespoke LLM instead of ChatGPT to minimise accidental over-help. We instead explore the incorporation of ChatGPT into the compiler directly, such that it may generate on-demand, novice-friendly explanations of compile- and run-time errors.

As the complexity of the problem grows, so does the reliance on human input and prompting (Austin et al., 2021). There are further concerns that tools such as Codex can lead to over-reliance on programming tasks (Chen et al., 2021), where students agree with LLM output even when it is incorrect. Some research has shown that producing explanations can reduce the over-reliance on the LLM models and improve overall decision-making (Vasconcelos et al., 2023; Bansal et al., 2021); however, issues regarding student integrity and over-reliance persist. To produce prompts capable of asking the right questions, the user must be able to understand the problem they are experiencing in the first instance, which is not often the case with a novice programming student.

Recent work by Leinonen et al. (Leinonen et al., 2023) explored the evaluation of Codex in producing enhanced programming error messages. The study selected a subset of Python error messages that were reported by students as being the least readable, and then the researchers produced code examples that would trigger these types of errors. One of the limitations of this work is the lack of an authentic classroom setting and the absence of programs written by the students themselves. The study evaluated a series of prompts designed to explain compiler errors and generate actionable fixes, and subsequently, the quality of the code fixes and error explanations generated in response to the prompts. Leinonen et al. found that error message explanations and proposed fixes require improvement before being introduced in CS1 due to students’ over-reliance and trust in the correctness of the message explanation. Still, Leinonen et al. found that plain-language explanations of errors can decrease how threatening compiler error messages appear, and could be instrumental in improving learning outcomes for students at scale.

In our case, we intend for the generative explanations to guide students to understand compiler output, including DCC output, which may be confusing or lacking in contextual details. By providing a simple AI-generated explanation in the development environment, we hope to support student progress and understanding without requiring delays until staff are available to assist.

Additional design decisions were made in order to safeguard the student learning experience. For example, dcc --help warns students that AI-generated explanations may not be correct. The tool also detects if a student is generating many dcc --help explanations in a short amount of time, warning them that they should use dcc --help sparingly, and should always understand the code they are writing—we do not make the AI help tool available in the exam environment. We present our reflections on these design choices in the discussion (section 6).

This study utilises the student-generated error/explanation occurrences, including the source code of problematic code that arose throughout the students’ regular coursework activities. We randomly sampled from the over 64,000 uses of dcc --help and extracted 200 compile-time errors and 200 run-time errors for a total evaluation set of 400 error/explanation pairs. An additional randomly selected mix of 15 error/explanation pairs was categorised jointly by the four reviewers together to reach consensus on categorisation strategies and to ensure a consistent approach.

In line with the requirements of our relevant Ethics body’s approval of this research project, data processing included anonymising the source code, including stripping potential student identifiers from comments and logs. Regular expressions were able to remove these from source code and filenames effectively, and best efforts were taken to remove names from comments (such as header-comments). Other comments, as they may describe the intended functionality of the code, were preserved—these source code comments will impact the quality of the LLM’s responses. Finally, any staff who may have generated logs when testing the tool were filtered out.

Four reviewers (three authors of this paper and one student researcher) were each asked to evaluate 100 error/explanations from the extracted set of 400 as described in subsection 4.2. Each of the author reviewers has a significant history of teaching introductory computing, and the student researcher has been teaching introductory computing for the last two years. Reviewers were instructed to assess each LLM-generated explanation (student source code with error, compiler error message, and LLM-generated contextual explanation) across the following properties:

• •

  Conceptual accuracy (Yes/No) - is the generated response conceptually correct?

• •

  Inaccuracy (Yes/No) - are there inaccuracies present?

• •

  Correctness (Yes/No) - is the provided guidance technically correct resulting in being able to solve the problem?

• •

  Relevance (Yes/No) - is the generated message relevant to the encountered error?

• •

  Completeness (Yes/No) - is the provided explanation complete, not missing any critical information that would help students understand the error?

• •

  Code Solution (Yes/No) - is the solution provided as code in the generated response?

• •

  Response type (Peer/Tutor) - is the generated response commensurate in quality with a peer or a tutor?

Prior to commencing the classification, the reviewers categorised 15 error/explanation pairs, separate from the analysis set, as a group to ensure that everyone had the same interpretation of the measured aspects. For each error/explanation pair, the reviewers could access the source code that generated the error, the output from DCC specifying the line number of where the error has occurred, the DCC enhanced explanation of the error, which also included the state of variables at the time when the error was produced (for run-time errors), and the LLM-generated explanation produced with this data. All of these were used in analysing the generated explanation by each reviewer.

Each reviewer was randomly assigned 100 errors/explanation pairs (50 compile-time, 50 run-time) for evaluation. In addition, ten percent of each reviewer’s errors were randomly allocated to all other reviewers to determine inter-rater reliability, bringing the total errors analysed by each reviewer to 130. To address limitations of percentage agreement which does not take into account reviewers agreeing by chance, Light’s Kappa (Landis and Koch, 1977) was used to determine an overall index of agreement.

We have seen overwhelming adoption of the tool amongst our students, with consistent growth in usage since its introduction shown in Figure 2. We observed significant engagement in weeks preceding a major assessment (week 4 and 6), indicating students were turning to the tool. Overall, 47% of dcc --help use occurs between the hours of 18.00 and 08.00, when teaching assistance is not readily available. This highlights a key advantage of the tool – a method for student assistance outside staff office hours.

Cursory evaluation of performance in the invigilated, closed-book final exam in which dcc --help was not available indicates no significant performance loss compared to previous terms.

Following introduction of dcc --help into our computing courses, questions and discussions naturally arose. Firstly, should dcc --help be made available to more students? Secondly, and on the flip side, does the tool in its current form provide too much assistance? While these are open questions, we acknowledge that regardless of our choices, students can access their own explanations using ChatGPT themselves. We believe that providing access to this tool outweighs the potential risks, especially as we designed the tool with limits (such as rate-limiting explanations with warnings), and removed its availability from the final exam.

dcc --help also affords us the opportunity to identify and discuss with our students limitations of generative AI tools, especially when the tools provide incorrect explanations. Ingraining a scepticism-first approach provides meaningful learning opportunities for students, ensuring that even when seeking assistance from dcc --help they should always have a clear understanding of their goals and code when debugging. In many cases, we observed that the benefits of dcc --help were the clear and friendly language of the generative explanations, particularly at compile-time – reinterpretations of cryptic error messages could lead students to an understanding of their errors and successful resolutions thereof.

Overall, the use of LLMs to generate contextual explanations for compiler errors provides a way to scaffold existing student support. The automated nature of the tool benefits the scale of learning that we face. Whilst at some stage, those scaffolds need to be removed (Sweller et al., 2019) and students need to understand compiler error messages on their own, they need not be expected to do this at the start of their learning journey – and especially not in languages such as C. All cognitive focus should be on learning the programming language, as opposed to interpreting cryptic error messages provided by the compiler at the introductory level.