Towards Pedagogical LLMs with Supervised Fine Tuning for Computing Education

Developments in generative AI, led by the widespread commercial release of OpenAI’s large language models (LLMs) sparked a flurry of applications in computing education, promising to improve learning outcomes and reduce attrition, with a focus on AI-generated programming error message explanations (PEMs) (Taylor et al., 2024; Prather et al., 2023; Liu et al., 2024; Leinonen et al., 2023; Kimmel et al., 2024; Wang et al., 2024; Liffiton et al., 2023). Leinonen et al. (Leinonen et al., 2023) found that Codex produced novice-understandable PEMs for common Python errors. Taylor et al. (Taylor et al., 2024) found 83% of AI-generated PEMs to be accurate. Students also self-report LLM error explanations to be helpful (Liu et al., 2024).

Despite reported benefits, commercially available LLMs are aligned to be helpful assistants, which may oppose key tenets of education. Constructivism, one dominant pedagogical theory, states that learners build knowledge by doing rather than being told (Ben-Ari, 1998), and is supported by cognitive psychology literature, which states acquiring knowledge is a function of time and conscious effort (Sweller, 2023). Commercially available LLMs contradict these tenets, displaying a propensity to provide students with solutions despite being instructed otherwise (Taylor et al., 2024; Prather et al., 2024); potentially harming learning by reducing self-efficacy and grades (Padiyath et al., 2024; Dalalah and Dalalah, 2023; Denny et al., 2024).

This paper presents our process and initial findings of fine-tuning ChatGPT3.5 to improve pedagogical alignment within computing education. The design of the fine-tuned model, now deployed to over 600 programming students in a large Australian university, was guided by the following research questions:

RQ1: How effectively can university course forums contribute to fine-tuning datasets?

RQ2: How can supervised fine-tuning better align large language models with pedagogical behaviours?

The Debugging C Compiler (DCC) is an educationally-focused C compiler built upon the Clang implementation (Taylor et al., 2023). DCC automates industry-grade dynamic and static memory analysis tools such as Valgrind and AddressSanitizer, simplifying complex tools. An extension to DCC, DCC Help embeds an AI-generated error explainer system into the tool. Source code, error messages, and GDB memory stack frame information are utilised to produce better error explanations tailored to novices (Taylor et al., 2024). Built upon ChatGPT 3.5 Turbo and used over 450,000 times, it is distinct from other tools as it is abstracted and integrated into DCC itself, meaning novice students do not need to play any role in prompt construction, or passing in source code and error messages.

A proprietary dataset of high-quality question/answer pairs across five terms of our programming course forums were sourced from verified tutors during paid hours, and confirmed to be verified responses. This dataset was then used to fine-tune OpenAI’s ChatGPT 3.5 model (then state-of-the-art), and made available to students via DCC Help.

The following, iterative process was conducted to produce the final fine-tune (FT2). Steps were repeated when necessary, gradually refining the model until it met acceptance criteria.

1. Step 1

   Data Collection: Simple API to extract raw Q/A pairs from our programming course forums.

2. Step 2

   Data Cleansing: Automated filtering (URLs, proper nouns), grammatical corrections using ChatGPT 4o.

3. Step 3

   Manual filtering and quality control: Filtering for the inclusion criteria (outlined below).

Data cleansing was critical, as many responses were informal, containing grammatical issues which impacted fine-tuning. ChatGPT-4o was used to correct grammatical issues such as incorrect spelling, typos, adding capitalisation and adding code-block fences.

We evaluated an early fine-tuned model (FT1), but it suffered from prohibitive quality issues. Manual filtering in Step 3 was therefore motivated: five tutors each reviewed 500 of 2500 randomly assigned Q/A pairs, applying the following inclusion criteria: a) answers must be correct, helpful, and self-contained; b) provide suggestions rather than solutions; c) have formal tone without being dismissive; d) include code blocks as examples only; e) avoid names, specific assignments, or lab exercises; f) focus on programming language understanding, bugs, and style.

There were 528 pairs that fit all the criteria described above, constituting 21% of the dataset. While time-consuming, this process was vital in improving the dataset and resulted in a far improved fine-tune (FT2). Fifty real student questions were then evaluated with GPT-3.5, GPT-4o, and FT2. The FT2 results featured a more informal language style, in line with that of our tutors. Compared to the instructive tone of GPT-3.5, where solutions are plainly stated and sometimes given, FT2 Socratically prompts the student to consider a particular approach (Table 1). The FT2 responses are concise, while conveying similar information. Comparatively, GPT-4o responses are verbose: overwhelming the terminal environment. Future work will rigorously measure FT2 quality utilising the methodology presented in (Taylor et al., 2024).