From GPT-3 to GPT-4: On the Evolving Efficacy of LLMs to Answer Multiple-choice Questions for Programming Classes in Higher Education

As a motivation for this study, consider the following Python script that expects a number as the input:

try:    value = input("Enter a value: ")    print(int(value) / len(value))except ZeroDivisionError:    print("Very bad input...")

If a user enters 44 the output would be 22.0 (i.e., 44 / 2 cast into float as a result of applying the / operator). To get an anecdotal insight into the differences between text-davinci-001 (GPT-3), text-davinci-003 (GPT-3.5) and GPT-4 (see Section 4.1 for additional details on these models), we submitted the following queries to each model related to the provided code snippet:

1. 1.

   Explain the code snippet line by line.

2. 2.

   What is the output of the code snippet if the user enters 0.

3. 3.

   Generate the code snippet based on the provided description in natural language (see Figure 1).

Python’s try statement allows you to execute a block of code, and provides a way to handle errors that may occur.

try:    input = int(raw_input)except ZeroDivisionError:    print("Invalid input")else:    print("The result is: {}".format(input / length))

Possibly the most intriguing are the completions to the above described prompts generated by the GPT-3.5 model. As shown in Figure 1, this model still answers the MCQ about the output of the code snippet incorrectly. However, when asked to explain the code snippet line by line the model produces a detailed and correct explanation, including the line where the division takes place. For that line, it states that:

[it] takes the input value and first converts it to an integer using the int() function, then divides it by the length of the input value using the len() function.

The GPT-4 model not only produces the correct line-by-line explanation and implementation of the code snippet, it also answers the MCQ correctly. Hence, we observe a clear progression in the capabilities of the three studied models in handling the MCQ that involves the tricky piece of code (summarized in Table 1). The oldest of the three models demonstrates rather poor capabilities in handling of the code snippet and consequently fails to answer the MCQ correctly. The newer GPT-3.5 model manifests what could be considered quite a robust understanding if performed by a student, yet, it still fails to generate the correct answer. Finally, the state-of-the art GPT-4 model not only demonstrates the understanding but also produces the correct answer. This study aims to investigate the systematic differences among the three models in how successfully they handle MCQs from programming courses at the postsecondary education level.

In prior work, we evaluated the capability of various GPT models to pass a diverse set of assessment instruments, including MCQs, in the realistic context of full-fledged programming courses [36, 35, 34]. We found that the state-of-the-art GPT models are capable of confidently passing the full spectrum of assessments typically involved in a Python programming course. In this paper, we further explore the evolution of these models, focusing on discovery of more fine-grained properties of MCQs that used to be challenging for the GPT models to handle but are not anymore.

To the best of our knowledge, there are no other studies of GPT models’ handling of MCQs from programming courses. However, there are works analyzing the performance on MCQ datasets from other areas. Robinson et al. use InstructGPT [26] and Codex to answer questions from the OpenBookQA [23], StoryCloze [24], and RACE-m [16] datasets focused on multi-hop reasoning, recall, and reading comprehension. They report 77.4-89.2% accuracy [32]. Drori and Verma apply Codex to write Python programs to solve 60 computational linear algebra MCQs with 100% accuracy [10]. Hendryks et al. created a dataset of MCQs covering various STEM, humanities and arts topics. GPT-3 performs at levels above 50% for subjects such as marketing and foreign policy. It scores below 30% for topics such as formal logic [14]. Further, they observed that GPT-3 performed poorly in quantitative subjects, such as elementary mathematics. Lu et al. evaluated GPT models on a large dataset of 21,208 MCQs [20].

In computing education context, LLMs have been shown to be highly effective at generating code and explanations of the code for entry-level programmers [33]. Such explanations have even been observed to out-class student explanations of the same code [17]. Denny et al. discovered that well-structured prompts could yield correct solutions to many programming exercises [7]. Piccolo et al. demonstrated that LLMs can perform most entry-level programming tasks in the context of introductory bioinformatics course [28]. Phung et al. introduced a system that harnessed LLMs to provide precision feedback on syntax errors in students’ code [27]. Such feedback explanations went far beyond describing the code line-by-line. MacNeil et al. demonstrated that explanations of generated code can be offered at multiple different levels of abstraction [22, 21]. Finally, Doughty et al. proposed an LLM-powered pipeline to generate MCQs for programming classes [9, 1]. They showed that in many respects the generated MCQs are comparable to the human-crafted ones and well aligned with learning objectives, that can also be automatically generated [39].

In the near future, it is reasonable to expect LLMs to facilitate teacher-student exchanges similar to those that only occur in a classroom, and are invaluable to student learning [40]. Liffiton et al. described and analyzed the usage of CodeHelp [18, 38], an early example of a tool capable of such interactions. They also showed how students submit different types of questions (e.g., aimed at debugging or conceptual understanding), envisioning a system that can provide responses specifically tailored to each question type [37].

The original GPT model (i.e., GPT-1) [30] is a 12-layer decoder-only transformer [41] with masked self-attention heads. Its core capability is fine-tuning on a downstream task. The GPT-2 model [31] largely follows the details of the original GPT model with a few modifications, such as layer normalization moved to the input of each sub-block, additional layer-normalization after the first self-attention block, and a modified initialization. Compared to the original model it displays remarkable multi-task learning capabilities [31]. The third generation of GPT models (i.e., GPT-3) [6] uses almost the same architecture as GPT-2. The only difference is that it alternates dense and locally banded sparse attention patterns in the layers of the transformer. The main focus of Brown et al. was to study the dependence of performance and model size where eight differently sized models were trained (from 125 million to 175 billion parameters). The largest of the models is commonly referred to as GPT-3. The interesting property of these models is that they appear to be very strong zero- and few-shot learners. This ability appears to improve with the increasing size of the model [6]. The technical details about the GPT-4 model have not been disclosed due to concerns about potential misuses of the technology as well as highly competitive market with generative AI [25].

We focus on the evolving capabilities of the models prior to the release of ChatGPT (Nov ’22), at the time of the release, and today (i.e., Aug ’23). We employ the InstructGPT text-davinci-001 model (GPT-3) to evaluate the performance prior to the ChatGPT’s release. To understand the performance close to the date when ChatGPT was released, we use the text-davinci-003 (GPT-3.5). It builds on top of previous text-davinci-002, which in turn is based on code-davinci-002 (focused on code-completion tasks) which is sometimes referred to as codex.⁵⁵5OpenAI: Model index for researchers. Available at: https://beta.openai.com/docs/model-index-for-researchers/instructgpt-models [Accessed 2023-01-15] As of writing of this paper, GPT-4 is by far the most advanced model released by OpenAI. The model is focused on dialog between a user and a system. Hence, to gauge the rate of improvement over the several recent years, we compare the performance of GPT-4 to GPT-3.5 as well as to the previous generation’s GPT-3 on the MCQ answering task.

We set the temperature of all the models to 0.0, which corresponds to no randomness. The higher the temperature the more creative the output but it can also be less factual. As the temperature approaches 0.0, the model becomes more deterministic, which we deem as important for reproducibility. Given that existing literature does use different temperatures for testing, we did initially test a variety of temperatures, but found that setting temperature to 0.0 worked well for our context. Note that this is quite a common choice in the existing work regarding MCQs [3, 19, 20]. We set max_tokens to 500 (a token roughly corresponds to a word). This parameter controls the maximum length of the completion (i.e., the output). Note that each model has a length limit on the prompt, and the completion counts towards that limit. While GPT-4 allows for 8,192 tokens⁶⁶6There is also a variant of the model that supports up to 32,768 tokens. the GPT-3.5 can only accept up to 4,097 tokens, and GPT-3 is limited to 2,048. We set top_p to 1, as is recommended when temperature is set to 0.0. This parameter is related to temperature and also influences creativeness of the output. We set frequency_penalty to 0, which allows repetition by ensuring no penalty is applied to repetitions. Finally, we set presence_penalty to 0, ensuring no penalty is applied to tokens appearing multiple times in the output.

To test the performance of the three GPT models on the task of answering programming MCQs, we submit questions one by one using the openai Python library⁷⁷7GitHub: OpenAI Python Library. Available at: https://github.com/openai/openai-python [Accessed 2023-08-16] which is a wrapper for the OpenAI’s REST API. For GPT-3 and GPT-3.5, we embed each question in the prompt template shown in Figure 2. Since GPT-4 is a model optimized for dialogue, we use different prompts—the ones shown in Figure 3. Note that the prompt for GPT-4 is designed with the intent to prevent the model from explaining the answer to a user as we are only interested in the answer(s) themselves. Each model returns one or more of the choices as the prompt completion (response), which is then compared to the reference answer. Partially correct answers are considered to be incorrect.

For the quantitative evaluation of the models (Section 5.1), we report their performance across the individual units of each course. Furthermore, we show how the three GPT models perform with respect to the MCQ types described in Section 3. Finally, we analyze the differences in the success rates of answering the MCQs across the types between GPT-3 and GPT-3.5 as well as between GPT-3.5 and GPT-4.

For the qualitative analysis of the differences between the successive generations of the GPT models (Section 5.2), we conducted a thematic analysis across the subsets of the MCQs to which the preceding model generated a wrong answer whereas the more recent model answered correctly. During the analysis we extracted interesting features of each data point as codes. These were then collated into higher-level themes [5]. This enabled us to understand the main qualitative improvements between the two studied pairs of the models, i.e., GPT-3 vs GPT-3.5 and GPT-3.5 vs GPT-4.

Table 3 reports the results of applying the three GPT models to the MCQs from the three courses. These experiments were originally conducted in [36, 35, 34]. Overall, the GPT-3 model correctly answered 199 out of the 530 questions (37.5%). GPT-3.5 was more successful with 341 correctly answered MCQs (64.3%). The most capable GPT-4 model successfully answered 446 questions (84.1%). Therefore, we conclude that there are noticeable improvements across the successive generations of the GPT models.

Table 4 summarizes how the three GPT models perform on MCQs of various types introduced in Section 3. For all the three models, there is a noticeable difference between their performance on the MCQs with code compared to the no code MCQs. This is not surprising because the code and natural language combined together are certainly more challenging than natural language alone. It is also possible that the questions containing code snippets may tend to be more difficult than questions that do not. Interestingly, the difference seems to be much more pronounced in case of the GPT-3 (29.9% vs 53.3%) and GPT-3.5 (59.5% vs 77.9%) models as compared to GPT-4 (81.3% vs 91.3%). Therefore, we tentatively conclude that GPT-4 is much more capable in handling MCQs with code when compared to GPT-3 and GPT-3.5.

Table 5 focuses on the differences between the GPT-3 and GPT-3.5 models. Overall, GPT-3.5 fixed 192 (58%) incorrect answers provided by GPT-3. On the other hand, GPT-3.5 incorrectly answered 44 questions that were answered correctly by the older GPT-3 model. The rate of improvement appears to be higher for the no code MCQs (65% of mistakes fixed) compared to the with code questions (55.7%). The no code questions together with the other questions with code are the main contributors towards the noticeably higher performance of GPT-3.5 compared to GPT-3. For these MCQs, the GPT-3.5 model corrected 126 questions incorrectly answered by GPT-3 while only committing 20 novel mistakes. It appears that GPT-3.5 still exhibits noticeable limitations when it comes to the true/false, identify true/false and output MCQs with code. While correcting 44 of GPT-3’s mistakes it mishandled 20 questions that were answered correctly by the GPT-3 model.

Table 6 summarizes the differences between GPT-3.5 and GPT-4. Overall, the GPT-4 model fixed 125 (68.3%) mistakes committed by GPT-3.5. At the same time, it introduced only 26 novel mistakes. The rate of improvement appears to be higher for the no code MCQs (76.3% of mistakes fixed) compared to the with code questions (66.2%). However, the with code questions of the output and other types are the chief contributors towards the higher performance of GPT-4 as compared to GPT-3.5. For these MCQs, the GPT-4 model corrected 78 questions incorrectly answered by GPT-3.5 while only committing 13 novel mistakes. Overall, it appears that GPT-4 improves over the GPT-3.5 model reliably across all the question types.