Fine-tuning ChatGPT for Automatic Scoring of Written Scientific Explanations in Chinese

LLMs like ChatGPT have shown great potential because they can handle many different uses in general. However, it is important to fine-tune these models using specialized multi-lingual (e.g., Chinese) datasets for specific areas like education. This is especially true when computers automatically check how well students answer questions. In this study, we explain how to finetune ChatGPT to score complex written responses of students. Our dataset (see Table 2) consists of student-written responses and human-expert grading for holistic and analytical classes with 5 scoring perspectives. Before fine-tuning the ChatGPT using OpenAI service, it is crucial to process the data based on the OpenAI guidelines¹¹1https://platform.openai.com/docs/guides/fine-tuning/preparing-your-dataset as follows:

Data Cleaning and Privacy Assurance: The first stage was going over the dataset in great detail to find and eliminate any information that wasn’t directly related to the study’s goals. More crucially, this step was essential to protecting the pupils’ privacy and confidentiality. All personally identifying information, including names and unique identifiers, was painstakingly deleted. This action was taken to preserve ethical norms, protect the privacy of each individual whose data was used in this study, and ensure that the data was clean.

Tokenization and Language Adaptation: The students’ textual responses had to be carefully converted into a format that the model could understand, as the language structures of Chinese are somewhat complicated. This was a complex process of translating the rich subtleties of the Chinese language into a digital format that preserves the breadth and context of the student’s remarks rather than just a technical tokenization approach. Special care was taken to ensure that the intricacies inherent in the Chinese language, such as idioms and implicit reasoning, were not overlooked during the tokenization process.

Data Standardization and Uploading: The dataset was subjected to a standardization process following tokenization and cleaning. This puts the data in an OpenAI platform-compatible structured format, usually JSON. Then, using the file upload API²²2https://platform.openai.com/docs/api-reference/files, the standardized files were safely uploaded to the OpenAI server. This was an important step since it took our carefully prepared dataset and put it into the AI processing domain, which started the process of turning raw data into meaningful conclusions.

Anonymization for Ethical Compliance: The data was further anonymized after identifying information was eliminated. Random codes were used in place of any possible identifiers during this procedure. This step maintained a high quality of ethical research practice by ensuring that even if someone were to acquire the data, it would be impossible to relate it back to specific students.

Quality Checks and Balancing: To ensure the dataset was representative and intact, it went through several quality checks. This involved distributing the dataset among various student answer durations and reasoning complexity levels. The goal was to produce a clean, well-structured dataset that accurately represented various student skills and cognitive processes.

Given the unique challenges of scoring scientific explanations in Chinese, fine-tuning ChatGPT was conducted with specific considerations:

Loss Function: A regression-based loss function was used for continuous scores and categorical scores, and a classification loss was used. The intricate organization of scientific explanations, where scores frequently fall into complex categories or necessitate a scale that indicates differing degrees of accuracy and completeness, impacted this decision.

Learning Rate: A cautiously low learning rate was our starting point. Because Chinese has distinct syntactic and semantic features that set it apart from the primarily English corpus, the model was initially trained on, this was important. Periodic assessments implemented gradual modifications in the learning rate to provide the best possible adaptation while retaining the model’s original strengths.

Epochs: It was vital to use multiple epochs, particularly given the complexity of scientific reasoning in the student explanations. Because reasoning in Chinese texts is implicit and context-dependent, it was important to control validation loss to avoid overfitting carefully and instead learn sophisticated reasoning patterns.

Batch Size: To meet the computational requirements of processing Chinese characters and the intricate structures of scientific explanations, the batch size was optimized. The selected size allowed for effective processing without sacrificing the caliber of the training.

Data Augmentation: More data augmentation methods were applied to address the problem of multi-step reasoning in Chinese texts. To improve the model’s comprehension and scoring of such responses, artificially generated samples that imitate complicated reasoning processes were incorporated.

Post-fine-tuning, the model was evaluated and validated using a set distinct from the training data. The evaluation involved:

Metrics: We used metrics such as Mean Absolute Error (MAE) or classification accuracy to gauge the model’s scoring precision against human raters. This step is essential to assess the effectiveness of the fine-tuning process.

Comparison with Baseline: The performance of the fine-tuned model was compared with the original GPT-3.5-turbo model. This comparison helps quantify the benefits of domain-specific fine-tuning and understanding the improvements in the model’s ability to score student responses.

The fine-tuning process was meticulously aligned with the research objectives and the questions raised in the introduction:

Scoring Accuracy: The fine-tuning procedure was designed to consider the subtleties of scientific language and reasoning in student writings to answer the first research question about the model’s accuracy in evaluating written scientific explanations in Chinese. The model’s performance was continuously assessed against various explanations to guarantee high accuracy. Scoring accuracy refers to the agreement between machine scoring and human scoring[53]. For example, an accuracy of 80% indicates that the machine scores align with human scores for 80% of the student samples.

Reasoning Complexity: Special focus was paid to fine-tuning the model to distinguish different levels of reasoning difficulty in response to the second and third questions regarding the effect of reasoning complexity on scoring accuracy. As part of this, the model was trained to recognize implicit reasoning processes and context-dependent explanations prevalent in Chinese texts.

Correlation analysis is a statistical method used to evaluate the strength and direction of the relationship between two or more variables. Kendall correlation analysis is a technique suited for determining significant associations between two ranked variables [1]. A Kendall correlation coefficient measures the strength of the relationship between two variables, ranging from -1 to 1. Table 4 shows a conventional approach to interpreting a correlation coefficient. The correlation coefficient’s absolute value closer to 1 indicates a stronger linear relationship between the variables. An absolute value close to 0 suggests negligible correlation. Specifically, a coefficient less than 0.39 indicates a weak correlation; from 0.40 to 0.69, a moderate correlation; from 0.70 to 0.89, a strong correlation; and from 0.90 to 1.00, a very strong correlation [39]. This study used Kendall correlation analysis to understand the extent to which the reasoning complexity of student writing in Chinese correlates with the scoring accuracy of Fine-tuned ChatGPT.

Firstly, we coded the reasoning complexity of students’ responses using the criteria adapted from [7, 21]. Responses with low reasoning complexity mainly involve generalizing, summarizing, or simple causal reasoning about entities or elements perceptible by the senses. Medium reasoning complexity responses include a few variables elements, or processes beyond the sensory experience, attempting to express causality, nonetheless, not yet at a level that uses the parts of a theory to represent causal processes or ongoing mechanisms. Responses with high reasoning complexity engage with theories or abstract concepts to explain non-visible or non-perceptible elements, processes, or mechanisms, reasoning at this level is at a more sophisticated stage than in the previous levels. It can clarify the covariation relationship among multiple variables, and apply hypothesis deduction, conservation thinking, proportional reasoning, variable control, probability reasoning, and correlation reasoning to construct causal chains or explain running mechanisms. Table 5 shows the criteria for reasoning complexity and Table 6 outlines the distribution of students’ responses across various reasoning complexity levels in seven items.

Secondly, we chose the Kendall correlation analysis to assess the correlation between the reasoning complexity of student writing in Chinese and the scoring accuracy of Fine-tuned ChatGPT. We controlled for students’ explanation performance during this process, thereby isolating this confounding variable. To control students’ performance, we created two performance groups: those scoring 0 were placed in the lower-level group, and those scoring 1 in the higher-level group. Then, we constructed contingency tables to represent the observed frequencies of each combination of reasoning complexity and scoring accuracy categories. Then, we calculated the number of concordant pairs(n_c) and the number of discordant pairs (n_d), the number of samples (n), and the minimum value between rows and columns. Following this, we calculated Kendall’s Tau-c using the corresponding formula:

Take Holistic scoring in item 1 as an example. Out of 242 testing samples, 97 scored 0, forming the lower-level group, and 145 scored 1, categorized as the higher-level group. Within the lower-level group, 79 out of 97 samples were with human-machine consistency, 18 samples with inconsistency, 24 out of 97 samples in the lower-level group were coded into low reasoning complexity, 41 samples were coded into medium reasoning complexity, and 32 samples were coded into high reasoning complexity, as detailed in Table 7. At last, we concluded the Kendall correlation coefficient, τ(97)=-0.32 (p$<$0.05), indicating a moderate negative correlation between scoring accuracy and reasoning complexity.

Although the classification process of automatic scoring is usually not transparent, the scoring is based on all information provided. In this study, the input information came from students’ writing responses. Theoretically, all response characteristics could be factors that account for automatic scoring accuracy. Thus, to answer Research Question 3, we employed a qualitative manual matching approach to identify characteristics that might account for scoring differences. We analyzed both machine-misscored responses and correctly scored responses, focusing on their distinct languish characteristics following three steps, seeing Figure 2.

The first step was categorization, during which researchers segregated the higher-level group and lower-level group into four subgroups based on scoring accuracy: “Higher level and Misscored subgroup (HM subgroup),” “Higher level and Correctly scored subgroup (HC subgroup),” “Lower level and Misscored subgroup (LM subgroup),” and “Lower level and Correctly scored subgroup (LC subgroup).” We then selected all student responses from items where reasoning complexity significantly affected scoring accuracy and categorized them into four subgroups. The second step was discerning characteristics, in which two researchers reviewed all categorized responses to identify linguistic characteristics at both word and sentence levels. According to an analytical framework from [34], the researchers determined the linguistic features such as Technical terms, Frequency of words, Polysemous words, Pronouns, Synonyms/Misspellings, and Symbols at the word level, and Syntax, Noun phrases, Verb structures, Number of sentences, Passive constructions, Prepositional phrases, Sentence length, Sentence structure at the sentence level. The third step was a comparison, where researchers compared the linguistic features between the HM and the HC subgroups, as well as between the LM and the LC subgroups. This cross-subgroup comparison aimed to account for discrepancies scored by Fine-tuned ChatGPT. This approach enabled the identification of key linguistic characteristics in student writings that may affect scoring accuracy. Given the diversity of student responses and the large dataset, the matching process was challenging and time-consuming. The findings are also deemed exploratory.

The lower-level group received 0 for their responses, the correlation between reasoning complexity and scoring accuracy for the lower-level group responses is shown in Table 8, we calculated a total of 30 Kendall correlations to explore the relationships between reasoning complexity and scoring accuracy across different scoring elements for seven items. Among these correlations, 29 correlations were negative, with 28 being statistically significant (p $<$0.05 or p $<$0.1). The average correlation coefficient across all items and scores was -0.31 (SD = 0.17). These results indicate that for lower-level performing students, the higher the complexity of reasoning in their explanations, the lower the scoring accuracy yielded by ChatGPT models.

The higher-level group received 1 for their responses, we conducted the same total of 30 Kendall correlation analyses to assess the relationship between reasoning complexity and scoring accuracy for the higher-level group responses, Table 9 shows the results. Among these, 26 out of 30 correlations were positive, with 10 being significant (p $<$0.05 or p $<$0.1). The average value of these significant correlation coefficients was 0.19 (SD =0.11 ). These results indicate that for higher-level students, the higher the reasoning complexity is, the higher of scoring accuracy yielded by ChatGPT.

Overall, responses from the lower-level group received 0 for their responses and demonstrated a low frequency of misspellings, polysemous words, synonyms, and pronouns, tend to use academic terms, and describe phenomena directly at the word level, rarely use passive constructions at the sentence level. Additionally, we also obtain some distinct patterns between the correctly scored subgroup and the misscored subgroup.

The lower level and correctly scored subgroup. At the word level, the written responses from the Lower level and Correctly scored subgroup (LC subgroup) used fewer technical terms, with a higher proportion of everyday language, and averaged only 20 Chinese characters, significantly shorter than the 40 Chinese characters found in the Lower level and Misscored subgroup (LM subgroup). At the sentence level, the LC subgroup exhibited several distinct characteristics compared to the LM subgroup:(1) they contain fewer and shorter sentences, typically with simple structures like subject-verb-object; (2) the syntax was relatively straightforward, using more simple and compound sentences; (3) noun and verb phrases were more straightforward, and prepositional phrases were less frequent, making the descriptions more comprehensible.

Take a typical LC subgroup response in item 7 as an example. The response was “There are gaps between molecules that allow for transmission (No. 194).” This response only describes one phenomenon and fails to incorporate theories related to molecular thermal motion. Therefore, it receives a human score of 0 in the Theory element. It acknowledges the interaction between molecules at the microscopic level but lacks reasoning chains to show why the gaps between molecules lead to transmission, resulting in a lower reasoning complexity.

Moreover, this response is short, containing only ten Chinese characters, and uses everyday terms like “gaps” and “transmission”. Its structure is simple, with just one subject-verb-object sentence, and lacks complex clauses or modifiers. The phrases, such as “gaps between molecules” were straightforward and clear, without complex technical terms or multilayered modifications. This direct and easily understandable structure and terms aided in conveying the information more clearly. Due to the simplicity and directness, the machine scoring system can more easily parse this type of response. The simple sentence structure reduces the likelihood of parsing errors and makes information extraction straightforward and efficient. This concise and clear expression style helps the machine algorithm accurately evaluate the student’s response, allowing the machine scoring to achieve high accuracy in this context, as the scoring criteria align more easily with the straightforward information provided by the student.

The lower level and misscored subgroup. Unlike the LC subgroup, the written responses by the Lower level and Misscored subgroup (LM subgroup) tend to be longer, averaging about 40 Chinese characters. These responses frequently use technical terms and professional vocabulary but not always accurately, especially when describing physical processes. At the sentence level, LM subgroup responses typically contain multiple sentences, providing detailed explanations. Their sentence structures are complex, with varying lengths and multiple modifying elements. They frequently used complex syntactic structures, including multiple clauses and relative clauses. Additionally, these responses feature complex and technical noun phrases and verb phrases, and they use prepositional phrases frequently, particularly when describing physical locations and states. These linguistic characteristics often result in higher reasoning complexity. However, they may also complicate the understanding and parsing of these responses, potentially challenging the scoring accuracy of ChatGPT.

For instance, the response to item 7 is “The principle of diffusion is that molecules are constantly in random motion. The higher the temperature, the greater the internal energy, the internal energy is converted into kinetic energy and released, causing vigorous molecular motion, increased gaps, reduced forces, and greater diffusion (No. 59).” The response displays a misconception– “internal energy is converted into kinetic energy”, which led to a score of 0 in the Theory element. However, it surpasses sensory experience by using a microscopic molecular thermal motion model and forming an evidence-to-theory-to-phenomenon reasoning chain, indicating a high reasoning complexity level.

This response exceeds 60 Chinese characters and uses numerous technical terms such as “kinetic energy” “internal energy” and “force”. Regarding sentence structure, the response contains a long sentence segmented by commas into various clauses and phrases and contains information equivalent to multiple shorter sentences. Each clause carries a complete idea. For instance, “The higher the temperature, the greater the internal energy, the internal energy is converted into kinetic energy and released” shows a progressive structure, adding complexity and information. Besides, the syntax is relatively complex, employing causal clauses and explanatory phrases. For example, “The principle of diffusion is” introduces a definition followed by several clauses and phrases attempting to explain the mechanism of diffusion. This use of multiple causal relationships and parallel structures enhances the complexity of the sentence. It also employs multiple complex noun and verb phrases, such as “molecules are constantly in random motion” and “internal energy is converted into kinetic energy and released,” which are both scientifically rigorous and descriptive of specific physical processes, demonstrating high technicality and professionalism in language use.

The sentence’s structure is intricate, involving several physical processes and transformations, potentially making it difficult for the ChatGPT to accurately capture the logical relationships and scientific concepts within each clause. This complexity necessitates a scoring algorithm capable of high semantic understanding and precise handling of scientific terminology and logical structures, which may result in less accurate scoring and lead to a high risk of misscored, where students with lower levels of performance may receive higher scores.

Compared to lower-performing responses, higher-performing ones included scientific terms more frequently. They used polysemous words, pronouns, and synonyms less often. Additionally, distinctive patterns emerged between the Higher level and Correctly scored (HC) subgroup and the Higher level and Misscored (HM) subgroup.

The higher level and correctly scored subgroup. At the word level, responses from the HC subgroup averaged about 46 Chinese characters, which had 20 more characters compared with the HM subgroup. Additionally, the HC subgroup used mathematical symbols or formulas more frequently than the HM subgroup. The longer responses and the use of symbols/formulas offered more detailed and precise explanations. At the sentence level, the HC subgroup’s responses exhibited several features distinct from those of the HM subgroup: (1) most responses of the HC subgroup included multiple sentences with complex syntactic structures, incorporating multiple clauses and modifiers; (2) They often employed numerous noun phrases to explain phenomena, such as “rate of change of magnetic flux”; (3) Verb structures were more complex and varied, such as “partially converted the mechanical energy of the magnet into thermal energy”; (4) Each student response contained a higher number of sentences, typically ranging from 5 to 10; (5) Prepositional phrases were also more prevalent, such as “entering the coil” and “during the free fall of the magnet” clearly indicating action states.

For instance, the response to item 3, “As the magnet enters the coil, due to the changing magnetic field inside the coil induces an electric current, heat is generated. According to the equation △Ep = △Ek + Q, as Q is generated, △Ek decreases, resulting in a decrease in △V, which means the magnet’s motion is hindered (No. 43),” accurately applies the principle of energy conservation and correctly uses the equation, thus receiving a score of 1 on the Theory dimension. It effectively employs non-visible theories and equations to link multiple related variables, explaining the mechanism of the magnet’s fall. Therefore, it demonstrates a higher reasoning complexity level.

This response exceeded 70 Chinese characters, used numerous technical terms, such as “electric current” and “magnetic field”, and employed symbols like “Q”, “△Ek”, and “△V”, as well as related equations to precisely describe the physical phenomena and processes, reducing ambiguity. It also includes multiple prepositional phrases, such as “due to the change in the magnetic field inside the coil” and “as Q is generated,” providing temporal and causal background information for a more complete description. Although it is a long sentence, it uses commas to separate different clauses and phrases, maintaining a strict logical progression. Starting from the fact that “the magnet enters the coil,” it connects through the theory “△Ep = △Ek + Q,” and finally concludes with “the magnet’s motion is hindered,” thoroughly explaining the phenomenon of the magnet’s fall. This complete and precise expression style reduces the likelihood of parsing errors and helps the machine algorithm accurately evaluate the student’s response, allowing the machine scoring to achieve high accuracy in this context, as the machine has more accurate information provided by the student to compare with the scoring criteria.

The Higher level and Misscored subgroup. Responses from the HM subgroup averaged about 26 Chinese characters. These responses seldom use mathematical symbols and formulas, relying mainly on textual explanations. The syntax was not complex, consisting mostly of simple and short sentences, which made them easy to understand. The noun phrases used were limited and simple, such as “change in a magnetic field” and “The work done by the pump on the gas”. The verb structures are relatively straightforward, like “induces an electric current” and “generate heat”. Compared with the HC subgroup, each response in the HM subgroup contained fewer sentences, typically ranging from 1 to 3, and prepositional phrases were fewer and simpler, such as “doing work on the gas” resulting in an unclear representation of physical states.

Taking the HM subgroup response to item 3 as an example, the response, “The coil will exert resistance on the magnet because of energy conservation (No. 101),” employs the principle of energy conservation to provide a concise explanation of the phenomenon, earning a score of 1 on the theory element. However, it simply links energy conservation with the conclusion of resistance, offering some causal analyses but failing to explain the mechanism of the magnet’s fall. Therefore, it demonstrates a medium reasoning complexity.

This response contains only 19 Chinese characters and uses mostly non-technical terms, such as “resistance” instead of precise terms like “repulsive force” or “repulsion,” which may lead to ambiguity in model interpretation. The sentences do not include mathematical symbols or formulas, and the syntax is simple, consisting of only a single simple sentence. It lacks prepositional phrases to describe the state of physical phenomena, resulting in a lack of detailed contextual explanation and specific physical mechanism inference. These linguistic features will lead to a risk of being misscored, where students with higher performance levels may receive lower scores.