Leveraging Large Language Model and Story-Based Gamification in Intelligent Tutoring System to Scaffold Introductory Programming Courses: A Design-Based Research Study

Chen Cao [email protected] 0000-0003-4368-0336 University of SheffieldUnited KingdomS10 2TN

(2023)

Abstract.

Programming skills are rapidly becoming essential for many educational paths and career opportunities. Yet, for many international students, the traditional approach to teaching introductory programming courses can be a significant challenge due to the complexities of the language, the lack of prior programming knowledge, and the language and cultural barriers. This study explores how large language models and gamification can scaffold coding learning and increase Chinese students’ sense of belonging in introductory programming courses. In this project, a gamification intelligent tutoring system was developed to adapt to Chinese international students’ learning needs and provides scaffolding to support their success in introductory computer programming courses.

My research includes three studies: a formative study, a user study of an initial prototype, and a computer simulation study with a user study in progress. Both qualitative and quantitative data were collected through surveys, observations, focus group discussions and computer simulation. The preliminary findings suggest that GPT-3-enhanced gamification has great potential in scaffolding introductory programming learning by providing adaptive and personalised feedback, increasing students’ sense of belonging, and reducing their anxiety about learning programming.

^†^†conference: IUI’23: 28th International Conference on Intelligent User Interfaces; March 27 – 31, 2023; Sydney, Australia^†^†booktitle: IUI’23: 28th International Conference on Intelligent User Interfaces, March 27 – 31, 2023, Sydney, Australia^†^†journalyear: 2023^†^†copyright: rightsretained^†^†conference: 28th International Conference on Intelligent User Interfaces; March 27–31, 2023; Sydney, NSW, Australia^†^†booktitle: 28th International Conference on Intelligent User Interfaces (IUI ’23 Companion), March 27–31, 2023, Sydney, NSW, Australia^†^†doi: 10.1145/3581754.3584111^†^†isbn: 979-8-4007-0107-8/23/03

1. Problem Statement

Computing and technology are increasingly ubiquitous and have become a necessary part of many educational paths, professional opportunities, and industries (Pedro et al., 2019). Consequently, the importance of programming knowledge and coding skills has grown significantly in recent years. Introductory programming courses, often referred to as computer science 1 (CS1) courses are designed to introduce students to the fundamentals of programming and coding (Becker and Quille, 2019). However, these courses can be challenging for many international students, who may lack prior programming expertise and are confronted with a language and cultural barriers(Khanal and Gaulee, 2019).

Recent years have also seen an increased number of Chinese international students enrolling in universities in the UK(de Wit, 2020). Despite the clear benefits of international student enrollment, Chinese international students can often feel isolated in their new environment and struggle to integrate into university life(Chen and Zhou, 2019). In the context of programming education, Chinese international students often face hardship due to a lack of prior exposure to coding concepts and language (Alaofi and Russell, 2022). As a result, they are more likely to experience anxiety and a feeling of low belonging. However, previous research showed bias and stereotypes when describing Chinese students’ learning behaviours in global higher education (Heng, 2018). Chinese international students are often perceived as passive and reluctant learners when adapting to British educational systems (Zhu and O’Sullivan, 2022). This stereotype, however, fails to take into account the cultural and educational factors driving these behaviours.

At the same time, gamification has seen a surge in popularity in the educational sector. It has been used as a tool to engage students and encourage learning in a range of contexts (Welbers et al., 2019). This presents an opportunity to investigate the potential of gamification to scaffold coding learning and increase the sense of belonging among Chinese international students in introductory programming courses. Additionally, the use of educational technologies, such as intelligent tutoring systems in educational settings has been growing steadily in recent years, as an increasing number of educators recognize the potential of this approach to engage and motivate learners (Szymkowiak et al., 2021). An intelligent tutoring system is particularly attractive due to its ability to personalise the learning experience, offering tailored activities to meet the needs of diverse learners (Shemshack et al., 2021). In addition, an intelligent tutoring system can provide feedback in a timely and effective manner, offering learners the opportunity to improve their skills in a supportive and engaging environment (AlShaikh and Hewahi, 2021).

The research background of this thesis is rooted in the idea that AI-enhanced gamification can be used to scaffold learning and increase belonging among Chinese international students in introductory programming courses. AI-enhanced gamification combines the use of gamification techniques with AI-driven intelligent tutoring systems to create a personalised learning experience. This approach encourages learners to engage in the learning process and offers them the opportunity to practise and improve their coding skills in a supportive and engaging environment.

This research is motivated by the need to bridge the gaps in programming knowledge between Chinese international students and their domestic peers. It is also driven by the urgent need to create an inclusive learning environment that is accessible and welcoming to all students. The research findings could have a considerable impact on the teaching and learning of programming for Chinese international students. It has the potential to provide insight into how AI-enhanced gamification can be used to make the learning process easier and more effective for these students, as well as to foster a sense of belonging. The findings could inform instructors, administrators, and policymakers of the most effective strategies for teaching and learning to program for this population. This study can also be expanded to support other underrepresented student groups (such as female STEM students and other international students from different countries and cohorts) who also need a sense of belonging to their peers, faculty, and subject-related careers.

2. Research aim and questions

The main focus of this research project is to explore the potential of AI-enhanced gamification to scaffold coding learning and increase the sense of belonging among Chinese international students in introductory programming courses. Specifically, the focus will be on designing, evaluating, and refining the use of AI-enhanced gamification to improve learning outcomes and increase motivation among Chinese international students in introductory programming courses.

The research project is framed within the context of design-based research, which emphasises the importance of designing, implementing, and evaluating a learning environment, with the goal of optimising the learning experience. The research project is guided by two research questions:

•

RQ1: what are the challenges that Chinese international students are facing regarding developing a sense of belonging and code learning in introductory programming courses?
•

RQ2: how can AI and gamification be used to scaffold coding learning and increase the sense of belonging among Chinese international students in introductory programming courses?

3. Related work

This research is situated within the larger context of educational technology and game-based learning. The use of technology in education has become increasingly popular, with a growing number of educators recognising the potential of this approach to engage and motivate learners (Szymkowiak et al., 2021). This study specifically focuses on the application of AI-enhanced gamification to scaffold coding learning among Chinese international students.

There has been a considerable amount of research examining the challenges and barriers associated with teaching introductory programming courses, e.g. (Alam, 2022). These studies have highlighted the need for more effective pedagogical approaches to engage learners in CS1 courses. Several researchers have proposed the use of game-based learning techniques such as serious games and simulations to increase engagement and motivation among students (Papadakis and Kalogiannakis, 2019) . Gamification combines elements of gaming (e.g., points, rewards, leaderboards) with traditional educational activities to create an engaging learning experience.

The use of intelligent tutoring systems (ITS) in programming education has also been studied extensively. Research has shown that ITS can improve student performance and reduce cognitive load in programming courses. For example, Grenander et al. (Grenander et al., 2021) proposed an AI-enhanced educational system that was designed to provide personalised feedback based on individual learners’ needs and evaluated its effectiveness using deep discourse analysis. Similarly, Eguchi (Eguchi et al., 2021) investigated the use of AI-enhanced games to support STEM education for children with visual impairments. They found that the AI-enhanced game had a positive impact on engagement and motivation among participants. Furthermore, ITS can provide personalised instruction and targeted remediation, which can be particularly beneficial for those who lack prior experience in programming. In particular, GPT-3 (Generative Pre-trained Transformer 3), an advanced language model developed by OpenAI, has been used to improve the performance of ITSs by providing natural language understanding capabilities(Tack and Piech, 2022).

This research project is also informed by recent studies on belonging in higher education. It has been argued that belonging is an important factor when it comes to understanding and supporting the learning experiences of international students (Cureton and Gravestock, 2019). Studies have also shown that international students often face difficulties in developing a sense of belonging due to language and cultural barriers (Cena et al., 2021).

4. Research method

This research project applies design-based research (DBR) as its methodological approach. DBR is an iterative process that emphasises the importance of designing, implementing, and evaluating educational technology to optimise the learning experience. This approach has been widely used in educational settings to investigate the effectiveness of new technologies and approaches in engaging and motivating learners. The use of DBR provides a unique opportunity to combine theoretical insights with practical implementation to create meaningful interventions that are responsive to the needs of diverse learners.

The research is divided into three phases: 1) a survey to identify Chinese students’ needs and the challenges they met regarding a sense of belonging and learning experience in introductory programming courses; 2) a design probe with a story-based gamification prototype to increase Chinese students’ sense of belonging and improve their learning experience in CS1 courses with user evaluation; 3) the development and systematic evaluations of an intelligent tutoring system leveraging large language model and gamification features for CS1 courses.

The study used a mixed-methods approach, incorporating both quantitative and qualitative data. The quantitative data were collected through a survey measuring students’ sense of belonging, academic performance, and academic emotions, and a computer simulation study evaluating the performances of the ITS. The qualitative data were collected through participatory observations and focus group interviews to explore students’ experiences and perceptions in more depth. The data were analyzed using descriptive statistics, thematic analysis, and computational analysis.

5. Preliminary results

5.1. Study 1: Formative study

The initial survey was conducted as a formative study in the first semester of the 21/22 academic year. Based on the questionnaire with 57 Chinese international students and in-depth interviews with nine of them, the study found a number of unique challenges faced by the participants in the CS1 courses. Chinese students generally found it difficult to cope with the demands of the programming course. A number of factors were identified as contributing to this difficulty, including the challenges to adapt to new teaching methods emphasizing independent thinking, critical thinking, and innovative learning; less interaction with teachers and classmates; disconnection between the knowledge and real-life cases and not receiving enough academic support with timely help and real-time feedback. As a result, they have low engagement in classes, low retention and negative academic emotions. One of the key factors that have been identified as contributing to these unsmooth academic transition experiences is a lack of intrinsic motivation, especially a lack of sense of belonging.

5.2. Study 2: A prototype of story-based ITS

The second study designed, deployed and evaluated a prototype of a story-based gamification design on a learning management system Blackboard. Based on the initial findings from the survey, the prototype adopted a set of gamification features, including animated trailers, story-telling, role-play, teamwork, points, leaderboards and feedback. The study was conducted over a period of two weeks with a total of 34 Chinese students enrolled in the introductory programming course INF6032 Big Data Analytics, which is one of the core modules of the MSc in Data Science programme at the University of Sheffield. The practical session in week 6 was set in the context of airline companies’ survival during the pandemic. In week 7, the story was about solving a criminal case. The narrative storytelling of each session with real-life cases was designed to make students feel more connected to the programme.

The findings from questionnaires (n=32), focus groups (n=32) and participatory observation revealed that Chinese students generally had positive perceptions of the effect of story-based gamification design on their sense of belonging in the introductory programming course. The majority of students reported feeling more motivated to learn, more engaged in the course, and more connected to their classmates as a result of the story-based gamification design. In addition, the findings suggested that story-based gamification design had a positive effect on Chinese students’ sense of belonging in the introductory programming course. Most students reported feeling a greater sense of belonging in the course after the story-based gamification design was implemented. It was also found that Chinese students have some unique cultural needs which should be considered in the story-based gamification design. These findings suggest that story-based gamification design can be an effective means of improving Chinese students’ sense of belonging in the introductory programming course.

5.3. Study 3: A model of the story-based ITS empowered by GPT-3

The reflection upon the findings from the previous study encouraged the development of the current model of the story-based intelligent tutoring system (ITS). The ITS empowered by GPT-3 was designed to better fit the needs of individual learners, and increase their sense of belonging to the class, institution and career-related subject, in order to further foster inclusion in higher education.

The design of the user interface is inspired by the English TV series Sherlock with abundant modern English elements, which is ideal to engage international students coming to the UK for higher education. The user interface of the ITS was built on a popular open-source template¹¹1Bootstrap 4 UI kit, https://demos.creative-tim.com/now-ui-kit-react/#/index?ref=nukr-github-readme in React, a mainstream JavaScript library for developing front-ends. The overall design principles of the web app are inspired by the famous detective, Sherlock Holmes. The colour scheme, typography and navigation are all based on the idea of the Mind Palace, with a focus on the dark blue and white colours to convey a feeling of sophistication and intelligence.

The development of the learning system consists of four main components: the instructional content, the gamification mechanism design, the user interface, and the generative language model. The learning content consists of the introduction of programming knowledge, demo explanations and exercises to provide the learning materials to the users. The game engine was designed to provide a fun and engaging learning experience. It consisted of a set of gamification elements, including alternative reality, points, badges, personalized feedback and encouragements, levels and challenges, an avatar, a progress bar and an exploratory word cloud. The user interface is responsible for displaying the learning content and gamification mechanisms to the users and accepting user input. The generative language model is used for providing AI capabilities such as Q&A and explaining code to the system.

The system is designed to be a web-based application. It consists of three main components, which are the front end, the back end and the GPT-3 platform. The front end used React library to develop the user interface and enable interactions. The back end is responsible for data storage, processing and retrieval. In this system, Firebase is applied to connect with the interface and synchronise data. All the states (user behaviours and inputs) in the front end were centralised with Redux and synchronised to Firebase, where all the user data, such as the user’s avatar, learning progress and chat history with the AI conversational agents were stored for further analysis.

GPT-3 was used in this system to train the AI model for the intelligent agents. The agents were designed to interact with the users, and provide guidance and support throughout the learning process. In addition, the agents were also responsible for marking the user’s progress and giving feedback. There were four chatbots providing support to students with different functions, including the intelligent tutoring bot Sherlock, the peer and critical thinking bot Watson, the career engagement bot Inspector Lestrade and the emotion support bot Mrs Hudson. When the student asked a question, Sherlock answered it first. Then Watson, Inspector Lestrade and Mrs Hudson generated follow-up questions to continue the conversation, in order to foster inquiry-based learning and improve students’ computational thinking and understanding of job market needs.

In the interface of marking and feedback, students can submit the codes for quizzes and get real-time and tailored feedback about their answers. The model first extracts the code snippet from the user input and then generates an explanation for the code snippet by calling the GPT-3 API with the code snippet and a prompt as input. The prompt is a natural language question that can be used to generate an explanation for the code snippet. The code-explaining module uses a set of predefined prompts to generate explanations for the code snippets. The prompts are selected to cover a wide range of programming concepts.

A computer simulation study and a pilot user study were conducted to evaluate the ITS. By using prompt programming, a dataset containing 360 rounds of simulated questions that students may ask in the class and answers were simulated based on the learning objectives and instructors’ reflections. The findings from semantic similarity analysis, topic modelling and sentiment analysis indicated that GPT-3 empowered agents performed well in providing feedback and having insightful conversations, but the quality of answers depend on the form and level of the questions. The ITS was also piloted with different stakeholders, including potential users, instructors, practitioners and researchers to assess the usability, acceptability and effectiveness of the system. Findings from the pilot study indicated that the system performs well in terms of usability and acceptability. Most participants reported feeling positive about their experience with the intelligent tutoring system, with many saying they were impressed and felt supported by the AI agents. They also commented positively on the gamification elements in the system, such as the intriguing storyline of Sherlock.

6. Implications and future work

The current studies suggested that GPT-3 as a large language model trained on a vast amount of text data is particularly well-suited to answering questions at the remembering and understanding levels of Bloom’s taxonomy. These levels involve recalling and comprehending information, which is a strength of large language models like GPT-3. IT also performed well in answering questions at the higher levels of the taxonomy, such as analysis, synthesis, and evaluation if the questions are well-formed. Generally, GPT-3’s ability to answer questions in the educational scenario of CS1 courses is promising but limited by its lack of domain-specific knowledge and expertise.

Despite the promising results of this study, there are still a number of areas for further research and development. Firstly, more detailed studies should be conducted to investigate the long-term impact of AI-enhanced gamification on students’ learning outcomes and sense of belonging in introductory programming courses. Secondly, contextual factors such as culture, language and prior knowledge should be taken into account when designing AI-enhanced gamification systems to better support Chinese international students. Thirdly, more sophisticated machine learning models can be explored to improve the performance of AI agents. Finally, more user studies need to be conducted to explore how AI-enhanced gamification can be used to foster collaboration among team members, reduce anxiety and facilitate learning transfer.

In order to fully realize the potential of the GPT-3 enhanced ITS to increase student sense of belonging in higher education, more research must be conducted in real educational scenarios. More user studies are needed to assess how GPT-3 affects student academic performance, motivation, and sense of belonging. In addition, researchers must continue to refine GPT-3’s capabilities to improve the ITS’s ability to pick up on subtle social cues and to develop a better understanding of student emotions and motivations.

References

(1)
Alam (2022) Ashraf Alam. 2022. Platform Utilising Blockchain Technology for eLearning and Online Education for Open Sharing of Academic Proficiency and Progress Records. In Smart Data Intelligence. Springer, 307–320.
Alaofi and Russell (2022) Suad Alaofi and Seán Russell. 2022. The Influence of Foreign Language Classroom Anxiety on Academic Performance in English-based CS1 Courses. ACM International Conference Proceeding Series. https://doi.org/10.1145/3555009.3555020
AlShaikh and Hewahi (2021) Fatema AlShaikh and Nabil Hewahi. 2021. Ai and machine learning techniques in the development of Intelligent Tutoring System: A review. In 2021 International Conference on innovation and Intelligence for informatics, computing, and technologies (3ICT). IEEE, 403–410.
Becker and Quille (2019) Brett A Becker and Keith Quille. 2019. 50 years of cs1 at sigcse: A review of the evolution of introductory programming education research. In Proceedings of the 50th acm technical symposium on computer science education. 338–344.
Cena et al. (2021) Elida Cena, Stephanie Burns, and Paul Wilson. 2021. Sense of Belonging, Intercultural and Academic Experiences among International Students at a University in Northern Ireland. Journal of International Students (2021).
Chen and Zhou (2019) Jia Chen and George Zhou. 2019. Chinese international students’ sense of belonging in North American postsecondary institutions: A critical literature review. Brock Education Journal 28, 2 (2019), 48–63.
Cureton and Gravestock (2019) Debra Cureton and Phil Gravestock. 2019. ‘We Belong’: differential sense of belonging and its meaning for different ethnicity groups in higher education. Compass: Journal of Learning and Teaching (2019).
de Wit (2020) Hans de Wit. 2020. Internationalization of Higher Education: The Need for a More Ethical and Qualitative Approach. Journal of International Students 10 (2020).
Eguchi et al. (2021) Amy Eguchi, Hiroyuki Okada, and Yumiko Muto. 2021. Contextualizing AI education for K-12 students to enhance their learning of AI literacy through culturally responsive approaches. KI-Künstliche Intelligenz 35, 2 (2021), 153–161.
Grenander et al. (2021) Matt Grenander, Robert Belfer, Ekaterina Kochmar, Iulian V Serban, François St-Hilaire, and Jackie C K Cheung. 2021. Deep Discourse Analysis for Generating Personalized Feedback in Intelligent Tutor Systems. https://github.com/UKPLab/sentence-transformers
Heng (2018) Tang T. Heng. 2018. Different is not deficient: contradicting stereotypes of Chinese international students in US higher education. Studies in Higher Education 43 (1 2018), 22–36. Issue 1. https://doi.org/10.1080/03075079.2016.1152466
Khanal and Gaulee (2019) Jeevan Khanal and Uttam Gaulee. 2019. Challenges of international students from pre-departure to post-study: A literature review. Journal of International Students 9, 2 (2019), 560–581.
Papadakis and Kalogiannakis (2019) Stamatios Papadakis and Michail Kalogiannakis. 2019. Evaluating the effectiveness of a game-based learning approach in modifying students’ behavioural outcomes and competence, in an introductory programming course. A case study in Greece. International Journal of Teaching and Case Studies 10, 3 (2019), 235–250.
Pedro et al. (2019) Francesc Pedro, Miguel Subosa, Axel Rivas, and Paula Valverde. 2019. Artificial intelligence in education: Challenges and opportunities for sustainable development. (2019).
Shemshack et al. (2021) Atikah Shemshack, Jonathan Michael Spector, et al. 2021. A comprehensive analysis of personalized learning components. Journal of Computers in Education 8, 4 (2021), 485–503.
Szymkowiak et al. (2021) Andrzej Szymkowiak, Boban Melović, Marina Dabić, Kishokanth Jeganathan, and Gagandeep Singh Kundi. 2021. Information technology and Gen Z: The role of teachers, the internet, and technology in the education of young people. Technology in Society 65 (2021), 101565.
Tack and Piech (2022) Anaïs Tack and Chris Piech. 2022. The AI Teacher Test: Measuring the Pedagogical Ability of Blender and GPT-3 in Educational Dialogues. ArXiv abs/2205.07540 (2022).
Welbers et al. (2019) Kasper Welbers, Elly A Konijn, Christian Burgers, Anna Bij De Vaate, Allison Eden, and Britta C Brugman. 2019. Gamification as a tool for engaging student learning: A field experiment with a gamified app. E-Learning and Digital Media 16, 2 (2019), 92–109.
Zhu and O’Sullivan (2022) Haiping Zhu and Helen O’Sullivan. 2022. Shhhh! Chinese students are studying quietly in the UK. Innovations in Education and Teaching International 59, 3 (2022), 275–284.