BoilerTAI: A Platform for Enhancing Instruction Using Generative AI in Educational Forums

BoilerTAI represents a significant advancement in how TAs interact with students over online discussion forums. BoilerTAI is a web-based application that integrates with Ed-Discussion ²²2https://edtsem.org to generate responses to student posts and inquiries using OpenAI’s API in order to assist AI-TAs (TAs that utilize the BoilerTAI platform). The AI-generated responses are reviewed by the AI-TAs for approval, modification, or reprompting, when necessary, to ensure accuracy and appropriateness.

While BoilerTAI functions as an intermediary platform, it is designed to minimize disruption to a TA’s workflow. The approval workflow introduced into the AI’s response generation process enhances the relevance of responses, ensures compliance with academic policies, and maintains a consistent tone with that of a TA. This AI-TA intervention is key to how BoilerTAI is meant to run.

BoilerTAI underwent a comprehensive five-month design, prototyping, and testing phase. This process involved collaboration with instructional specialists and computer science professors to ensure that the platform’s prompt building and data collection features effectively utilized the strengths of OpenAI’s API, focusing on usability and relevance.

The frontend is a static web application deployed on Surge.sh with Firebase Hosting, managed by a Cloudflare worker that handles load balancing through routing and caching strategies. The backend is a Dockerized Flask service deployed on Fly.io, which provides containerized micro-VMs across multiple regions globally³³3https://fly.io/.

Firebase’s Cloud Firestore is used as the database, managing user states, authentication, synchronization between online discussion forums and AI, and data metrics collection. This NoSQL database efficiently handles data exporting and filtering in JSON format, ensuring scalability and real-time interaction capabilities.

OpenAI’s Python API is leveraged to build a pipeline that processes student questions from the discussion forum, retrieves necessary files, and integrates these elements into prompts. The responses are stored in the database, formatted for the user interface, and presented on the frontend.

Next.js is used to build a Tailwind-styled responsive user interface. Users can pull the latest student posts from Ed Discussion through their API, view posts with existing AI-generated responses, and approve responses for posting. They can manually edit AI-generated responses or select options for re-prompting.

The next layer utilizes Firebase’s Cloud Firestore as the database that manages user states and authentication, synchronization between online discussion forums and AI, and data metrics collection. As a NoSQL database, it seamlessly handles exporting and filtering data in JSON format, ensuring efficient scalability and real-time interaction capabilities.

Adjacent to the database, we leverage OpenAI’s Python API to build a pipeline. This system processes student questions from an online discussion forum, carries out necessary file retrieval tasks, and integrates these elements into our prompt. The received response is then stored in the database, formatted to fit the user interface, and presented on the frontend.

On the topmost layer, Next.js is used to build out a Tailwind-styled responsive user interface. Here, a user is able to pull the latest student posts from Ed Discussion through their API, view posts with existing AI-generated responses, and approve a response for posting. By viewing, they are able to manually edit the AI-generated response or select from options for re-prompting (discussed in further detail below).

Please refer to Figure 1 during the following discussion for the operational pipeline of the BoilerTAI platform.

The operational workflow of the BoilerTAI platform begins when a student posts a question on the EdStem discussion board. Once the question is picked up on the automated periodic checks for unanswered queries, it is fed into the GPT-4 model. This step involves using a carefully crafted prompt, which is crucial for guiding the AI to generate contextually appropriate and pedagogically sound responses.

Once a response is generated, it is displayed on the BoilerTAI platform for an AI-TA to review. It is at this stage that our human-in-the-loop approach becomes pivotal. The AI-TAs play a critical role in ensuring the quality and appropriateness of the AI-generated responses. They have the authority to approve the response directly, reprompt the GenAI model in various ways, or choose to modify it directly all to better meet educational standards and the specific needs of the student.

The AI-TAs have several tools at their disposal to refine the AI’s output via reprompting. They can decide to preserve the historical context, which involves using information from previously generated responses to inform current and future responses. Additionally, they can alter the response by allowing or disallowing code, adjusting the level of detail, or even crafting a completely custom prompt. This last option allows AI-TAs to provide specific context to the AI, guiding it to produce responses that are not only accurate but also aligned with the instructional goals of the course.

Prompt engineering has been identified as a crucial element for optimizing the performance of AI tools like GPT, ensuring the generation of high-quality, contextually appropriate responses [29]. The crafting of effective prompts is guided by principles outlined in the CLEAR Framework, a set of guidelines that underscores the significance of precision and clarity in communication, especially crucial in the Information Age [30].

For BoilerTAI, using the CLEAR principles meant constructing a prompt that was straightforward and simple, focusing on facilitating problem-solving capabilities within students to enhance their self-sufficiency. This approach aligns with the broader educational objectives of ensuring independent learning and critical thinking among students.

To achieve these objectives, the prompt designed for BoilerTAI strictly defines the role of the AI as an objective teaching assistant in an undergraduate Computer Science course. The AI is configured to interact on discussion boards with a focus on guiding students towards solutions rather than providing direct answers. This ensures that the AI supports the learning process without bypassing the essential educational steps that students need to undertake themselves.

The results of the study demonstrate that the implementation of BoilerTAI led to a significant enhancement in the efficiency of responses to student inquiries. Notably, 75% (3 out of 4) of participants reported experiencing improvements across various degrees of magnitude, as outlined in Table I. Moreover, all respondents (4 out of 4) unequivocally acknowledged a discernible improvement in the quality of responses provided, as detailed in Table II. These empirical observations substantiate the primary objective of the investigation, which aims to optimize response efficiency while concurrently enhancing the overall quality of educational interactions.

The reception of AI-assisted responses from students has been notably favorable. Analysis indicates that approximately 75% (3 out of 4) of teaching assistants observed that responses generated through BoilerTAI elicited positive feedback from students (Table III). This finding underscores the effectiveness of AI integration in augmenting educational interactions and suggests that the utilization of AI technologies does not compromise the quality of the educational experience. Such considerations are critical for the widespread acceptance and implementation of such innovations within learning environments.

An overwhelming majority of the teaching assistants perceived the AI-generated responses as relevant and personalized. Specifically, approximately 75% (3 out of 4) of respondents concurred that BoilerTAI’s responses were appropriately tailored to individual student inquiries (Table IV). This observation underscores the significance of personalized feedback within educational settings, where tailored responses are pivotal in enhancing student understanding and engagement. Such findings highlight the importance of leveraging AI technologies to deliver tailored educational experiences that cater to the diverse needs of students.

However, it was also noted that there persists a need for oversight by the teaching assistants. Approximately 50% (2 out of 4) of the teaching assistants reported requiring significant alterations or corrections to the responses suggested by BoilerTAI in approximately half of the instances (Table V). This observation highlights the crucial role of preserving a human element in the AI-assisted educational process. It underscores the necessity that responses are not solely factually accurate but also contextually appropriate. Such supervision ensures that educational interactions uphold the requisite depth and nuance essential for fostering effective learning outcomes.

The teaching assistants recognized a robust potential for further integrating BoilerTAI into educational practices. Seventy-five percent (3 out of 4) somewhat agreed, while 25% (1 out of 4) strongly agreed that there exists significant potential for enhancing learning experiences through deeper integration of this AI tool into regular teaching practices (Table VI). This sentiment not only reflects an optimistic outlook for the future of AI in education but also indicates an acknowledgment of its broader applications and potential for deeper integration across various educational scenarios. Such recognition underscores the importance of continued exploration and implementation of AI technologies to enrich educational experiences and improve learning outcomes.

The open-ended responses highlight how BoilerTAI enhances efficiency, reduces cognitive load, saves time, and effectively addresses certain types of student queries, particularly those related to course concepts. Users find that BoilerTAI simplifies the process of responding to student queries by reducing the cognitive load. Instead of needing to consider the formulation of an answer, its clarity, and correctness simultaneously, they can focus primarily on ensuring the accuracy of the response. This streamlining of the process makes it more efficient and less mentally taxing.

In particular, one participant noted:

Prior to using the tool, I had to think about the answer to the question, phrasing that answer in a way that the student understands, and making sure that the final response is correct. Now, I mainly have to focus on only one of those three (ensuring correctness).

Another participant highlighted the time-saving aspect of BoilerTAI:

It decreases the time it takes me to google a concept, type out a response and send it. The scaffold it creates for an answer is really helpful.

A third participant found BoilerTAI particularly helpful for addressing generic questions:

It has been helpful in answering generic questions that are well described by students in the post related to concepts of the course.

BoilerTAI has shown to significantly enhance the efficiency and quality of TA responses. According to the study, 75% of AI-TAs reported improved efficiency in handling queries, while all participants noted an enhancement in response quality. Furthermore, the platform’s capacity to generate scaffolds for answers has been particularly praised, as it aids in streamlining communication and reduces the cognitive load involved in crafting clear and accurate responses, enabling quicker and more organized exchanges between students and instructors.

AI tools, while guiding students beyond their current understanding, typically lack the nuanced interactions human MKOs offer, such as using questioning techniques or providing partial solutions to promote deeper learning [32, 33, 34]. BoilerTAI mitigates this through advanced prompt engineering to ensure AI responses closely mimic those of human TAs, fostering engagement and critical thinking.

Positioning AI-TAs as intermediaries between students and the AI ensures essential human oversight, enhancing rather than replacing the human element in educational interactions. This strategy not only reduces the workload of TAs but also maintains the educational benefits of the MKO model. Despite the positive reception, with 75% of AI-TAs noting student satisfaction, 50% of AI-TAs still needed to correct or significantly alter AI responses about half the time. This underscores the importance of combining AI capabilities with human judgment to ensure the accuracy and contextual appropriateness of responses. Such an approach is also important for mitigating ethical concerns, particularly with maintaining academic integrity and ensuring that AI-generated responses do not mislead students or provide incorrect information [35]. However, it is important to keep in mind that as the field of GenAI grows, GenAI agents may be able to meet new ethical and accuracy standards (possibly even above TAs). We maintain that at the time of publication, GenAI agents are untested against such standards (in fact, one set of such standards has not been accepted by the general community) and therefore still require human oversight.

The BoilerTAI platform has been engineered to address specific challenges and limitations of utilizing AI in education, particularly in complex subject areas like science education. Issues such as the lack of context and evidence in AI-generated responses, concerns about academic integrity, potential for hallucinations (incorrect or misleading results) [36, 35, 37], and concerns like the Junior-Year Wall [38] are addressed by having TAs add context and corroborate facts, effectively saving time and ensuring reliability.

However, the study faced certain limitations, including a smaller TA sample size and the fact that not all TAs frequently answered questions on their discussion forums (greater than 2 days out of the week). These factors may have affected the generalizability of the findings and the overall evaluation of BoilerTAI’s effectiveness. Nonetheless, the results should still be considered impactful, as there were consistent improvements in response efficiency and quality reported by the participating AI-TAs across their 4 distinct programming courses, which provides a strong indication of the platform’s benefits.

The favorable reception of AI-assisted responses underscores BoilerTAI’s effectiveness in maintaining high-quality educational interactions. However, the recent rise of GenAI leaves open many other research avenues. Future research could explore: student perceptions on our (educators’) efforts to include (and restrict) GenAI in their education; the impact of our approaches on students’ achievement, in particular if it allows us to add content to the curriculum or effectively “raise the bar” on expected student performance; similarly, if including and allowing GenAI in education allows us to move our learning objective for particular assessment types “up” on scales like the revised Bloom’s Taxonomy [39]; importantly, how do the educational principles that underpin tools like BoilerTAI translate to the K-12 space; and lastly on this non-comprehensive list (but perhaps the most interesting to the authors), how students’ learning is impacted by (a) usage of AI and (b) our interventions that involve GenAI (including measures of learning from performance-based and cognitive/behavioral measures to affective and collaborative learning measures). For BoilerTAI in particular, further research is essential to fully understanding the scalability of the human-in-the-loop approach and to refine AI’s role in both enhancing efficiency and being a supportive tool within educational environments.